Browsed by
Tag: storage

Perennially reservations weird behaviour whilst not configured correctly

Perennially reservations weird behaviour whilst not configured correctly

Whilst using RDM disks in your environment you might notice long (even extremely long) boot time of your ESXi hosts. That’s because ESXi host uses a different technique to determine if Raw Device Mapped (RDM) LUNs are used for MSCS cluster devices, by introducing a configuration flag to mark each device as perennially reserved that is participating in an MSCS cluster. During the start of an ESXi host, the storage mid-layer attempts to discover all devices presented to an ESXi host during the device claiming phase. However, MSCS LUNs that have a permanent SCSI reservation cause the start process to lengthen as the ESXi host cannot interrogate the LUN due to the persistent SCSI reservation placed on a device by an active MSCS Node hosted on another ESXi host.

Configuring the device to be perennially reserved is local to each ESXi host, and must be performed on every ESXi host that has visibility to each device participating in an MSCS cluster. This improves the start time for all ESXi hosts that have visibility to the devices.

The process is described in this KB  and is requires to issue following command on each ESXi:

 esxcli storage core device setconfig -d –perennially-reserved=true

You can check the status using following command:

esxcli storage core device list -d

In the output of the esxcli command, search for the entry Is Perennially Reserved: true. This shows that the device is marked as perennially reserved.

However, recently I came across on a problem with snapshot consolidation, even storage vMotion was not possible for particular VM.

Whilst checking VM settings one of the disks was locked and indicated that it’s running on a delta disks which means there is a snapshot. However, Snapshot manager didn’t showed any snapshot, at all. Moreover, creating new and delete all snapshot which in most cases solves the consolidation problem didn’t help as well.


In the vmkernel.log while trying to consolidate VM lots of perenially reservation entries was present. Which initially I ignored because there were RDMs which were intentionally configured as perennially reserved to prevent long ESXi boot.


However, after digging deeper and checking a few things, I return to perenially reservations and decided to check what the LUN which generates these warnings is and why it creates these entries especially while trying consolidation or storage vMotion of a VM.

To my surprise I realised that datastore on which the VM’s disks reside is configured as perenially reserved! It was due to a mistake when the PowerCLi script was prepared accidentially someone configured all available LUNs as perenially reserved. Changing the value to false happily solved the problem.

The moral of the story is simple – logs are not issued to be ignored 🙂

Infinio Accelerator – how it works?

Infinio Accelerator – how it works?

In my last post about Infinio Accelerator we introduced product and basics about it. Now it is time to go more deep – how this server side cache is working ?

Infinio’s cache inserts server RAM (and optionally, flash devices) transparently into the I/O stream. By dynamically populating server-side media with the hottest data, Infinio’s software reduces storage requirements to a small fraction of the workload size. Infinio is built on VMware’s vSphere APIs for I/O Filtering (VAIO) framework. This enables administrators to use VMware’s Storage Policy Based Management to apply Infinio’s storage acceleration filter to VMs, VMDKs, or groups of VMs transparently.


An Infinio cluster seamlessly supports typical cluster-wide VMware operations, such as vMotion, HA, and DRS. Introduction of Infinio doesn’t require any changes to the environment. Datastore configuration, snapshot and replication setup, backup scripts, and integration with VMware features like VAAI and vMotion all remain the same.



Infinio’s core engine is a content-based memory cache that scales out to accommodate expanding workloads and additional nodes. Deduplication enables the memory-first design, which can be complemented with flash devices for large working sets. In tiered configuration such as this, the cache is persistent, enabling fast warming after either planned or unplanned downtime.


  Note. Infinio’s transparent server-side cache doesn’t require any changes to the environment !

 Lets go with installation – is easy and entirely non-disruptive with no reboots or downtime. It can be completed in just a few steps via an automated installation wizard. The installation wizard collects vCenter credentials and location, and desired Management Console information, then automatically deploys the console :

  1. Run infinio setup and agree to license terms


2. Add vcenter FQDN and user credentials (in example we go with sso admin)


3. Select destination esxi and other parameters to deploy ovf management console vm (datastore and network)


  1. Set management console hostname and network information (IP address, DNS)


  1. Create admin user for management console


  1. setup auto-support (in our trial scenario we skip this step)


  1. Preview config and deploy management console.



  1. Login to management console



In the next article we will provide some real performance result form our lab tests – so stay tuned 🙂


Mysterious Infinio – Product overview

Mysterious Infinio – Product overview

Shared storage  performance and characteristics (iops,latency)  is crucial for overall  vSphere platform performance and users satisfaction. In the advent of ssd and memory cache solutions we have many options to chose in case storage acceleration (local ssd, array side ssd , server side ssd). Lets discuse further server side caching – act of caching data on the server.

Data can be cached anywhere and at any point on the server that makes sense. It is common to cache commonly used data from the DB to prevent hitting the DB every time the data is required. We cache the results from competition scores since the operation is expensive in terms of both processor and database usage. It is also common to cache pages or page fragments so that they don’t need to be generated for every visitor.

In this article I would like to introduce one of the commercial server side caching solution from INFINIO – Infinio Accelerator 3.


Infinio Accelerator increases IOPS and decreases latency by caching a copy of the hottest data on serverside resources such as RAM and flash devices. Native inline deduplication ensures that all local storage resources are used as efficiently as possible,reducing the cost of performance. Infinio is built on VMware’s VAIO (vSphere APIs for I/O Filters) framework,which is the fastest and most secure way to intercept I/O coming from a virtual machine. Its benefits can be realized on any storage that VMware supports; in addition, integration with VMware features like DRS, SDRS, VAAI and vMotionall continue to function the same way once Infinio is installed. Finally, future storage innovation that VMware releases will be available immediately through I/O Filter integration.


The I/O Filter is the most direct path to storage for capabilities like caching and replication that need to intercept the data path. (Image courtesy of VMware)


Infinio is licensed per ESXi host in an Infinio cluster. Software may be purchased for perpetual or term use:

  • A perpetual license allows the use of the licensed software indefinitely with an annual cost for support and maintenance.
  • A term license allows the use of software for one year, including support and maintenance.

For more information on licensing and pricing, contact

System requirements

Infinio Accelerator requires min. VMware vSphere ESXi 6 U2 (Standard, Enterprise,or Enterprise Plus) and VMware vCenter 6 U2.

Note! vSphere 6.5 is supported and on VMware HCL !

Infinio works with any VMware supported datastore, including a variety of SAN, NAS, and DAS hardware supporting VMFS, Virtual Volumes (VVOLs), and Virtual SAN (vSAN).

  • Infinio’s cluster size mirrors that of VMware vSphere’s, scaling out to 64 nodes.
  • Infinio’s Management Console VM requires 1 vCPU, 8GB RAM, and 80GB of HDD space.

I’m very happy to announce that we received very friendly response from Infinio support and we get an option to download trial version of software – next articles will describe product in more depth and show “real life” examples of use in our lab environment.

Please, stay tuned 🙂


vSphere 6.5 – Stronger security with NFS 4.1

vSphere 6.5 – Stronger security with NFS 4.1

NFS 4.1 is been supported since vSphere 6.0 and  but now we are looking into providing stronger security. In vSphere 6.5 we have better security  by providing strong cryptographic algorithms with Kerberos (AES). Also, IPV6 is supported but not with Kerberos and that is another area we are looking into along with supporting integrity checks.

Aa we know vSphere 6 NFS client also does not support the more advanced encryption type know is AES. So lets take a look at what is new in vSphere 6.5 NFS in terms of encryption standard :


  • NFS 4.1 has been supported since vSphere 6.0 ,
  • Currently support stronger cryptographic algorithms with Kerberos authentication using AES ,
  • Introducing Kerberos integrity check (SEC_KRB5i) along with Kerberos authentication in vSphere 6.5,
  • Adding Support IPV6 with Kerberos ,
  • Added Host Profiles support for NFS 4.1 ,
  • Better security for customer environments .


vSphere 6.5 – New scale limits for paths & LUNs

vSphere 6.5 – New scale limits for paths & LUNs

In vSphere 6.5 VMware  doubled  the  current limits and continuously work on reaching new scale around this . Current limits (before 6.5) pose challenge as for example in some cases our customers have 8 paths to a LUN, in this configuration one can have max of 128 LUNs in a cluster. Also, many of the customers tend to have smaller size LUNs to segregate important data for easy backup and restore. This approach can also exhaust current LUN and Path limits.

Large LUN limits  enable  to have larger cluster sizes and hence reducing management over head.storage4


  • Current Limit is 256 LUNs and 1024 Paths ,
  • This limits customer deployments requiring higher Path counts ,
  • Customers requiring small sized LUNs for important files/data require larger LUN limits to work with ,
  • Larger Path/LUN limits can enable larger cluster sizes, reducing the overhead of managing multiple clusters ,
  • Support 512 LUNs and 2K paths in vSphere 6.5 .


vSphere 6.5 – Automatic UNMAP

vSphere 6.5 – Automatic UNMAP

In vSphere 6.5 VMware are looking into automating the UNMAP process, where VMFS  would track the deleted blocks and will be able to reclaim deleted space from the backend array in back ground. This background operation should make sure that there is a minimal storage I/O impact due to UNMAP operations.


Just to remaind – UNMAP is a VAAI primitive using which we can reclaim dead or stranded space on thinly provisioned VMFS volume. Currently this can be initiated by running a simple ESX CLI command and it can free up deleted blocks from storage.

In vSphere 6.5 VMware is looking into automating the UNMAP process, where VMFS  would track the deleted blocks and will be able to reclaim deleted space from the backend array in back ground. This background operation should make sure that there is a minimal storage I/O impact due to UNMAP operations.

Lets go  though an UNMAP example stating our thought process

  1. VM is being provisioned on a vSphere host and assigned a 6 TB VMDK.
  2. There will be thin provisioned VMDK storage space allocated on storage array.
  3. User installs a POC data analytics application and creates a 400 GB database VM
  4. Once the work is done with this database, user deletes this DB VM – VMFS  initiate a space reclamation in the back ground
  5. 400GB space on the array side should be freed or claimed back

One of the design goal will be to make sure there is minimal impact due to UNMAP on storage I/O. We are also looking into using new SESparse format as a snapshot file format to enable this.

Space reclamation is critical when customers are using All Flash storage due to higher cost of Flash and any storage usage optimization will provide better ROI for customers


  • Automatic UNMAP does not require any manual intervention or scripts
  • Space reclamation happens in the background
  • CLI based UNMAP continues to be supported
  • Storage I/O impact due to automatic UNMAP is minimal

Supported in vSphere 6.5 with new VMFS 6 datastores

vSphere 6.5 – VMFS6 & 512e HDD support

vSphere 6.5 – VMFS6 & 512e HDD support

vSphere 6.5 introduces a new VMFS 6 – but why we need new version You ask? –answer: to support new hdd type, and this  point  us to current storage market situation . Well because with  512bytes sector size HDD’s  vendors are hitting drive capacity limits. They can not go beyond a certain size without compromising the resilience and reliability (not the best option in case of our data).             To provide large capacity drives, Storage Industry is moving forward to Advance format (AF) drives. These drives use large physical sector size of 4096 bytes.


So how does it help? With new AF (4K sector size) format, Disk drive vendors can create more reliable and large capacity HDD to support the growing storage needs. These drives are more cost effective as they provide better $/GB ratio.

Two kinds of 4k drives:

  1. 512 Emulation (512e) mode – these are 4KN drives but expose logical sector size as 512 and have physical sector size as 4K. This mode is important as it continues to work with legacy OS and application and provide large capacity drives. Main disadwantage with these drives is that they will trigger a RMW for storage I/O smaller than 4K. This RMW happens in drive firmware and may have some performance impact in cases where large # of storage IO are smaller than 4K


  1. 4KN Drives – these drives expose logical sector size and physical sector size as 4K. This drives can not work with legacy OS and application. Whole of the stack from vm guest OS to ESXi to Storage has to be 4KN

Lets now  look at a few advantages of 4k drives.

  • 4K drives require less space for error correction codes than regular 512-byte sector drives . This result in greater data density on 4k drives which provides a better TCO(total cost of ownership) ,
  • 4K drives have a larger ECC field for error correction codes and so inherently provide better data integrity,
  • 4k drives are expected to have better performance than the current 512n drives. However this is only true when the guest OS has been configured to issue I/Os aligned to the 4K sector size.



  • VMFS 5 does not support 4k drives even in emulation mode.If a 512e drives is formatted with VMFS-5 it is still recognized but this configuration is not supported by Vmware,
  • VMFS-6 is designed from the ground up to support AF drives in 512e mode,
  • VMFS-6 metadata is designed to be in alignment with the 4k sector size,
  • 512e drives can only be used with VMFS-6.
Virtual SAN Storage performance tests in action

Virtual SAN Storage performance tests in action

Virtual SAN provides the Storage Performance Proactive Tests which lets you to check parameters of your environment in an easy way. You just need few clicks to run a test.

Well, we can see a lot of tests from nested labs in the Internet, however there are not so many real examples.

I decided to share some results of such a real environment which consists 5 ESXi hosts. Each is equipped with 1x SSD Disk 1,92 TB and 5x HDD 4 TB.

It’s almost 10% of space reserved for cache on SSD. VMware claims that the minimum is 10% so it shouldn’t be so bad.

Since Virtual SAN 6.2 released with vSphere 6.0 Update 2, VMware makes the testing of performance much more easy. That’s possible thanks to Storage Performance Proactive Tests which lets you to check parameters of your environment in an easy way. They are available from Web Client, you just need few clicks to run a test. Perhaps those aren’t the most sophisticated whilst they are really easy to use.

To start some tests, simply move to Monitor>Virtual SAN>Proactive Tests tab on VSAN cluster level and click at run test button (green triangle).

As you quickly realise there are few kind of tests:

  • Stress Test
  • Low Stress test
  • Basic Sainity Test, focus on Flash cache layer
  • Performance characterization – 100% Read, optimal RC usage
  • Performance characterization – 100% Write, optimal WB usage
  • Performance characterization – 100% Read, optimal RC usage after warm-up
  • Performance characterization – 70/30 read/write mix, realistic, optimal flash cache usage
  • Performance characterization – 70/30 read/write mix, realistic, High I/O Size, optimal flash cache usage
  • Performance characterization – 100% read, Low RC hit rate / All-Flash Demo
  • Performance characterization – 100% streaming reads
  • Performance characterization – 100% streaming writes


Let’s start from a multicast performance test of our network. In case the received bandwidth is below 75 MB/s the rest would fail.


Caution!!! VMware doesn’t recommend using this tests  on production environments especially during business hours.

Test number 1 – Low stress test, the duration set to 5 minutes.


As we can see the IOPS counts around 40K for my cluster of five hosts. The average throughput around 30 MB/s, gives ca. 155 MB/s total.

Test number 2 – Stress test, duration set to 5 minutes.


Here we can see that my VSAN reached about 37K of IOPS and almost 260 MB /s of throughput.

Test number 3 – Basic Sanity test, focus on Flash cache layer, duration set to 5 minutes


Test number 4 – 100% read, optimal RC usage, duration set to 5 minutes.


Here we can see how does the SSD performs while most of the reads

Test number 5 –  100% read, optimal RC usage after warm-up


Test number 6 – 100 % write, optimal WB usage




If you have any other real results of your VSAN I’d be glad to see it and compare different configurations.