Browsed by
Month: November 2016

vSphere 6.5 – VMFS6 & 512e HDD support

vSphere 6.5 – VMFS6 & 512e HDD support

vSphere 6.5 introduces a new VMFS 6 – but why we need new version You ask? –answer: to support new hdd type, and this  point  us to current storage market situation . Well because with  512bytes sector size HDD’s  vendors are hitting drive capacity limits. They can not go beyond a certain size without compromising the resilience and reliability (not the best option in case of our data).             To provide large capacity drives, Storage Industry is moving forward to Advance format (AF) drives. These drives use large physical sector size of 4096 bytes.


So how does it help? With new AF (4K sector size) format, Disk drive vendors can create more reliable and large capacity HDD to support the growing storage needs. These drives are more cost effective as they provide better $/GB ratio.

Two kinds of 4k drives:

  1. 512 Emulation (512e) mode – these are 4KN drives but expose logical sector size as 512 and have physical sector size as 4K. This mode is important as it continues to work with legacy OS and application and provide large capacity drives. Main disadwantage with these drives is that they will trigger a RMW for storage I/O smaller than 4K. This RMW happens in drive firmware and may have some performance impact in cases where large # of storage IO are smaller than 4K


  1. 4KN Drives – these drives expose logical sector size and physical sector size as 4K. This drives can not work with legacy OS and application. Whole of the stack from vm guest OS to ESXi to Storage has to be 4KN

Lets now  look at a few advantages of 4k drives.

  • 4K drives require less space for error correction codes than regular 512-byte sector drives . This result in greater data density on 4k drives which provides a better TCO(total cost of ownership) ,
  • 4K drives have a larger ECC field for error correction codes and so inherently provide better data integrity,
  • 4k drives are expected to have better performance than the current 512n drives. However this is only true when the guest OS has been configured to issue I/Os aligned to the 4K sector size.



  • VMFS 5 does not support 4k drives even in emulation mode.If a 512e drives is formatted with VMFS-5 it is still recognized but this configuration is not supported by Vmware,
  • VMFS-6 is designed from the ground up to support AF drives in 512e mode,
  • VMFS-6 metadata is designed to be in alignment with the 4k sector size,
  • 512e drives can only be used with VMFS-6.
vSphere 6.5 – Secure Boot (ESXi and VMs)

vSphere 6.5 – Secure Boot (ESXi and VMs)

When new feature Secure Boot is enabled, the UEFI firmware validates the digitally signed kernel of an operating system against the digital certificate stored in the UEF firmware. For ESXi 6.5 this capability is further leveraged by the ESXi kernel, adding cryptographic assurance of ESXi components.


ESXi is already made up of digitally signed packages, called VIB’s. (vSphere Installation Bundle) These packages are never broken open. At boot time the ESXi file system maps to the content of those packages. By leveraging the same digital certificate in the host UEFI firmware used to validate the signed ESXi kernel the  kernel will then validate each VIB using the Secure Boot Verifier against the firmware-based certificate, ensuring a cryptographically “clean” boot.


ESXi Secure Boot operations

  • Installation of un-signed VIB’s/code will be prevented if SecureBoot is enabled.
  • Installation of un-signed VIB’s can only be done if SecureBoot is disabled in the BIOS
  • Enabling SecureBoot after un-signed VIB installation will cause a PSOD at boot time
  • If you are running unsigned drivers you cannot use SecureBoot
  • VIB Certificate Chaining

Virtual Machine Secure Boot

Enabling Virtual Machine Secure Boot is as simple as just checking the box in the UI.


Virtual machines must be boot from the EFI firmware to enable Secure Boot. There is support for Windows, Linux and nested ESXi in the EFI firmware. In order for Secure Boot to work, the Guest OS must also support Secure Boot.  Some examples are Windows 8 and Server 2012 and newer, VMware Photon OS, RHEL/Centos 7.0, Ubuntu 14.04 and ESXi 6.5.  For virtual machines, enabling Secure Boot requires that the VM is running with “EFI” firmware. Note that you can’t just change the firmware for some OS’s. When using EFI firmware, the Secure Boot checkbox becomes enabled.

vSphere 6.5 Installation – Part 4 – vCenter Server Appliance Upgrade

vSphere 6.5 Installation – Part 4 – vCenter Server Appliance Upgrade

In the previous parts I described the process of instalation new components of vSphere 6.5. In this article I’ll show you how the upgrade process looks like. It’s pretty straightforwar providing your DNS is working correctly 😉

During the upgrade process new appliance is deployed with temporary IP address to exchange the original appliance in the final step. This approach gives you the easy way to rollback, because you can turn off the new appliance and power on the original one at any stage.

1. The first step is to download the vCSA ISO image and mount it for instance to your desktop. Inside the ISO image go to D:\vcsa-ui-installer\win32 and run the installer.exe. (This is the same ISO image used for PSC deployment). This time we choose the Upgrade option.


2. As other types of installation In vSphere 6.5 the process of upgrade is divided into two phases, or more correctly stages. The first stage contains the deployment of the new appliance where it is deployed in our infrastructure. The second stage will let you configure the PSC and creating SSO domain, etc. It’s nothing to do on this step, simply click Next.

2migracja3. Accept the End User License Agreement and click Next.


4. Now you have to specify the information about source appliance (the original one which you are going to upgrade) and also the FQDN and credentials of the system where this appliance resides on. It could be either ESXi host or another vCenter server (e.g. Management’s cluster vCenter). Make sure you typed it correctly and click Next.


5. Verify and confirm the Certificate warning.


6. Specify the destination host/vCenter where the new appliance will be deployed. Click Next.


7. In case you chose the vCenter to connect, you can indicate a specific folder for the appliance. Click Next.


8. Additionally you can choose the specific host. 


9. Provide the virtual machine name for new appliance and root password.


10. Choose the right size of your deployment. VMware was so nice and inclueded information on the screen to let you choose the right size for your environment. In my case it’s “just” the lab so Thiny version will be more than enough. This is the time when you can easily expand the appliance limit if you need so. Click Next to go further.


11. Choose the datastore you want to use. There is also an option to configure the appliance this in thin provisioning mode. If you prefer so, simply mark the checkbox “Enable Thin Disk Mode” and click Next.


12. This steps requires the network configuration. You are supposed to provide port group from the available list of port groups configured on ESXi host/vCenter you choosed before. Furthermore you need to provide a temporary IP address which will be used untill the configuration from oryginal appliance is transfered.12migracja

13. Last step in stage one – confirmation. If all settings are correct click Finish to start the deployment process.


14. When the deployment stage is done you go pass to stage 2 – the real migration of your vCenter Server Appliance data and configuration. To do that, click Continue.

15. Again you will see the welcome screean which informs you that this is the beginning of stage 2. Click Next to start.


16. Here you will see the summary of information you provided in previous stage. 17migracja

Caution!! From this panel you can not change these settings at this step. If you made a mistake before and you an error that source host is not reachable on specific address you can close this installer and access it at https://IP_of_temporary-VCSA:5480 and there you can change these settings and continue the installation.


17. Here you can choose which ought to be migrated. In my case I want to perform complete upgrade, preserving all data, tah’s why I’m choosing the last option – COnfiguration, events, tasks and performance metrics. You have to notice that this option will need to migrate the most amount of data.

18. Untick the participation in VMware CEIP program and click Next.


19. Review the summary and click Finish.



After few minutes (depending on the amount of data that have to be replicated) your new vCenter Appliance will be up and ready.


Please, see also the other parts of the series:

vSphere 6.5 – Encrypted vMotion

vSphere 6.5 – Encrypted vMotion

Another  good news is that You can encrypt the vMotion of any VM, encrypted or not – encrypted VM’s will always use encrypted vMotion :


Disabled – do not use encrypted vMotion

Opportunistic – use encrypted vMotion if source and destination hosts support it.

Required -Allow only encrypted vMotion.

Note !!!  Mixed cluster and you have a requirement of encrypted vMotion, then setting to “Required” will not let you vMotion to a host that doesn’t support it. (only vSphee ESXi 6.5 )

VMware add new vmkcypto framework subsystem to vmkernel. It is used by Virtual Mchine Encryption and vMotion for cryptographic operations :



Now let’s look at new vMotion process:

  1. As part of that, a 256bit random key and 64-bit Nonce is generated. The Nonce used to generate a unique counter for every packet sent over the network. This prevents replay
  2. The key and the Nonce are packaged into a vMotion Migration Specification is created for the vMotion. This spec is sent to both systems in a the cluster.
  3. The vMotion traffic begins with every packet being encrypted on Host A and the counter incrementing.
  4. The packets are decrypted on the receiving host and the vMotion completes
vSphere 6.5 – Network-aware DRS

vSphere 6.5 – Network-aware DRS

VMware Distributed Resource Scheduler is a well known VMware feature which is one of the most helpful escpecially in bigger environments. It’s used to balance the load (CPU and Memory) between ESXi hosts in cluster. However, in previous releases it has an imperfection.

Let’s imagine a fallowing situation shown below:


Assume you have three host in the cluster with 6 VM’s powered on. If you power on  another VM it will be placed on the first host by DRS.

Although host 1 has saturated it’s network in 100% but the VMs running on it are not consuming a large amount of CPU/Memory the next VM will be placed on it. That will cause even bigger network troubles.

Fortunately in vSphere 6.5 DRS will help us in avoiding such situations. That’s due to new feature called Network-aware DRS, which are using the new DRS algorithm. It will now consider network bandwidth when making placement recommendations.  It will calculate the Tx and Rx of the connected physical uplinks and avoid placing new VMs on hosts that are over 80% utilized.  This is an additional placement consideration after all other placement decisions are made.


Caution! DRS will not reactively balance the hosts based on network utilization.  Perhaps in future releases it will ?


To sum up – Network-aware DRS:

  • Adds network bandwidth considerations by calculating host network saturation (Tx & Rx of connected physical uplinks)
  • Avoids a over-subscribing a host network links, although not guaranteed. Best effort approach.  CPU & MEM performance is still priorities over network.
vSphere 6.5 – Backup and Restore encrypted VMs

vSphere 6.5 – Backup and Restore encrypted VMs

New encryption gives many possibilities but also make some impact to other tasks in our environment. Let’s consider backup implications – backup and restore of encrypted disks is possible with NBD and HotAdd transport, but SAN mode does not support encrypted virtual machine backup. No API change is involved – ESXi hosts encrypt by attaching an IO Filter. To back up encrypted virtual machines using HotAdd, the backup proxy must have been encrypted as well. The backup process requires “Cryptographer.DirectAccess” permission. Data on backup media will be not encrypted!



  • SAN Mode backups not supported (SAN has no visibility in to encrypted content)
  • No API changes to Backup products
  • When using HotAdd the Backup Proxy VM must be encrypted
  • Backup User must have “DirectAccess” permission
  • Backup data is not backed up encrypted
  • Not supported with VM Encryption
    • Suspend/Resume
    • Encrypting a VM with pre-existing snapshots
    • vSphere Replication
    • Serial/Parallel port
    • Content Library
  • Don’t encrypt your vCenter or PSC VM’s  -> Because You need vCenter to get the keys!!!
What’s New in vSphere 6.5 – ProactiveHA

What’s New in vSphere 6.5 – ProactiveHA

Proactive HA is a new feature Available in vSphere 6.5 released recently. It’s a kind of feature which will even better help you to protect you environment in case of hardware failure.

Currently all of the hardware components are redundant including power supplies, fans, network cards etc. However the most possible cause of whole server failure occurs while one of these theoretically redundant components fails. To better imagine that let’s think about power supply fail. There is still the second one but during there is only one it is much more loaded. (Similar things you can observe with hard disks in a RAID group – the biggest possibility of a disk fail is during RAID re-building).

ProactiveHA will help you protect the environment in such situations. It will detect hardware conditions of a host and allow you to evacuate the VMs before the trivial issue causes the serious outage.  For this feature to function, the hardware vendor must participate.  Their hardware monitoring solution will advertise the health of the hardware, and vCenter will query that system to get a status of the hardware components such as the fans, memory, and power supplies.  vSphere can then be configured to respond according to the failure.


To let it functions there is a new ESXi host state in vSphere 6.5 – Quarantine mode. It’s similar to maintenance mode but it is not as severe as maintenance mode. It’s mean that DRS will attempt to evacuate all VMs from the host, but only if:

  • No performance impact on any virtual machine in the cluster
  • None of the business rules is disregarded
  • Additionally, any soft affinity or-anti-affinity rules will not be overridden by the evacuation. However, DRS will seek to avoid placing any new VMs.

To set the Proactive HA features, find the Partial Failures and Responses section and set how vSphere should respond to partial failures.  The options are to place a degraded host into Quarantine Mode, Maintenance Mode, or Mixed Mode.

Mixed mode means that for moderate degradation, the host will be placed into Quarantine Mode.  For Severe failures, it will be placed into Maintenance Mode.


For the moment of writing and availibility of vSphere 6.5 GA the supported failure condition types are:

  • Memory
  • Power
  • Fan
  • Network
  • Storage
vSphere 6.5 – VM Encryption

vSphere 6.5 – VM Encryption


Next new security  functionality in vSphere 6.5 – encryption is implemented via Storage Policies. If You add to the vm an encryption storage policy it will encrypt the disk.


Key features:

  • No modification within the Guest.
  • VM Agnostic
    • Guest OS
    • DataStore
    • HW Version
    • Policy driven
  • Encrypts both VMDK and VM files
  • No access to encryption keys by the Guest
  • Full support of vMotion

Diagram below shows how it works:



  1. Register a VM on a host and configure the (new or existing ) VM with Encryption Enabled storage policy and KMIP server
  2. vCenter gets a key from the KMIP server. That key is used to encrypt the VM files and the VM Disks.
  3. VC loads the key into the ESXi hosts. All hosts that don’t have the key will get the key to support DRS/HA.
  4. Once the key is loaded into the KeyCache on the ESXi host, encryption and decryption of the disk will happen at the IO Filter (introduced in 6.0 U1) level.

But let’s ask who can manage vm encryption ?

… Security Administrators will manage KMS and keys only “subset” of vSphere Admins will / should  manage encryption within vSphere. We have new default role “No Cryptography Administrator” , additional we got new vCenter crypto priviledges like : Encrypt, Decrypt, Manage Keys , Clone. So we basically can delegate encryption priviledges to varius admins via custom roles in the way that we well know from previous environment editions – below example :


Lets see how to add in our vCenter KMS configuration – it is straight forward You just  need to find new tab  at web client and add new connection :



… and finally examples of supported KMS servers (below is not a full list)

Note !!!

most  KMIP 1.1 compliant key managers get the approval – but as usual verify with VMware interoperability matrix to have 100% sure


vSphere 6.5 – enhanced logging

vSphere 6.5 – enhanced logging


vSphere 6.5 introduces audit logging, before vSphere 6.5  logs were more focused on finding root causes of a problem – not releate deep  to IT operations or security use cases. For example, if a virtual machine was reconfigured from one storage adapter to another in logs we would find only “Virtual Machine <name> reconfigured”.

 But now logs which are coming from vCenter via Syslog will contain data from vCenter Events. These logs will clearly show “Before” and “After” setting changes.  This enhances the ability of IT and Security administrators to troubleshoot issues by providing information what was exactly changed in the vSphere environment.


Enhanced logging summary:

  • Improved vCenter/ESXi event logs quality
    • Informative auditing without having to enable verbose mode
  • Structured vCenter Events SysLog Stream
    • Minimal VC overhead
    • Simplified deployment
    • Enables upper level intelligence
  • Customer auditing examples:
    • VM was moved to a wrong network
    • VM disk was deleted by accident
    • VM was under/over provisioned

Now let’s see how to enable streaming VC events to remote syslog server :



NOTE!!! This feature is not available on Windows VC

1. Enable event syslog:


2. Configure connection parameters:


And finally let’s look at some examples of vCenter events audit quality:




vSphere 6.5 Installation – Part 3 – vCenter Server on Windows

vSphere 6.5 Installation – Part 3 – vCenter Server on Windows

This is the third part of the series in which I’m describing the process of installation vSphere 6.5 components. In this part I’ll go throught the process of installation vCenter Server on MS Windows Server 2012 R2.

The others parts are:

Now let’s check how the Windows installation looks like in the latest vSphere version.


  •  ISO for vCenter Server installation. Keep in mind that this is different ISO than used to PSC or vCSA deployment.
  • Account which can log in as a service (you can add it from Local Security Policy -> Local Policies -> User Rights Assignment -> Log on as a service)
  • (Optional) External Database with correctly configured ODBC.

For this demonstration I’m using embedded Postgress SQL instance which have the same limit as an external database and In my opinion in most cases it’s the best option.



1. The first step you have to do after you download the ISO image is to start the autorun. Then you choose the vCenter Serwer for Windows option and click Install button. You will also notice that there is no option to install vSphere Client for Windows from the installer page. However in vSphere 6.0 you were able to find the installer on the ISO in vSphere-Client folder. Unfortunatelly in vSphere 6.5 there is no more C# client for windows…For most of you (including me) it’s really bad news, althought there is the light at the end of the tunnel – new HTML5 client 🙂 It’s completely new and different user interface and I’ll devote the whole part of the series to show closer how it looks like.


2. On the welcome screan simply click Next.


3. Accept the End User License Agreement and click Next.


4. Choose the deployment type page allows you to choose which component should be deployed. In my case the external PSC was deployed in previous step and now I’m choosing vCenter Server. It’s important to notice that external PSC must be deployed before you start the deployment of vCenter Server Appliance.


5. Define the System Name. Notice that the FQDN you specify is already present and configured in your DNS. Click Next.


6. This is the step when you have to connect with your PSC and SSO domain. Provide the PSC FQDN name, SSO domain name, password for Administrator, and click Next.


7. Choose the account you would like to use for the installation. You could use the current account you are logged in or specify an other service account. Click Next.


8. This is the step where you can choose the database type. As I mentioned before in most cases I recommend to use embedded PostgreSQL. However you could also specify the external one. Click Next.


9. Confirm or specify some unusual port addresses you are going to use for specific services used by vCenter. Click Next.

10. Specify the installation directory or leave the default values and click Next.


11. Review the summary page and click Install to begin the installation process.


12.Monitor the installation progress.


13. After the installation is completed you can Launche vSphere Web Client and begin the real configuration of your management server.



If you want to try new HTML5 client you can type the https://vC_FQDN_IP in your browser and choose HTML5 option or go straight to https://vC_FQDN/ui

As you can notice on the vC welcome screan the HTML5 client is not fully functional yet. In the next part of the series I’ll go throught the new client to show you how it looks and how to navigate so please stay tuned 🙂