Browsed by
Tag: vsphere

Esxi Net.ReversePathFwdCheckPromisc Advanced setting

Esxi Net.ReversePathFwdCheckPromisc Advanced setting

During deployment of Cisco proxy appliance, we discovered a problem. According to cisco to resolve this problem qa“Net.ReversePathFwdCheckPromisc” should be set to “1” on ESX’s.

The question is – do you know any negative effects which such change could cause. We believe that there must be a reason why by default this option is set to 0 ? That’s why I decided to figure our what it is used for.

After some research I was able to find answer:

Setting – > Net.ReversePathFwdCheckPromisc = 1 — > this is when you are expecting the reverse filters to filter the mirrored packets, to prevent multicast packets getting duplicated.

Note: If the value of the Net.ReversePathFwdCheckPromisc configuration option is changed when the ESXi instance is running, you need to enable or re-enable the promiscuous mode for the change in the configuration to take effect.

The reason you would use promiscuous mode depends on the requirement and configuration. Please check the below KB Article:

  • This option is not enabled by default because we are not aware of the vSwitch configuration and can’t predict what it could be as it has configurable options.

VMware does not advise to enable this option if we do not have a use case scenario with teamed uplinks and have monitoring software running on the VMs ideally. As When promiscuous mode is enabled at the port group level, objects defined within that port group have the option of receiving all incoming traffic on the vSwitch. Interfaces and virtual machines within the port group will be able to see all traffic passing on the vSwitch causing VM performance impact.

Should the ESX server be rebooted for this change to take effect:  answer is – > Yes, and Yes you can enable this option with the VMs running on the existing portgroup.

Do you have any interesting virtualization related question?


SAP application on vSphere platform

SAP application on vSphere platform

This is a mini article to start our Q&A set, a set of not easy to find answer real life questions 😉 qa
Recently I received a question-related to advanced settings SAP app on vSphere platform:
“One of our customer ask us to set the following option to their virtual system: Misc.GuestLibAllowHostInfo This is according to SAP note: 1606643 where SAP requires reconfigure virtual system default configuration. I can’t find details information, which host data would be exposed to virtual system. Could you please point me to documentation or describe which information is being transferred from HOST to virtual systems?“

  • After some research I was able to find answer :

“Misc.GuestLibAllowHostInfo” and “tools.guestlib.enableHostInfo” these configurations if enabled allow the guest OS to access some of the ESXi host configurations, mainly performance metrics e.g. how many CPU cores the host has, their utilization and contention etc. There is no confidential information from other customers which would be visible, however, it may give the user of those SAP VMs access to performance/resource information which you may not want to share.

The following document outlines the effect of the changes as I have described above.

I believe the “might use the information to perform further attacks on the host” could only apply to other vulnerabilities which may exist for the particular hardware information that the guestOS can gather from the ESXi host.
Other than that I am not sure there is any other concern to worry about.

Do you have any interesting virtualization related question?

VMware vSphere tags limit – is it known ?

VMware vSphere tags limit – is it known ?

Recently I received quite interesting question – what is the supported maximum quantity  for tags in vCenter 6.0U2 ?

Malignant author of the question is a good friend of mine and VMware administrator in one person. He ssked about tags limit because he want to use them to provide more information about each of its production VM’s – roughly speaking need to create about 20000 tags.

I thought ok., give me couple seconds to verify this,  and looked fast in vmware configuration maxims …. couple minuntes later it was clear that this is not a easy question 😉

Furthermore after some additional research (no clear statement in official documentation)  we decide to perform tests in lab environment !

We used simple powercli script to create 20000 tags in test vcenter appliance (6.0U2) , below our script:

$i -le 20000
New-Tag -Name $i -Description $i -Category test

Script worked like a charm without any issue – so far so good :), but when we tried to assign one tag to first vm we encounter web client error 1009  – very strange!

We decided to perform additional test and find out that limit is below 10000.  At this stage we decide to clear this issue with Vmware support and after some time received wery interesting feedback:

  1. NGC has upper bound of retrieve 10000 objects max.
  2. If the tags are less than 10000 then data service timeouts after 120 seconds(default dataservice timeout is 120 seconds).
  3. Decreasing the count to 9994 tags and increasing dataservice timeout, shows up all the tags(Assign) now.

As a temporary workaround for now.
1. Have total created tags less than 10000.
2. Increase data service timeout to 600 seconds(10 min).

VMware GSS stands that engineering working now to remove tag limit boundary in next releases vSphere 6.x.

VMware Auto Deploy Configuration in vSphere 6.5

VMware Auto Deploy Configuration in vSphere 6.5




The architecture of auto deploy has changed in vSphere 6.5, one of the main difference is the ImageBuilder build in vCenter and the fact that you can create image profiles through the GUI instead of PowerCLI. That is really good news for those how is not keen on PowerCLI. But let’s go throgh the new configuration process of Auto Deploy. Below I gathered all the necessary steps to configure Auto Deploy in your environment.

  1. Enable Auto Deploy services on vCenter Server. Move to Administration -> System Configuration -> Related Objects, look for and start fallowing services:
  • Auto Deploy
  • ImageBuilder Service

You can change the startup type to start them with the vCenter server automatically as well.

Caution! In case you do not see any services like on the screan below, probably vmonapi and vmware-sca services are stopped.ad1

To start them, log in to vCenter Server through SSH and use fallowing commands:

#service-control  – -status         // to verify the status of these services

#service-control  – -start vmonapi vmware-sca       //to start services


Next, go back to Web Client and refresh the page.


  1. Prepare the DHCP server and configure DHCP scope including default gateway. A Dynamic Host Configuration Protocol (DHCP) scope is the consecutive range of possible IP addresses that the DHCP server can lease to clients on a subnet. Scopes typically define a single physical subnet on your network to which DHCP services are offered. Scopes are the primary way for the DHCP server to manage distribution and assignment of IP addresses and any related configuration parameters to DHCP clients on the network.

When basic DHCP scope settings are ready, you need to configure additional options:

  • Option 066 – with the Boot Server Host Name
  • Option 067 – with the Bootfile Name (it is a file name observed at Auto Deploy Configuration tab on vCenter Server – kpxe.vmw-hardwired)


  1. Configure TFTP server. For lab purposes I nearly always using the SolarWinds TFTP server, it is very easy to manage. You need to copy the TFTP Boot Zip files available at Auto Deploy Configuration page observed in step 2 to TFTP server file folder and start the TFTP service.


At this stage when you are try to boot you fresh server should get the IP Address and connect to TFTP server. In the  Discovered Hosts tab of Auto Deploy Configuration you will be able to see these host which received IP addresses and some information from TFTP server, but no Deploy Rule has been assigned to them.


  1. Create an Image Profile.

Go to Auto Deploy Configuration page -> Software Depots tab  and Import Software Depot



Click on Image Profiles so see the Image Profiles that are defined in this Software Depot.


The ESXi software depot contains the image profiles and software packages (VIBs) that are used to run ESXi. An image profile is a list of VIBs.


Image profiles define the set of VIBs to boot ESXi hosts with. VMware and VMware partners make image profiles and VIBs available in public depots. Use the Image Builder PowerCLI to  examine the depot and the Auto Deploy rule engine to specify which image profile to assign to which host. VMware customers can create a custom image profile based on the public image profiles and VIBs in the depot and apply that image profile to the host.


  1. Add Software Depot.

Click on Add Software Depot icon and add custom depot.


Next point in the newly created custom software depot select Image Profiles and click  New Image Profile.


I selected the minimum required VIBs to boot ESXi host which are:

  • esx-base 6.5.0-0.0.4073352 VMware ESXi is a thin hypervisor integrated into server hardware.
  • misc-drivers 6.5.0-0.0.4073352 This package contains miscellaneous vmklinux drivers
  • net-vmxnet3 VMware vmxnet3
  • scsi-mptspi LSI Logic Fusion MPT SPI driver
  • shim-vmklinux-9-2-2-0 6.5.0-0.0.4073352 Package for driver vmklinux_9_2_2_0
  • shim-vmklinux-9-2-3-0 6.5.0-0.0.4073352 Package for driver vmklinux_9_2_3_0
  • vmkplexer-vmkplexer 6.5.0-0.0.4073352 Package for driver vmkplexer
  • vsan 6.5.0-0.0.4073352 VSAN for ESXi.
  • vsanhealth 6.5.0-0.0.4073352 VSAN Health for ESXi.
  • ehci-ehci-hcd 1.0-3vmw.650.0.0.4073352 USB 2.0 ehci host driver
  • xhci-xhci 1.0-3vmw.650.0.0.4073352 USB 3.0 xhci host driver
  • usbcore-usb 1.0-3vmw.650.0.0.4073352 USB core driver
  • vmkusb 0.1-1vmw.650.0.0.4073352 USB Native Driver for VMware

But the list could be different for you.



  1. Create a Deploy Rule.






  1. Activate Deploy Rule


  1. That’s it, now you can restart you host, it should boot and install according to your configuration now.
Adding a sound card to ESXi hosted VM

Adding a sound card to ESXi hosted VM

Sound Card in vSphere Virtual Machine is an unsupported configuration. This is feature dedicated to Virtual Machines created in VMware Workstation. However, you can still add HD Audio device to vSphere Virtual Machine by manually editing .vmx file. I have tested it in our lab environment and it works just fine.

Below  procedure how to do this:

1. Verify storage where VM with no soundcard reside


  1. Login with root to the ESXi host where VM reside using SSH.
    3. Navigate to /vmfs/volumes/<VM LUN>/<VM folder>
    In my example it was:
    ~# cd /vmfs/volumes/Local_03esx-mgmt_b/V11_GSS_DO
    4. Shut down problematic VM
    5. Edit .vmx file using VI editor.

Make a backup copy of the .vmx file. If your edits break the virtual machine, you can roll back to the original version of the file.
More information about editing files on ESXi host, refer to KB article:

  1. Once you have open vmx to edit, navigate to the bottom of the file and add following lines to the .vmx configuration file:
    sound.present = “true”
    sound.allowGuestConnectionControl = “false”
    sound.virtualDev = “hdaudio”
    sound.fileName = “-1”
    sound.autodetect = “true”
  2. Save file and Power-On Virtual machine.
  3. Once it have booted, and you have enabled Windows Audio Service, sound will work fine.

If you go to “Edit Settings” of the VM, you can see information that device is unsupported. Please be aware that if after adding sound card to you virtual machine, you may exprience any kind of unexpected behavior (tip: in our lab env work this config without issues).

VCAP6-DCV Design exam experience(s)

VCAP6-DCV Design exam experience(s)

Finally, I’m pround to announce that VCIX6-DCV goal is achived!

Previously I passed the Deploy Exam (you can read about it in this post) which for me personally was far more intuitive and effortless. If you are a practitioner person than visioner and designer it would be quite tought to get used to these kind of questions and reasoning. In my opinion there are a few points which I can not agree with and I would be glad to discuss with the authors of these questions about their points of view 🙂

However, as I read on one of the blogs this is a VMware exam and they could have their own point of view and opinion about best practicies in designing virtual environments.

As you realized I used plural in word experience – it’s not so hard to guess why. Yes, I had to take the exam twice. Nevertheless, I finished the first try quite satisfied and full of hope the reality was brutal. 243 points appeared not to be enought to pass it…That was a food for thoughts.

That made me aware that I need to prepare better and figure out about the key used in design quiestions. It’s not exacly the key but the way of designs constructions. As usually Internet was priceless. First of all I found tips that the exam is similar to VCAP5 version and fallowing this idea I read the VCAP5-DCD Official Cert Guide. This was quite useful. Then I tried to think about the design questions I met and gind out what could be wrong there.

After a few more white papers, blog articles and other readings I took the second try and happily this the reult was much more better and of course I finally managed to pass and gain complete VCIX title.

The few tips from me:

  1. Be fresh and rested at the exam day ( there are 205 minutes, it’s quite a long to sit in front of the screen).
  2. Stay focused and read carefully all the questions and instructions at least twice.
  3. Start from the design questions which would take you a little bit more time.
  4. Be prepared.

Materials I found usefull during preparation time:

  1. VCAP6-Design Blueprint and all associated documents especially those from objective 1.2 and 1.3 should be read more than once
  2. VCAP5-DCD Official Cert Guide
  3. Study Guides of other people
  4. Google+ VCAP-DCD Study Group

I also recommend to get yourself familiar with scoring methodology described at The Cloud JAR’s Blog




VirtualVillage’s home LAB

VirtualVillage’s home LAB

It is possible to learn especially about VMware products using just books, official trainings, blogs, etc. However, we believe that the real knowledge is available only by practice and not all could be tested or verified using production environments 🙂

And again, you can test a lot just using Workstation on your notebook (providing it is powerful enough) but these days there are more and more virtual infrastructure component which requires a lot of resources. Furthermore, having real servers and storage array is also a little bit different than deploying a few small virtual machines running on a notebook.

That is why a few years ago we decided to join forces and build the real laboratory where we are able to test even the most sophisticated  deployments not only with VMware products without being constraint by the resources.

The main hardware components of our lab infrastructure are included in the table below.

Hardware Component Quantity Details Purpose
ServerFujitsu TX200 S7 2 2x CPU E5-4220, 128 GB RAM Payload Cluster
Server Fujitsu TX100 S1 2 Router/Firewall and Backup
Server Fujitsu TX100 S3 3 1x CPU E3-1240, 32 GB RAM Management Cluster
NAS Synology DS2413+ 1 12 x 1 TB SATA 7,2K Gold Storage
NAS Synology RS3617+ 1 12 x 600 GB SAS 15K Silver Storage
NAS QNAP T410 1 4 x 1TB SATA 5,4K Bronze Storage (ISO)
Switch HPE 1910 1 48x 1 Gbps Connectivity


Of course we didn’t buy it at once. The environment evaluates with increasing needs. ( In the near future we are going to expand management cluster with 4 host and deploy NSX).

The logical topology looks like this:



Despite the fact that most of our servers use tower cases, we installed them in a self made 42U Rack. Unfortunatelly, especially during the summer it could not go without air conditoning (this is one of the most power consuming part of the lab..)


Later, either me or Daniel will describe the software layer of our Lab. I hope, it will give an inspiration to anyone who is thinking about own lab.


VMware PowerCLI – Introduction

VMware PowerCLI – Introduction

To begin the jurney with PowerCLI we need to start from the installation of PowerCLI itself.

The installation can be done on a Windows based system, that could be some kind of an administration server. The installation files can be found on this VMware site.

There are a few versions available, they are released asynchronously with vSphere and the version numbers do not exactly correspond to vSphere versions. The most recent version is 6.5 whilst there are other like 6.3, 6.0 or 5.8 available.

Before you install the PowerCLI I recommend to change the Execution Policy of Powershell. It is required to run scripts. To do it, run Windows PowerShell as administrator and execute fallowing command:

Set-ExecutionPolicy RemoteSigned

The installation process is really straightforward, that’s why I will not spam the screanshoots of installations here.

After you finish the installation you can run it and see the first Welcome screen like this:



The first command I suggest to use is:


it lists all the available commands. However to display any information about virtual infrastructure you need to connect to a vCenter server or ESXi host. We will do it in the next part after introduction of useful tools which can be used in conjunction with PowerCLI.

How to monitor virtual network – story about netflow in vSphere environment.

How to monitor virtual network – story about netflow in vSphere environment.

Before we start talking about NetFlow configuration on VMware vSphere let’s back to basics and review protocol itself. NetFlow was originally developed by Cisco and has become a reasonably standard mechanism to perform network analysis. NetFlow collect network traffic statistics on designated interfaces. Commonly used in the physical world to help gain visibility into traffic and understanding just who is sending what and to where.

NetFlow comes in a variety of versions, from v1 to v10. VMware uses the IPFIX version

of NetFlow, which is version 10. Each NetFlow monitoring environment need to have exporter ( device carrying  netflow flow’s) , collector (main component ) and of course some network to monitor and analyze 😉

Below You can see basic environment diagram:


We can describe flow as tcp/ip packets sequence (without direction) that have common:

  • Input interface
  • Source IP
  • Destination IP
  • TCP/IP Protocol
  • Source Port (TCP/UDP)
  • Destination Port (TCP/UDP)
  • ToS IP

Note. vSphere 5.0 uses NetFlow version 5, while vSphere 5.1 and beyond uses IPFIX (version 10).

Ok, we know that distributed virtual is needed to configure NetFlow on vSphere but what about main component NetFlow collector – as usual we have couple options that we can simply divide in commercial software with fancy graphical interfaces and open source staff for admins that still like old good cli 😉

Below I will show simple implementation steps describing examples from both approach :

Manage engine NetFlow analyzer v12.2, more about software on my lab VM setup:

  • Guest OS:Windows 2008R2
  • 4GB RAM
  • 2vCPU
  • 60 GB HDD
  • vNIC interface connected to ESXi management network

Installation (using embedded database just for demo purpose) is really simple and straight forward. Let’s start from starting the installer:



  1. accept license agreements


  1. choose installation folder on vm hdd


  1. choose installation component option – for this demo purpose we go with simple environment with only one collector server, central reporting is not necessary


  1. choose web server and collector services TCP/IP ports


  1. provide communication details – again in this demo we have all components on one server and we can simply go with localhost



  1. optional – configuration proxy server details


  1. select database – on this demo i used embedded Postgresql , but if You choose MS database remember about ODBC config.


  1. installation is quite fast – couple more minutes and solution will be ready and available to start work:



… Web client like in VMware need couple CPU cycles to start 😉


.. and finally we can see fancy ManageEngine NetFlow collector


II) Open-Source netdump tool  – nfdump is distributed under the BSD license, and can be downloaded at: my lab VM steup:

  • GOS: Debian 8.6
  • 4GB RAM
  • 2vCPU
  • 60 HDD
  • vNIC interface connected to ESXi management network


  1. We need to start from adding some sources to our debian distribution:


  1. CLI Installation nfdump packet:



  1. Run simple flow capture to verify if collector is running and creating output flow statictics files (you can see that i use same tcp port 9995 and folder on my desktop as output destination):



Ok, now it is time to back to vSphere and configure DVS to send network traffic to collector:



  • IP Address: This is the IP of the NetFlow Collector
  • Port: This is the port used by the NetFlow Collector.
  • Switch IP Address: This one can be confusing – by assigning an IP address of here, the NetFlow Collector will treat the VDS as one single entity. It does not need to be a valid, routable IP, but is merely used as an identifier.
  • Active flow export timeout in seconds: The amount of time that must pass before
  • the switch fragments the flow and ships it off to the collector.
  • Idle flow export timeout in seconds: Similar to the active flow timeout, but for flows
  • that have entered an idle state.
  • Sampling rate: This determines the interval packet to collect. By default, the value is 0,
  • meaning to collect all packets. If you set the value to something other than 0, it will
  • collect every X packet.
  • Process internal flows only: Enabling ensures that the only flows collected are ones that occur between VMs on the same host.

And enable it at designated port group level:


Finally we can create simple lab scenario and capture some ftp flow statistics between two vm’s on different ESXi :


VM’s are running in dedicated vlan on the same DVS port group, collector is running on management network to communicate with vCenter and ESXi hosts. I used ftp connection to generate traffic between vm’s below examples output from two collectors (test ran separate as collector share the same ip)  :


ftp client on first vm:


ftp server on second vm:


flow statistics example from netdump:


flow statistics from ManageEngine



Increase VMware ESXi iSCSI storage performance ? – lets demistyfy all tips and tricks

Increase VMware ESXi iSCSI storage performance ? – lets demistyfy all tips and tricks


Before we start I would like to describe main motivation to write this article which is quite simple – to gather in one place all basic theoretical background about iscsi protocol and best practices at its implementation on vSphere platform with special consideration about potential performance tuning tips & tricks . This is first part of the series where We (I’m counting on readers participation) try to gather and verify all this “magical” parameters often treated as myths by many Admins.

To begin let’s start from something boring but as usual necessary 😉 … theoretical background.

iSCSI is an network based storage standard that enable connectivity between iSCSI initiator (client) and target (storage device) over well known IP network. To explain this storage standard in very simple way we can say that SCSI packets are encapsulated in IP packet and sent over traditional TCP/IP network where targets and initiators can de-encapsulate TCP/IP datagrams to read SCSI commands. We have couple options in case of implementation this standard because TCP/IP network model components transporting SCSI commands can be realized at software and/or hardware layer.


Important iSCSI standard concepts and terminology:

  • Initiator – functions as an iSCSI client. An initiator typically serves the same purpose to a computer as a SCSI bus adapter would, except that, instead of physically cabling SCSI devices (like hard drives and tape changers), an iSCSI initiator sends SCSI commands over an IP network. Initiators can be divided into two broad types:
    • A software initiator implement iSCSI using code component that use existing network card to emulate SCSI device and communicate thru iSCSI protocol. Software initiators are available for most popular operating systems and are the simplest and best economic method of deploying iSCSI.
    • A hardware initiator based on dedicated hardware, typically use special firmware running on that hardware and implementing iSCSI above network adapter acting as HBA card in server. Hardware decrease CPU overhead of iSCSI and TCP/IP processing that is why it may improve the performance of servers thet use iSCSI protocol to communicate with storage devices.
  • Target – functions as resource located on an iSCSI server, most often dedicated network connected storage device (well known as storage array) that provide target as access gateway to its resources. But it may also be a “general-purpose” computer or even virtual machine – because as with initiators iSCSI target can be realized at software layer.
  • Logical unit number – in iscsi terms LUN stands for logical unit and is specified by unique number. A LUN is representation of an individual SCSI (logical) device that is provided /accessible thru target. After iscsi connection is establish (emulate connection to scsi hdd) initiators treat iSCSI LUNs as they would a raw SCSI or IDE hard drive. In many deployments LUN usually representing part of large RAID (Redundant Array of Independent Disksdisk) array, it leaves access to underlying filesystem – regarding of the operating system that use it.
  • Addressing – iSCSI uses TCP/IP pots (usual 860 and 3260) for the protocol to name objects use to address it with special names refer to both iSCSI initiators and targets. iSCSI provides name-formats:
    • iSCSI Qualified Name (IQN)
      • iqn -iSCSI qualified name
      • datethat the naming authority took ownership of the domain
      • reversed domain name of the authority
      • Optional “:” prefixing a storage target name specified by the naming authority.
    • Extended Unique Identifier (EUI)

Format: eui.{EUI-64 bit address} (eui.xxxxxxxxx)

  • T11 Network Address Authority (NAA)

Format: naa.{NAA 64 or 128 bit identifier} (naa.xxxxxxxxxxx)

Note : IQN format addresses occur most commonly.

  • iSNS – iSCSI initiators can locate appropriate storage resources using theInternet Storage Name Service (iSNS) protocol. iSNS provide provides iSCSI SANs with the same management model as dedicated Fibre Channel  In practice, administrators can implement many deployment goals for iSCSI without using iSNS.

iSCSI protocol is over IETF responsibility – to have more information please see RFC 3720, 3721, 3722, 3723, 3747, 3780,3783, 4018,4173,4544,4850,4939, 5046, 50475048,7143



And finally for those who dare to read all boring theory part – main dish: my performance“ tips and tricks” list to demystify in this blog series journey :

  1. iSCSI initiator (hardware or software) queue depth:

        //example for softoware iscsi initiator

#esxcfg-module -s iscsivmk_LunQDepth=64 iscsi_vmk

  1. Adjusting Round Robin IOPS limit :

        //example for max iops and bytes parameter

#esxcli storage nmp psp roundrobin deviceconfig set -t=iops -I=10 -d=naa.xxxxxxxxxxxx

#esxcli storage nmp psp roundrobin deviceconfig set -t=bytes -B 8972 -d=naa.xxxxxxxxxxx

     3. NIC/HBA Driver and firmware version on esxi hypervisor


    4. Using jumbo frames for iSCSI


    5. Controlling LUN queue depth throttling

//example based on kb:

#esxcli storage core device set –device naa.xxxxxxxxxx–queue-full-threshold  8 –queue-full-sample-size 32

    6. Delay ACK enable /disable


    7. Port binding considerations use / not use



On next article I will try to gather all ESXi hypervisor layer configuration level best practices and describe test environment and test methodology.

So let’s end this pilot episode with open question – is it worth to use/implement any of them in vSphere environment ?