Browsed by
Tag: vsphere

vCenter Appliance 6.0 U3 email notifications are not sent when multiple email addresses are defined in an alarm action

vCenter Appliance 6.0 U3 email notifications are not sent when multiple email addresses are defined in an alarm action

Recently I tried to configure email notifications on my lab vCenter Server Appliance (6.0u3), but  experience issue:

 “Diagnostic-Code: SMTP;550 5.7.60 SMTP; Client does not have permissions to send as this sender”

I tried to use solution from kb: https://kb.vmware.com/kb/2075153 but apparently, the solution does not work with latest 6.0.x appliance!

After some research and digging deeper (header analysis ), it seems that root cause was invalid return path in the email header. To resolve this you need to edit two system files:

1. SSH to VCSA and enable shell:

#Command>shell.set –enabled True

# Command>shell

2. Open catalog : /etc/sysconfig

mail1

3. Edit “mail” using vi and made a change as in below prtsc:

#vi email

mail2

  • simply check using cat:

mail3

4. In the same catalog edit “sendmail” file adding a domain name “SENDMAIL_GENERICS_DOMAIN=”:

mail4

5. Subsequently, go to /etc/mail catalog and add a user to mask root in “genericstable”:

mail56. Regenerate table:

# makemap -r hash /etc/mail/genericstable.db < /etc/mail/genericstable

7. create file sendmail.mc:

#/sbin/conf.d/SuSEconfig.sendmail -m4 > /sendmail.mc

Note. Do not edit file “sendmail” like in abowe procedure

8. Double check if “sendmail.cf” file in catalog /etc exist if yes then change it a name:

   #mv /etc/sendmail.cf /etc/sendmail.cf.orig

9. Create a new config file:

#m4 /sendmail.mc > /etc/sendmail.cf

10. Open config file “sendmail.cf” (vi) and add IP SMTP/Exchange (DS[xxx.xxx.xxx.xxx] ) server in environment :

mail611. Restart sendmail service:

# /etc/init.d/sendmail restart

 

Now it should work fine !

Esxi Net.ReversePathFwdCheckPromisc Advanced setting

Esxi Net.ReversePathFwdCheckPromisc Advanced setting

During deployment of Cisco proxy appliance, we discovered a problem. According to cisco to resolve this problem qa“Net.ReversePathFwdCheckPromisc” should be set to “1” on ESX’s.

The question is – do you know any negative effects which such change could cause. We believe that there must be a reason why by default this option is set to 0 ? That’s why I decided to figure our what it is used for.

After some research I was able to find answer:

Setting – > Net.ReversePathFwdCheckPromisc = 1 — > this is when you are expecting the reverse filters to filter the mirrored packets, to prevent multicast packets getting duplicated.

Note: If the value of the Net.ReversePathFwdCheckPromisc configuration option is changed when the ESXi instance is running, you need to enable or re-enable the promiscuous mode for the change in the configuration to take effect.

The reason you would use promiscuous mode depends on the requirement and configuration. Please check the below KB Article:

http://kb.vmware.com/kb/1004099

  • This option is not enabled by default because we are not aware of the vSwitch configuration and can’t predict what it could be as it has configurable options.

VMware does not advise to enable this option if we do not have a use case scenario with teamed uplinks and have monitoring software running on the VMs ideally. As When promiscuous mode is enabled at the port group level, objects defined within that port group have the option of receiving all incoming traffic on the vSwitch. Interfaces and virtual machines within the port group will be able to see all traffic passing on the vSwitch causing VM performance impact.

Should the ESX server be rebooted for this change to take effect:  answer is – > Yes, and Yes you can enable this option with the VMs running on the existing portgroup.

Do you have any interesting virtualization related question?

 

SAP application on vSphere platform

SAP application on vSphere platform

This is a mini article to start our Q&A set, a set of not easy to find answer real life questions 😉 qa
Recently I received a question-related to advanced settings SAP app on vSphere platform:
“One of our customer ask us to set the following option to their virtual system: Misc.GuestLibAllowHostInfo This is according to SAP note: 1606643 where SAP requires reconfigure virtual system default configuration. I can’t find details information, which host data would be exposed to virtual system. Could you please point me to documentation or describe which information is being transferred from HOST to virtual systems?“

  • After some research I was able to find answer :

“Misc.GuestLibAllowHostInfo” and “tools.guestlib.enableHostInfo” these configurations if enabled allow the guest OS to access some of the ESXi host configurations, mainly performance metrics e.g. how many CPU cores the host has, their utilization and contention etc. There is no confidential information from other customers which would be visible, however, it may give the user of those SAP VMs access to performance/resource information which you may not want to share.

The following document outlines the effect of the changes as I have described above.

I believe the “might use the information to perform further attacks on the host” could only apply to other vulnerabilities which may exist for the particular hardware information that the guestOS can gather from the ESXi host.
Other than that I am not sure there is any other concern to worry about.

Do you have any interesting virtualization related question?

VMware vSphere tags limit – is it known ?

VMware vSphere tags limit – is it known ?

Recently I received quite interesting question – what is the supported maximum quantity  for tags in vCenter 6.0U2 ?

Malignant author of the question is a good friend of mine and VMware administrator in one person. He ssked about tags limit because he want to use them to provide more information about each of its production VM’s – roughly speaking need to create about 20000 tags.

I thought ok., give me couple seconds to verify this,  and looked fast in vmware configuration maxims …. couple minuntes later it was clear that this is not a easy question 😉

Furthermore after some additional research (no clear statement in official documentation)  we decide to perform tests in lab environment !

We used simple powercli script to create 20000 tags in test vcenter appliance (6.0U2) , below our script:

for($i=1
$i -le 20000
$i++){
New-Tag -Name $i -Description $i -Category test
}

Script worked like a charm without any issue – so far so good :), but when we tried to assign one tag to first vm we encounter web client error 1009  – very strange!

We decided to perform additional test and find out that limit is below 10000.  At this stage we decide to clear this issue with Vmware support and after some time received wery interesting feedback:

  1. NGC has upper bound of retrieve 10000 objects max.
  2. If the tags are less than 10000 then data service timeouts after 120 seconds(default dataservice timeout is 120 seconds).
  3. Decreasing the count to 9994 tags and increasing dataservice timeout, shows up all the tags(Assign) now.

As a temporary workaround for now.
————————————-
1. Have total created tags less than 10000.
2. Increase data service timeout to 600 seconds(10 min).

VMware GSS stands that engineering working now to remove tag limit boundary in next releases vSphere 6.x.

VMware Auto Deploy Configuration in vSphere 6.5

VMware Auto Deploy Configuration in vSphere 6.5

 

 

 

The architecture of auto deploy has changed in vSphere 6.5, one of the main difference is the ImageBuilder build in vCenter and the fact that you can create image profiles through the GUI instead of PowerCLI. That is really good news for those how is not keen on PowerCLI. But let’s go throgh the new configuration process of Auto Deploy. Below I gathered all the necessary steps to configure Auto Deploy in your environment.

  1. Enable Auto Deploy services on vCenter Server. Move to Administration -> System Configuration -> Related Objects, look for and start fallowing services:
  • Auto Deploy
  • ImageBuilder Service

You can change the startup type to start them with the vCenter server automatically as well.

Caution! In case you do not see any services like on the screan below, probably vmonapi and vmware-sca services are stopped.ad1

To start them, log in to vCenter Server through SSH and use fallowing commands:

#service-control  – -status         // to verify the status of these services

#service-control  – -start vmonapi vmware-sca       //to start services

ad2

Next, go back to Web Client and refresh the page.

 

  1. Prepare the DHCP server and configure DHCP scope including default gateway. A Dynamic Host Configuration Protocol (DHCP) scope is the consecutive range of possible IP addresses that the DHCP server can lease to clients on a subnet. Scopes typically define a single physical subnet on your network to which DHCP services are offered. Scopes are the primary way for the DHCP server to manage distribution and assignment of IP addresses and any related configuration parameters to DHCP clients on the network.

When basic DHCP scope settings are ready, you need to configure additional options:

  • Option 066 – with the Boot Server Host Name
  • Option 067 – with the Bootfile Name (it is a file name observed at Auto Deploy Configuration tab on vCenter Server – kpxe.vmw-hardwired)

ad3

  1. Configure TFTP server. For lab purposes I nearly always using the SolarWinds TFTP server, it is very easy to manage. You need to copy the TFTP Boot Zip files available at Auto Deploy Configuration page observed in step 2 to TFTP server file folder and start the TFTP service.

ad4

At this stage when you are try to boot you fresh server should get the IP Address and connect to TFTP server. In the  Discovered Hosts tab of Auto Deploy Configuration you will be able to see these host which received IP addresses and some information from TFTP server, but no Deploy Rule has been assigned to them.

ad5

  1. Create an Image Profile.

Go to Auto Deploy Configuration page -> Software Depots tab  and Import Software Depot

ad6

 

Click on Image Profiles so see the Image Profiles that are defined in this Software Depot.

ad7

The ESXi software depot contains the image profiles and software packages (VIBs) that are used to run ESXi. An image profile is a list of VIBs.

 

Image profiles define the set of VIBs to boot ESXi hosts with. VMware and VMware partners make image profiles and VIBs available in public depots. Use the Image Builder PowerCLI to  examine the depot and the Auto Deploy rule engine to specify which image profile to assign to which host. VMware customers can create a custom image profile based on the public image profiles and VIBs in the depot and apply that image profile to the host.

 

  1. Add Software Depot.

Click on Add Software Depot icon and add custom depot.

ad8

Next point in the newly created custom software depot select Image Profiles and click  New Image Profile.

ad9

I selected the minimum required VIBs to boot ESXi host which are:

  • esx-base 6.5.0-0.0.4073352 VMware ESXi is a thin hypervisor integrated into server hardware.
  • misc-drivers 6.5.0-0.0.4073352 This package contains miscellaneous vmklinux drivers
  • net-vmxnet3 1.1.3.0-3vmw.650.0.0.4073352 VMware vmxnet3
  • scsi-mptspi 4.23.01.00-10vmw.650.0.0.4073352 LSI Logic Fusion MPT SPI driver
  • shim-vmklinux-9-2-2-0 6.5.0-0.0.4073352 Package for driver vmklinux_9_2_2_0
  • shim-vmklinux-9-2-3-0 6.5.0-0.0.4073352 Package for driver vmklinux_9_2_3_0
  • vmkplexer-vmkplexer 6.5.0-0.0.4073352 Package for driver vmkplexer
  • vsan 6.5.0-0.0.4073352 VSAN for ESXi.
  • vsanhealth 6.5.0-0.0.4073352 VSAN Health for ESXi.
  • ehci-ehci-hcd 1.0-3vmw.650.0.0.4073352 USB 2.0 ehci host driver
  • xhci-xhci 1.0-3vmw.650.0.0.4073352 USB 3.0 xhci host driver
  • usbcore-usb 1.0-3vmw.650.0.0.4073352 USB core driver
  • vmkusb 0.1-1vmw.650.0.0.4073352 USB Native Driver for VMware

But the list could be different for you.

 

ad10

  1. Create a Deploy Rule.

ad11

ad12

ad13

ad14

ad15

  1. Activate Deploy Rule

ad16

  1. That’s it, now you can restart you host, it should boot and install according to your configuration now.
Adding a sound card to ESXi hosted VM

Adding a sound card to ESXi hosted VM

Sound Card in vSphere Virtual Machine is an unsupported configuration. This is feature dedicated to Virtual Machines created in VMware Workstation. However, you can still add HD Audio device to vSphere Virtual Machine by manually editing .vmx file. I have tested it in our lab environment and it works just fine.

Below  procedure how to do this:

1. Verify storage where VM with no soundcard reside

soundcard1

  1. Login with root to the ESXi host where VM reside using SSH.
    3. Navigate to /vmfs/volumes/<VM LUN>/<VM folder>
    In my example it was:
    ~# cd /vmfs/volumes/Local_03esx-mgmt_b/V11_GSS_DO
    4. Shut down problematic VM
    5. Edit .vmx file using VI editor.

IMPORTANT:
Make a backup copy of the .vmx file. If your edits break the virtual machine, you can roll back to the original version of the file.
More information about editing files on ESXi host, refer to KB article: https://kb.vmware.com/kb/1020302

  1. Once you have open vmx to edit, navigate to the bottom of the file and add following lines to the .vmx configuration file:
    sound.present = “true”
    sound.allowGuestConnectionControl = “false”
    sound.virtualDev = “hdaudio”
    sound.fileName = “-1”
    sound.autodetect = “true”
  2. Save file and Power-On Virtual machine.
  3. Once it have booted, and you have enabled Windows Audio Service, sound will work fine.

If you go to “Edit Settings” of the VM, you can see information that device is unsupported. Please be aware that if after adding sound card to you virtual machine, you may exprience any kind of unexpected behavior (tip: in our lab env work this config without issues).

VCAP6-DCV Design exam experience(s)

VCAP6-DCV Design exam experience(s)

Finally, I’m pround to announce that VCIX6-DCV goal is achived!

Previously I passed the Deploy Exam (you can read about it in this post) which for me personally was far more intuitive and effortless. If you are a practitioner person than visioner and designer it would be quite tought to get used to these kind of questions and reasoning. In my opinion there are a few points which I can not agree with and I would be glad to discuss with the authors of these questions about their points of view 🙂

However, as I read on one of the blogs this is a VMware exam and they could have their own point of view and opinion about best practicies in designing virtual environments.

As you realized I used plural in word experience – it’s not so hard to guess why. Yes, I had to take the exam twice. Nevertheless, I finished the first try quite satisfied and full of hope the reality was brutal. 243 points appeared not to be enought to pass it…That was a food for thoughts.

That made me aware that I need to prepare better and figure out about the key used in design quiestions. It’s not exacly the key but the way of designs constructions. As usually Internet was priceless. First of all I found tips that the exam is similar to VCAP5 version and fallowing this idea I read the VCAP5-DCD Official Cert Guide. This was quite useful. Then I tried to think about the design questions I met and gind out what could be wrong there.

After a few more white papers, blog articles and other readings I took the second try and happily this the reult was much more better and of course I finally managed to pass and gain complete VCIX title.

The few tips from me:

  1. Be fresh and rested at the exam day ( there are 205 minutes, it’s quite a long to sit in front of the screen).
  2. Stay focused and read carefully all the questions and instructions at least twice.
  3. Start from the design questions which would take you a little bit more time.
  4. Be prepared.

Materials I found usefull during preparation time:

  1. VCAP6-Design Blueprint and all associated documents especially those from objective 1.2 and 1.3 should be read more than once
  2. VCAP5-DCD Official Cert Guide
  3. Study Guides of other people
  4. Google+ VCAP-DCD Study Group

I also recommend to get yourself familiar with scoring methodology described at The Cloud JAR’s Blog

 

 

 

VirtualVillage’s home LAB

VirtualVillage’s home LAB

It is possible to learn especially about VMware products using just books, official trainings, blogs, etc. However, we believe that the real knowledge is available only by practice and not all could be tested or verified using production environments 🙂

And again, you can test a lot just using Workstation on your notebook (providing it is powerful enough) but these days there are more and more virtual infrastructure component which requires a lot of resources. Furthermore, having real servers and storage array is also a little bit different than deploying a few small virtual machines running on a notebook.

That is why a few years ago we decided to join forces and build the real laboratory where we are able to test even the most sophisticated  deployments not only with VMware products without being constraint by the resources.

The main hardware components of our lab infrastructure are included in the table below.

Hardware Component Quantity Details Purpose
ServerFujitsu TX200 S7 2 2x CPU E5-4220, 128 GB RAM Payload Cluster
Server Fujitsu TX100 S1 2 Router/Firewall and Backup
Server Fujitsu TX100 S3 3 1x CPU E3-1240, 32 GB RAM Management Cluster
NAS Synology DS2413+ 1 12 x 1 TB SATA 7,2K Gold Storage
NAS Synology RS3617+ 1 12 x 600 GB SAS 15K Silver Storage
NAS QNAP T410 1 4 x 1TB SATA 5,4K Bronze Storage (ISO)
Switch HPE 1910 1 48x 1 Gbps Connectivity

 

Of course we didn’t buy it at once. The environment evaluates with increasing needs. ( In the near future we are going to expand management cluster with 4 host and deploy NSX).

The logical topology looks like this:

lab

 

Despite the fact that most of our servers use tower cases, we installed them in a self made 42U Rack. Unfortunatelly, especially during the summer it could not go without air conditoning (this is one of the most power consuming part of the lab..)

 

Later, either me or Daniel will describe the software layer of our Lab. I hope, it will give an inspiration to anyone who is thinking about own lab.

 

VMware PowerCLI – Introduction

VMware PowerCLI – Introduction

To begin the jurney with PowerCLI we need to start from the installation of PowerCLI itself.

The installation can be done on a Windows based system, that could be some kind of an administration server. The installation files can be found on this VMware site.

There are a few versions available, they are released asynchronously with vSphere and the version numbers do not exactly correspond to vSphere versions. The most recent version is 6.5 whilst there are other like 6.3, 6.0 or 5.8 available.

Before you install the PowerCLI I recommend to change the Execution Policy of Powershell. It is required to run scripts. To do it, run Windows PowerShell as administrator and execute fallowing command:

Set-ExecutionPolicy RemoteSigned

The installation process is really straightforward, that’s why I will not spam the screanshoots of installations here.

After you finish the installation you can run it and see the first Welcome screen like this:

powercli1

 

The first command I suggest to use is:

Get-VICommand

it lists all the available commands. However to display any information about virtual infrastructure you need to connect to a vCenter server or ESXi host. We will do it in the next part after introduction of useful tools which can be used in conjunction with PowerCLI.

How to monitor virtual network – story about netflow in vSphere environment.

How to monitor virtual network – story about netflow in vSphere environment.

Before we start talking about NetFlow configuration on VMware vSphere let’s back to basics and review protocol itself. NetFlow was originally developed by Cisco and has become a reasonably standard mechanism to perform network analysis. NetFlow collect network traffic statistics on designated interfaces. Commonly used in the physical world to help gain visibility into traffic and understanding just who is sending what and to where.

NetFlow comes in a variety of versions, from v1 to v10. VMware uses the IPFIX version

of NetFlow, which is version 10. Each NetFlow monitoring environment need to have exporter ( device carrying  netflow flow’s) , collector (main component ) and of course some network to monitor and analyze 😉

Below You can see basic environment diagram:

netflow1

We can describe flow as tcp/ip packets sequence (without direction) that have common:

  • Input interface
  • Source IP
  • Destination IP
  • TCP/IP Protocol
  • Source Port (TCP/UDP)
  • Destination Port (TCP/UDP)
  • ToS IP

Note. vSphere 5.0 uses NetFlow version 5, while vSphere 5.1 and beyond uses IPFIX (version 10).

Ok, we know that distributed virtual is needed to configure NetFlow on vSphere but what about main component NetFlow collector – as usual we have couple options that we can simply divide in commercial software with fancy graphical interfaces and open source staff for admins that still like old good cli 😉

Below I will show simple implementation steps describing examples from both approach :

Manage engine NetFlow analyzer v12.2, more about software on https://www.manageengine.com/products/netflow/?gclid=CP3HlJbyv9ACFSQz0wod_UcDCw my lab VM setup:

  • Guest OS:Windows 2008R2
  • 4GB RAM
  • 2vCPU
  • 60 GB HDD
  • vNIC interface connected to ESXi management network

Installation (using embedded database just for demo purpose) is really simple and straight forward. Let’s start from starting the installer:

netflow2

 

  1. accept license agreements

netflow3

  1. choose installation folder on vm hdd

netflow4

  1. choose installation component option – for this demo purpose we go with simple environment with only one collector server, central reporting is not necessary

netflow5

  1. choose web server and collector services TCP/IP ports

netflow6

  1. provide communication details – again in this demo we have all components on one server and we can simply go with localhost

netflow7

 

  1. optional – configuration proxy server details

netflow8

  1. select database – on this demo i used embedded Postgresql , but if You choose MS database remember about ODBC config.

netflow9

  1. installation is quite fast – couple more minutes and solution will be ready and available to start work:

netflow10

 

… Web client like in VMware need couple CPU cycles to start 😉

netflow11

.. and finally we can see fancy ManageEngine NetFlow collector

netflow12

II) Open-Source netdump tool  – nfdump is distributed under the BSD license, and can be downloaded at: http://sourceforge.net/projects/nfdump/ my lab VM steup:

  • GOS: Debian 8.6
  • 4GB RAM
  • 2vCPU
  • 60 HDD
  • vNIC interface connected to ESXi management network

 

  1. We need to start from adding some sources to our debian distribution:

netflow13

  1. CLI Installation nfdump packet:

netflow15

netflow14

  1. Run simple flow capture to verify if collector is running and creating output flow statictics files (you can see that i use same tcp port 9995 and folder on my desktop as output destination):

netflow16

 

Ok, now it is time to back to vSphere and configure DVS to send network traffic to collector:

netflow17

 

  • IP Address: This is the IP of the NetFlow Collector
  • Port: This is the port used by the NetFlow Collector.
  • Switch IP Address: This one can be confusing – by assigning an IP address of here, the NetFlow Collector will treat the VDS as one single entity. It does not need to be a valid, routable IP, but is merely used as an identifier.
  • Active flow export timeout in seconds: The amount of time that must pass before
  • the switch fragments the flow and ships it off to the collector.
  • Idle flow export timeout in seconds: Similar to the active flow timeout, but for flows
  • that have entered an idle state.
  • Sampling rate: This determines the interval packet to collect. By default, the value is 0,
  • meaning to collect all packets. If you set the value to something other than 0, it will
  • collect every X packet.
  • Process internal flows only: Enabling ensures that the only flows collected are ones that occur between VMs on the same host.

And enable it at designated port group level:

netflow18

Finally we can create simple lab scenario and capture some ftp flow statistics between two vm’s on different ESXi :

netflow19

VM’s are running in dedicated vlan on the same DVS port group, collector is running on management network to communicate with vCenter and ESXi hosts. I used ftp connection to generate traffic between vm’s below examples output from two collectors (test ran separate as collector share the same ip)  :

 

ftp client on first vm:

netflow20

ftp server on second vm:

netflow21

flow statistics example from netdump:

netflow22

flow statistics from ManageEngine

netflow23