Browsed by
Category: Uncategorized

Part 1 – PVRDMA and how to test it in home lab.

Part 1 – PVRDMA and how to test it in home lab.

One of the members of the VMware User Community (VMTN) inspired me to build a configuration where two VMs use PVRDMA network adapters to communicate. The goal I wanted to achieve was to establish the communication between VMs
without using Host Channel Adapter cards installed in hosts. It’s possible to configure it as stated here, in the VMware vSphere documentation.

For virtual machines on the same ESXi hosts or virtual machines using the TCP-based fallback, the HCA is not required.

To do this task, I prepared one ESXi host (6.7U1) managed by vCSA (6.7U1). One of the requirements for RDMA is a vDS. First, I configured a dedicated vDS for RDMA communication. I simply set a basic vDS configuration (DSwitch-DVUplinks-34) with a default portgroup (DPortGroup) and, then, equipped it with just one uplink.

In vSphere, a virtual machine can use a PVRDMA network adapter to communicate with other virtual machines that have PVRDMA devices. The virtual machines must be connected to the same vSphere Distributed Switch.

Second, I created a VMkernel port (vmk1) dedicated to RDMA traffic in this DPortGroup without assigning an IP address to this port (No IPv4 settings).

Third, I set Advanced System Setting Net.PVRDMAVmknic on ESXi host and gave it a value pointing to VMkernel port (vmk1). Tag a VMkernel Adapter for PVRDMA.

Then, I enabled “pvrdma” firewall rule on host in Edit Security Profile window. Enable the Firewall Rule for PVRDMA.

The next steps are related to configuration of the VMs. First, I created a new virtual machine. Then, I added another Network adapter to it and connected it to DPortGroup on vDS. For the Adapter Type of this Network adapter I chose PVRDMA and Device Protocol RoCE v2. Assign a PVRDMA Adapter to a Virtual Machine.

Then, I installed Fedora 29 on a first VM. I chose it because there are many tools to easily test a communication using RDMA. After the OS installation, another network interface showed up on the VM. I addressed it in a different IP subnet. I used two network interfaces in VMs, the first one to have an access through SSH and the second one to test RDMA communication.

Then I set “Reserve all guest memory (All locked)” in VM’s Edit Settings window.

I had two VMs configured enough – in the infrastructure layer – to communicate using RDMA.

To do it I had to install appropriate tools. I found them on GitHub, here.
To use them, I had to install them first. I did it using the procedure described on the previously mentioned page.

dnf install cmake gcc libnl3-devel libudev-devel pkgconfig valgrind-devel ninja-build python3-devel python3-Cython	

Next, I installed the git client using the following command.

yum install git

Then I cloned the git project to a local directory.

mkdir /home/rdma
git clone https://github.com/linux-rdma/rdma-core.git /home/rdma

I built it.

cd /home/rdma
bash build.sh

Afterwards, I cloned a VM to have a communication partner for the first one. After cloning, I reconfigured an appropriate IP address in the cloned VM.

Finally I could test the communication using RDMA.

On the VM that functioned as a server I ran a listener service on the interface mapped to PVRDMA virtual adapter:

cd /home/rdma/build/bin
./rping -s -a 192.168.0.200 -P

I ran this command, which allowed me to connect the client VM to the server VM:

./rping -c -I 192.168.0.100 -a 192.168.0.200 -v

It was working beautifully!

pvrdma

dcli and how to shutdown vCSA

dcli and how to shutdown vCSA

Sometimes You want to shutdown vCSA or PSC gracefully, but You don’t have an access to GUI through vSphere Client or VAMI.

How to do it in CLI? I’m going to show You right now using dcli, because I’m exploring a potential of this tool and I can’t get enough.

  1. Open an SSH session to vCSA and log in as root user.
  2. Run dcli command in an interactive mode.

    dcli +i
  3. Use shutdown API call, to shutdown an appliance, giving a delay value (0 means now) and a description of disk task.

    com vmware appliance shutdown poweroff --delay 0 --reason 'Shutdown now'
  4. Enter an appropriate administrator user name e.g. administrator@vsphere.local and a password.
  5. Decide if You want save the credentials in the credstore. You can enter ‘y’ as yes.

Wait until the appliance will go down. 🙂

I know that it’s not the quickest way, but the point is to have fun.

dcli and orphaned VMs in vCenter Server inventory

dcli and orphaned VMs in vCenter Server inventory

The orphaned VMs in vCenter inventory is an unusual view in experienced administrator’s Web/vSphere Client window. But in large environments, where many people manage hosts and VMs it will happen sometimes.

You do know how to get rid of them using traditional methods described in VMware KB articles and by other well known bloggers, but there’s a quite elegant new method using dcli.
This handy tool is available in vCLI package, in 6.5/6.7 vCSA shell and vCenter Server on Windows command prompt. Dcli does use APIs to give an administrator the interface to call some methods to get done or to automate some tasks.

How to use it to remove orphaned VMs from vCenter inventory?

  1. Open an SSH session to vCSA and log in as root user.
  2. Run dcli command in an interactive mode.

    dcli +i
  3. Get a list of VMs registered in vCenter’s inventory. Log in as administrator user in your SSO domain. You can save credentials in the credstore for future use.

    com vmware vcenter vm list
  4. From the displayed list get VM’s MoID (Managed Object Id) of the affected VM, e.g. vm-103.
  5. Run this command to delete the record of the affected VM using its MoID from vCenter’s database.

    com vmware vcenter vm delete --vm vm-103
  6. Using Web/vSphere Client check the vCenter’s inventory if the affected VM is now deleted.

It’s working!

Part 2 – How to list vSwitch “MAC Address table” on ESXi host?

Part 2 – How to list vSwitch “MAC Address table” on ESXi host?

The other way to list MAC addresses of open ports on vSwitches on the ESXi host is based on net-stats tool.

Use this one-liner.

		
for VSWITCH in $(vsish -e ls /net/portsets/ | cut -c 1-8); do net-stats -S $VSWITCH | grep \{\"name | sed 's/[{,"]//g' | awk '{$9=$10=$11=$12=""; print $0}'; done		
		
	

This is not a final word. 🙂

Part 1 – How to list vSwitch “MAC Address table” on ESXi host?

Part 1 – How to list vSwitch “MAC Address table” on ESXi host?

Sometimes You need to list MAC addresses loged on host’s vSwitches to eliminate VM’s MAC address duplicates.

  1. Create a shell script:
  2. vi mac_address_list.sh
  3. Copy and past the code listed below:
  4. 
    #!/bin/sh
    #vmrale
    for VSWITCH in `vsish -e ls /net/portsets/ | cut -c 1-8`
    do
            echo $VSWITCH
            for PORT in `vsish -e ls /net/portsets/$VSWITCH/ports | cut -c 1-8`
            do
                    CLIENT_NAME=`vsish -e get /net/portsets/$VSWITCH/ports/$PORT/status | grep clientName | uniq`
                    ADDRESS=`vsish -e get /net/portsets/$VSWITCH/ports/$PORT/status | grep unicastAdd | uniq`
                    echo -e "\t$PORT\t$CLIENT_NAME\t$ADDRESS"
            done
    done        
    
    
  5. Change the file’s permissions
  6. chmod 755 mac_address_list.sh
  7. Run the script
  8. ./mac_address_list.sh

Simple, but useful! 🙂

… but this is not the only one possible method 🙂

Alternative methods to create virtual switch.

Alternative methods to create virtual switch.

Creating virtual switch through GUI is well described in documentation and pretty intuitive using GUI. However, sometimes it might be useful to know how to do it with CLI or Powershell, thus making the process part of a script to automate initial configuration of ESXi after installation.

Here you will find commands which are necessary to create and configure a standard virtual switch using CLI and Powershell. Those examples will describe the process of vSwitch creation for vMotion traffic which involves VMkernel creation.

I. vSwitch configuration through CLI

  1. Create a vSwitch named “vMotion”

esxcli network vswitch standard add -v vMotion

  1. Check whether your newly created vSwitch was configured and is available on the list.

esxcli network vswitch standard list

  1. Add physical uplink (vmnic) to your vSwitch

esxcli network vswitch standard uplink add -u vmnic4 -v vMotion

  1. Designate an uplink to be used as active.

esxcli network vswitch standard policy failover set -a vmnic4 -v vMotion

  1. Add a port group named “vMotion-PG” to previously created vSwitch

esxcli network vswitch standard portgroup add -v vMotion -p vMotion-PG

  1. Add a VMkernel interface to a port group (Optional – not necessary if you are creating a vSwitch just for VM traffic)

esxcli network ip interface add -p vMotion-PG -i vmk9

  1. Configure IP settings of a VMkernel adapter.

esxcli network ip interface ipv4 set -i vmk9 -t static -I 172.20.14.11 -N 255.255.255.0

  1. Tag VMkernel adapter for a vMotion service. NOTE – service tag is case sensitive.

esxcli network ip interface tag add -i vmk9 -t vmotion

Done, your vSwitch is configured and ready to service vMotion traffic.

 

II. vSwitch configuration through PowerCLI

  1. First thing is to connect to vCenter server.

Connect-VIServer -Server vcsa.vclass.local -User administrator@vsphere.local -Password VMware1!

  1. Indicate specific host and create new virtual switch, assigning vmnic at the same time.

$vswitch1 = New-VirtualSwitch -VMHost sa-esx01.vclass.local -Name vMotion -NIC vmnic4

  1. Create port group and add it to new virtual switch.

New-VirtualPortGroup -VirtualSwitch $vswitch1 -Name vMotion-PG

  1. Create and configure VMkernel adapter.

New-VMHostNetworkAdapter -VMHost sa-esx01.vclass.local -PortGroup vMotion-PG -VirtualSwitch vMotion -IP 172.20.11.11 -SubnetMask 255.255.255.0 -vmotionTrafficEnabled $true

 

VMware Virtual SAN 6.6 what’s new

VMware Virtual SAN 6.6 what’s new

1vsan

vSAN 6.6 it’s 6th generation of the product and there are more than 20+ new features and enhancements in this release, such as:

  • Native encryption for data-at-rest
  • Compliance certifications
  • Resilient management independent of vCenter
  • Degraded Disk Handling v2.0 (DDHv2)
  • Smart repairs and enhanced rebalancing
  • Intelligent rebuilds using partial repairs
  • Certified file service & data protection solutions
  • Stretched clusters with local failure protection
  • Site affinity for stretched clusters
  • 1-click witness change for Stretched Cluster
  • vSAN Management Pack for vRealize
  • Enhanced vSAN SDK and PowerCLI
  • Simple networking with Unicast
  • vSAN Cloud Analytics with real-time support notification and recommendations
  • vSAN Config Assist with 1-click hardware lifecycle management
  • Extended vSAN Health Services
  • vSAN Easy Install with 1-click fixes
  • Up to 50% greater IOPS for all-flash with optimized checksum and dedupe
  • Support for new next-gen workloads
  • vSAN for Photon in Photon Platform 1.1
  • Day 0 support for latest flash technologies
  • Expanded caching tier choice
  • Docker Volume Driver 1.1

 

… ok now lets review main enhancements:

vSAN 6.6 introduces the industry’s first native HCI security solution. vSAN will now offer data-at-rest encryption that is completely hardware-agnostic. No more concern about someone walking off with a drive or breaking in to a less-secure, edge IT location and stealing hardware. Encryption is applied at the cluster level, and any data written to a vSAN storage device, both at the cache layer and persistent layer can now be fully encrypted.  And vSAN 6.6 supports 2-factor authentication, including SecurID and CAC.

2vsan

Certified file services and data protection solutions are available from 3rd party partners in the VMware Ready for vSAN Program to enable customers to extend and complement their vSAN environment with proven, industry-leading solutions. These solutions provide customers with detailed guidance on how to complement vSAN. (EMC NetWorker is avaialble today with new solutions coming on soon)

3vsan

vSAN stretched cluster was released in Q3’15 to provide an Active-Active solution. vSAN 6.6 adds a major new capability that will deliver a highly-available stretched cluster that addresses the highest resiliency requirements of data centers. vSAN 6.6 adds support for local failure protection that can provide resiliency against both site failures and local component failures.

4vsan

PowerCLI Updates: Full featured vSAN PowerCLI cmdlets enable full automation that includes all the latest features. SDK/API updates also enable enterprise-class automation that brings cloud management flexibility to storage by supporting REST APIs.

VMware vRealize Operations Management Pack for vSAN released recently, provides customers with native integration for simplified management and monitoring. The vSAN management pack is specifically designed to accelerate time to production with vSAN, optimize application performance for workloads running on vSAN and provide unified management for the Software Defined Datacenter (SDDC). It provides additional options for monitoring, managing and troubleshooting vSAN along with the end-to-end infrastructure solutions.

5vsan

Finally, vSAN 6.6 is well suited for next-generation applications. Performance improvements, especially when combined with new flash technologies for write-intensive applications, enable vSAN to address more emerging applications like Big Data. The vSAN team has also tested and released numerous reference architectures for these types of solutions, including Big Data, Splunk and InterSystems Cache.

RESOURCES:

  • Splunk Reference Architecture: http://www.emc.com/collateral/service-overviews/h15699-splunk-vxrail-sg.pdf
  • Citrix XenDestkop/XenApp Blog: https://blogs.vmware.com/virtualblocks/2017/02/27/citrix-xenapp-xendesktop-7-12-vmware-vsan-6-5-flash/
  • vSAN, VxRail and Pivotal Cloud Foundry RA: https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsan/vmware-pcf-vxrail-reference-architeture.pdf
  • vSAN and InterSystems Blog: https://community.intersystems.com/post/intersystems-data-platforms-and-performance-%E2%80%93-part-8-hyper-converged-infrastructure-capacity
  • Intel, vSAN and Big Data Hadoop: https://builders.intel.com/docs/storagebuilders/Hyper-Converged_big_data_using_Hadoop_with_All-Flash_VMware_vSAN.pdf

 

 

vCenter 6.5 DSN permissions

vCenter 6.5 DSN permissions

Recently we had some strange problems with our 6.5 lab vCenter (Windows version with MSSQL Server db), which frequently crashed. After some digging in vpxd logs it seem to be related to vc db permissions:

17-05-28T19:36:53.443+02:00 error vpxd[05420] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: “ODBC error: (42000) – [Micrsoft][SQL Server Native Client 11.0][SQL Server]VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.” is returned when executing SQL statement “SELECT  DB_NAME(mf.DATABASE_ID) Db_Name,            CASE mf.FILE_ID WHEN 1 THEN ‘DATA’                            WHEN 2 THEN ‘LOG’            END File_Type,            vol.VOLUME_MOUNT_POINT AS Drive,            CONVERT(INT,vol.AVAILABLE_BYTES/1048576.0) FreeSpaceInMB,            (mf.SIZE*8)/1024 VCDB_Space_Mb,             mf.PHYSICAL_NAME Physical_Name,             SERVERPROPERTY(‘edition’) Sql_Server_Edition,             SERVERPROPERTY(‘productversion’) Sql_Server_Version            FROM            SYS.M” action.

The SQL execution is failing as the vCenter Server database user has no permisss on ‘master’ database, to resolve this issue, grant additional privileges to the vCenter Server database user:

use master
go
grant VIEW SERVER STATE to [vCenter_database_user]
go
GRANT VIEW ANY DEFINITION TO [vCenter_database_user]
go

 

SAP application on vSphere platform

SAP application on vSphere platform

This is a mini article to start our Q&A set, a set of not easy to find answer real life questions 😉 qa
Recently I received a question-related to advanced settings SAP app on vSphere platform:
“One of our customer ask us to set the following option to their virtual system: Misc.GuestLibAllowHostInfo This is according to SAP note: 1606643 where SAP requires reconfigure virtual system default configuration. I can’t find details information, which host data would be exposed to virtual system. Could you please point me to documentation or describe which information is being transferred from HOST to virtual systems?“

  • After some research I was able to find answer :

“Misc.GuestLibAllowHostInfo” and “tools.guestlib.enableHostInfo” these configurations if enabled allow the guest OS to access some of the ESXi host configurations, mainly performance metrics e.g. how many CPU cores the host has, their utilization and contention etc. There is no confidential information from other customers which would be visible, however, it may give the user of those SAP VMs access to performance/resource information which you may not want to share.

The following document outlines the effect of the changes as I have described above.

I believe the “might use the information to perform further attacks on the host” could only apply to other vulnerabilities which may exist for the particular hardware information that the guestOS can gather from the ESXi host.
Other than that I am not sure there is any other concern to worry about.

Do you have any interesting virtualization related question?

VMware vExperts 2017 awards

VMware vExperts 2017 awards

VMware vExpert is a non technical award granted by a special committee every year. It is a special award because there is no exam or anything like that. VMware judges give it for individuals who have demonstrated significant contributions to the community and share their expertise with others. This means different kinds of activities like blogging, sharing the knowledge by other channels or social media as well as  public speakers, book authors, CloudCred task writing, script writers, VMUG leaders and VMTN community moderators.
VMW-LOGO-vEXPERT-2017-k

In this year we are very pleased to announce that both of us (Paweł Piotrowski and Daniel Okrasa) were awarded with this mysterious title for the first time. The whole list of vExpert 2017 can be found here.

We would like to thanks to all of our readers and your feedback. We promise to develop our blog with more interesting articles about our experiences and observations mostly in relation to VMware products 🙂