One of the members of the VMware User Community (VMTN) inspired me to build a configuration where two VMs use PVRDMA network adapters to communicate. The goal I wanted to achieve was to establish the communication between VMs
without using Host Channel Adapter cards installed in hosts. It’s possible to configure it as stated here, in the VMware vSphere documentation.
For virtual machines on the same ESXi hosts or virtual machines using the TCP-based fallback, the HCA is not required.
To do this task, I prepared one ESXi host (6.7U1) managed by vCSA (6.7U1). One of the requirements for RDMA is a vDS. First, I configured a dedicated vDS for RDMA communication. I simply set a basic vDS configuration (DSwitch-DVUplinks-34) with a default portgroup (DPortGroup) and, then, equipped it with just one uplink.
In vSphere, a virtual machine can use a PVRDMA network adapter to communicate with other virtual machines that have PVRDMA devices. The virtual machines must be connected to the same vSphere Distributed Switch.
Second, I created a VMkernel port (vmk1) dedicated to RDMA traffic in this DPortGroup without assigning an IP address to this port (No IPv4 settings).
Third, I set Advanced System Setting Net.PVRDMAVmknic on ESXi host and gave it a value pointing to VMkernel port (vmk1). Tag a VMkernel Adapter for PVRDMA.
Then, I enabled “pvrdma” firewall rule on host in Edit Security Profile window. Enable the Firewall Rule for PVRDMA.
The next steps are related to configuration of the VMs. First, I created a new virtual machine. Then, I added another Network adapter to it and connected it to DPortGroup on vDS. For the Adapter Type of this Network adapter I chose PVRDMA and Device Protocol RoCE v2. Assign a PVRDMA Adapter to a Virtual Machine.
Then, I installed Fedora 29 on a first VM. I chose it because there are many tools to easily test a communication using RDMA. After the OS installation, another network interface showed up on the VM. I addressed it in a different IP subnet. I used two network interfaces in VMs, the first one to have an access through SSH and the second one to test RDMA communication.
Then I set “Reserve all guest memory (All locked)” in VM’s Edit Settings window.
I had two VMs configured enough – in the infrastructure layer – to communicate using RDMA.
To do it I had to install appropriate tools. I found them on GitHub, here.
To use them, I had to install them first. I did it using the procedure described on the previously mentioned page.
dnf install cmake gcc libnl3-devel libudev-devel pkgconfig valgrind-devel ninja-build python3-devel python3-Cython
Next, I installed the git client using the following command.
yum install git
Then I cloned the git project to a local directory.
mkdir /home/rdma git clone https://github.com/linux-rdma/rdma-core.git /home/rdma
I built it.
cd /home/rdma bash build.sh
Afterwards, I cloned a VM to have a communication partner for the first one. After cloning, I reconfigured an appropriate IP address in the cloned VM.
Finally I could test the communication using RDMA.
On the VM that functioned as a server I ran a listener service on the interface mapped to PVRDMA virtual adapter:
cd /home/rdma/build/bin ./rping -s -a 192.168.0.200 -P
I ran this command, which allowed me to connect the client VM to the server VM:
./rping -c -I 192.168.0.100 -a 192.168.0.200 -v
It was working beautifully!
Sometimes You want to shutdown vCSA or PSC gracefully, but You don’t have an access to GUI through vSphere Client or VAMI.
How to do it in CLI? I’m going to show You right now using dcli, because I’m exploring a potential of this tool and I can’t get enough.
- Open an SSH session to vCSA and log in as root user.
- Run dcli command in an interactive mode.
- Use shutdown API call, to shutdown an appliance, giving a delay value (0 means now) and a description of disk task.
- Enter an appropriate administrator user name e.g. firstname.lastname@example.org and a password.
- Decide if You want save the credentials in the credstore. You can enter ‘y’ as yes.
Wait until the appliance will go down. 🙂
I know that it’s not the quickest way, but the point is to have fun.
There’s a plenty of great CLI tools in VCSA that modern vSphere administrator should know, so I decided to share my knowledge and describe them in the series of articles.
The first one is journalctl. A tool that simplifies and quickens the VCSA troubleshooting process.
Below I’m presenting how I’m using it, to filter the logs records.
Log in to VCSA shell and run the commands below, regarding to the result you want to achive.
The logs from the current boot:
The boots that journald is aware of:
It will show this kind of results representing each known boot:
Now you can use it, to display the results of the chosen boot:
To limit the results to the given timeframe, use dates or keywords:
Limiting the logs by service name:
Get the service names first:
To show only records for vpxd service in current boot:
Filtering by Id (Process, User or Group):
Get the user Id, e.g.:
Filtering by the binary path:
Displaying kernel messages:
Limiting messages by their priorities:
where the priority codes are:
Of course those examples don’t exhaust all the possibilities this tool has, but consider them as a usage starting point. Feel free to add new, useful examples in comments below.
That’s all for today!
The orphaned VMs in vCenter inventory is an unusual view in experienced administrator’s Web/vSphere Client window. But in large environments, where many people manage hosts and VMs it will happen sometimes.
You do know how to get rid of them using traditional methods described in VMware KB articles and by other well known bloggers, but there’s a quite elegant new method using dcli.
This handy tool is available in vCLI package, in 6.5/6.7 vCSA shell and vCenter Server on Windows command prompt. Dcli does use APIs to give an administrator the interface to call some methods to get done or to automate some tasks.
How to use it to remove orphaned VMs from vCenter inventory?
- Open an SSH session to vCSA and log in as root user.
- Run dcli command in an interactive mode.
- Get a list of VMs registered in vCenter’s inventory. Log in as administrator user in your SSO domain. You can save credentials in the credstore for future use.
- From the displayed list get VM’s MoID (Managed Object Id) of the affected VM, e.g. vm-103.
- Run this command to delete the record of the affected VM using its MoID from vCenter’s database.
- Using Web/vSphere Client check the vCenter’s inventory if the affected VM is now deleted.
There are a few caveats that make the calculation and planning of VSAN capacity tough and gets even harder when you try to map it with real consumption on the VSAN datastore level.
- VSAN disks objects are thin provisioned by default.
- Configuring full reservation of storage space through Object Space Reservation rule in Storage Policy, does not mean
disk object block will be inflated on a datastore. This only means the space will be reserved and showed as used in VSAN Datastore Capacity pane.
Which makes it even harder to figure out why size of “files” on this datastore is not compliant with other information related to capacity.
- In order to plan capacity you need to include overhead of Storage Policies. Policies – as I haven’t met an environment which would use only one for all kinds of workloads. This means that planning should start with dividing workloads for different groups which might require different levels of protections.
- Apart from disks objects there are different objects especially SWAP which are not displayed in GUI and can be easily forgotten. However, based on the size of environment they might consume considerable amount of storage space.
- VM SWAP object does not adhere to Storage Policy assigned to VM. What does it mean? Even if you configure your VM’s disks with PFTT=0
SWAP will always utilize PFTT=1. Unless you configure advanced option (SwapThickProfivisionedDisabled) to disable it.
I have made a test to check how much space will consume my empty VM. (Empty means here without operating system even)
In order to see that a VM called Prod-01 has been created with 1 GB of memory and 2 GB of Hard disk and default storage policy assigned (PFTT=1)
Based on the Edit Setting window the VM disk size on datastore is 4 GB (Maximum sized based on disk size and policy). However, used storage space is 8 MB which means there will be 2 replicas 4 MB each, which is fine as there is no OS installed at all.
However, when you open datastore files you will see this list with Virtual Disk object you will notice that the size is 36 864 KB which gives us 36 MB. So it’s neither 4 GB nor 8 MB as displayed by edit setting consumption..
Meanwhile datastore provisioned space is listed as 5,07 GB.
So let’s power on that VM.
Now the disks size remain intact, but other files appear as for instance SWAP has been created as well as log and other temporary files.
Looking at datastore provisioned space now it shows 5,9 GB. Which again is confisung even if we forgot about previous findings powering on VM triggers SWAP creation which according to the theory should be protected with PFTT=1 and be thick provisioned. But if that’s the case then the provisioned storage consumption should be increased by 2 GB not 0,83 (where some space is consumed for logs and other small files included in Home namespace object)
Moreover during those observations I noticed that during the VM booting process the provisioned space is peaking up to 7,11 GB for a very short period of time
And this value after a few seconds decreases to 5.07 GB. Even after a few reboots those values stays consistent.
The question is why those information are not consistent and what heppens during booting of the VM that is the reason for peak of provisioned space?
That’s the quest for not to figure it out 🙂
The other way to list MAC addresses of open ports on vSwitches on the ESXi host is based on net-stats tool.
Use this one-liner.
This is not a final word. 🙂
Sometimes You need to list MAC addresses loged on host’s vSwitches to eliminate VM’s MAC address duplicates.
- Create a shell script:
- Copy and past the code listed below:
- Change the file’s permissions
- Run the script
Simple, but useful! 🙂
… but this is not the only one possible method 🙂
Creating virtual switch through GUI is well described in documentation and pretty intuitive using GUI. However, sometimes it might be useful to know how to do it with CLI or Powershell, thus making the process part of a script to automate initial configuration of ESXi after installation.
Here you will find commands which are necessary to create and configure a standard virtual switch using CLI and Powershell. Those examples will describe the process of vSwitch creation for vMotion traffic which involves VMkernel creation.
I. vSwitch configuration through CLI
- Create a vSwitch named “vMotion”
esxcli network vswitch standard add -v vMotion
- Check whether your newly created vSwitch was configured and is available on the list.
esxcli network vswitch standard list
- Add physical uplink (vmnic) to your vSwitch
esxcli network vswitch standard uplink add -u vmnic4 -v vMotion
- Designate an uplink to be used as active.
esxcli network vswitch standard policy failover set -a vmnic4 -v vMotion
- Add a port group named “vMotion-PG” to previously created vSwitch
esxcli network vswitch standard portgroup add -v vMotion -p vMotion-PG
- Add a VMkernel interface to a port group (Optional – not necessary if you are creating a vSwitch just for VM traffic)
esxcli network ip interface add -p vMotion-PG -i vmk9
- Configure IP settings of a VMkernel adapter.
esxcli network ip interface ipv4 set -i vmk9 -t static -I 172.20.14.11 -N 255.255.255.0
- Tag VMkernel adapter for a vMotion service. NOTE – service tag is case sensitive.
esxcli network ip interface tag add -i vmk9 -t vmotion
Done, your vSwitch is configured and ready to service vMotion traffic.
II. vSwitch configuration through PowerCLI
- First thing is to connect to vCenter server.
Connect-VIServer -Server vcsa.vclass.local -User email@example.com -Password VMware1!
- Indicate specific host and create new virtual switch, assigning vmnic at the same time.
$vswitch1 = New-VirtualSwitch -VMHost sa-esx01.vclass.local -Name vMotion -NIC vmnic4
- Create port group and add it to new virtual switch.
New-VirtualPortGroup -VirtualSwitch $vswitch1 -Name vMotion-PG
- Create and configure VMkernel adapter.
New-VMHostNetworkAdapter -VMHost sa-esx01.vclass.local -PortGroup vMotion-PG -VirtualSwitch vMotion -IP 172.20.11.11 -SubnetMask 255.255.255.0 -vmotionTrafficEnabled $true