May 17, 2021

Priority tagging of vSAN traffic

 Background

According to Cisco COS is defined as "Class of Service (CoS) or Quality of Service (QoS) is a way to manage multiple traffic profiles over a network by giving certain types of traffic priority over others. "

Note that there's also a similar technology called DSCP that can used in more or less the same way.

When using a vSphere Distributed Switch it's possible to configure this and create fairly granular rules per Port Group. It's not at all limited to vSAN traffic even though that was our use case.

Task

I was asked by the networking guys if we could enable this functionality for vSAN traffic by setting COS=3.

Solution

Identify the port group associated with the vmkernel adapter used by vSAN and choose  Edit Settings. / Advanced and enable Traffic filtering and marking. 


At configure level for the port group you will need to create the rule as outlined in the following steps:










Now that it was turned on it was instantly visible to the networking guys as they started seeing traffic within UC3 (Priority Group 3).


April 19, 2021

Autoinstall physical NSX Edge with custom passwords

Background

Setting up NSX Edge in an automatic way with a custom password is a good idea because by default you get a default password that needs to be changed at first login. If you're planning on using an extra strong password, setting it through iDRAC (or similar) can be a bit awkward. If you're using a non-english keyboard layout (like me) it can be even more non-trivial to hit the correct special characters.

Problem

1. We had a problem getting the physical Dell R640 server with Mellanox 25GbE nics to boot from PXE. It would say "Booting from PXE Device 1: Integrated NIC 1 Port 1 Partition 1 Downloading NBP file... NBP File downloaded successfully. Boot: Failed PXE Device 1: Integrated NIC 1 Port 1 Partition 1 No boot device available or Operating system detected. Please ensure a compatible bootable media is available."



2. VMware has provided us with a nice 19 step document that guides us through the needed steps for setting up everything we need. The optional step 16 of setting a non-default password is however a bit misleading (probably referring to an older version of NSX?) and doesn't quite work.

Solution

1. In order to get the physical server to PXE boot we had to change the boot mode from UEFI to BIOS.

2. I had a case open for months without a resolution. In the end I started studying the Debian manuals (that the NSX Edge installer is based upon). I eventually found a working solution. It turned out that adding the following commands to preseed.cfg right after the "di passwd/root..." line gave a working config:

d-i preseed/late_command       string \
        in-target usermod --password 'insert non escaped password hash here' root;\
        in-target usermod --password 'non escaped password hash' admin
You will need to create the password hash using mkpasswd -m sha-512 as described in the original 19 step document.



April 15, 2021

vSAN critical alert regarding a potential data inconsistency and maintenance mode problems after upgrade to 7.0U1

Background

Versions involved: 

VMware ESXi, 7.0.1, 17325551,  DEL-ESXi-701_17325551-A01

vCenter 7.0U1 Build 17491160

vCenter and ESXi hosts were upgraded from 6.7U3 to 7.0U1c an the vSAN disk format was upgraded to version 13.

Problem

After upgrading many clusters from 6.7U3 to 7.0U1c and upgrading the vSAN format to 13 we experienced a health warning after the upgrade.

The error message in Skyline Health was "vSAN critical alert regarding a potential data inconsistency"


For almost all clusters this error would fix itself within 60 minutes after the upgrade (typically in a much shorter time).

For one of our clusters this error did however stick and we were unable to put any hosts within this cluster in maintenance mode.

Trying to put a host in maintenance mode would fail after 1 hour. Before failing it would stop at a high percentage between 80 and even at 100% with a message "Objects Evacuated xxx of yyy. Data Evacuated xxx MB of yyy MB".

It's worth mentioning that this cluster had an active Horizon environment running during the upgrade and we suspect that it's constant tasks of creating and removing VMs has contributed to this problem.



Solution

We found a kb article with a similar error message even though we haven't changed the storage policy of any VMs for  a long time (but Horizon might have done something like that behind the scenes): https://kb.vmware.com/s/article/82383

This article states this is a rare issue, but we found a korean page referring this same issue. The VMware kb article has a python script that you will need to run on each host involved. After running the python script we were able to put hosts in maintenance mode and do 7.x single image patching.

We asked VMware support if it was a good idea that we had changed this setting and their response was "Yes, if you want the DeltaComponent functionality going forward then please change it back to 1. The delta component makes a temporary component when there are maintenance mode issues."

Because of this we decided to change the value back and wrote a powershell script instead of running a python script on each host:

param (

    [string]$clustername = $( Read-Host "Enter cluster name:" )

 )

get-cluster $clustername|Get-VMHost| Get-AdvancedSetting -Name "VSAN.DeltaComponent"| Set-AdvancedSetting -Value 1 -Confirm:$false

As we've only found a single article on this issue (in Korean) I guess this issue is indeed quite rare, but if it happens again we now know what to do.


December 2, 2020

How to check BIOS Power management settings of ESXi hosts

Background
The performance of your workload will possibly be greatly affected by the power saving settings of your hosts. There are power saving settings both in the vSphere client¹, in the BIOS of the hosts² and inside the VMs³. This can cause much confuzion and there are a number of articles related to this issue:
Select a CPU Power Management Policy
Performance Best Practices for VMware vSphere 6.7
Virtual machine application runs slower than expected in ESXi (1018206)
ARCHITECTING MICROSOFT SQL SERVER ON VMWARE VSPHERE®

The root cause of the problem is that servers are normally shipped with a BIOS setting of Balanced power saving. This means that C states are enabled in order to make the cpus sleep whenever they are idle.

You use the vSphere client to check the settings of your BIOS (ESXi host / Configure / Hardware - Power Management) and you can also configure how ESXi should treat power savings.

Note that in vSphere 7 this option has moved to ESXi host / Configure / Hardware - Overview - Power Management.


From the example above we can see P states are also enabled on this system. P states makes turbo mode work when something requires extra performance, but doesn't need all cores. Many systems tend however to come with only C states enabled. The information seen from the vSphere client does not reveal the level of C states that are configured.  C states does not always have a severe impact, but since all systems I have seen so far come with all C states enabled it will normally affect the performance if you see it in the vSphere client.

The Performance BP doc says the following:


The SQL BP doc says:
Both these documents agree on that disabling C states is the way to go (The OS Control mode setting in BIOS typically disables C states and enables P states). Earlier versions of this document have suggested disabling saving functionality completely. 

The document Best Practices for Performance Tuning of Latency-Sensitive Workloads in vSphere Virtual Machines also suggest to set "Power Management Mode to Maximum Performance" in BIOS, disabling Power Management completely.


The the good old .Net based vSphere client it was only possible to change the settings if either C or P states were available. In the HTML5 client you can set options here even if they are disabled, which doesn't really make much sense.

Many new servers now also come with a virtualization adapted predefined power scheme that you can choose in BIOS.

When you buy servers today it's also possible to specify to make this setting the default one and then all the servers will come correctly preconfigured.



Problem
Even if all your servers at one point had the BIOS settings set to Full Performance, you may at a later point see that not all servers perform equally good. I have seen that replacement of motherboards will normally lead to a Balanced power saving setting (and degraded performance).

Solution
With Powershell you can easily identify servers that has C or P states enabled.
Get-VMHost | Sort | Select Name,
    @{ N='HW Support';
       E={$_.ExtensionData.Hardware.CpuPowerManagementInfo.HardwareSupport}}


When you run it agains your clusters it will tell you if the ESXi hosts has any of these power states enabled in the BIOS:

C:\> get-biospowersettings.ps1
Name                          HW Support
----                          ----------
esxa001.mydomain.com          ACPI C-states
esxa002.mydomain.com          
esxb001.mydomain.com          ACPI P-states
esxb002.mydomain.com          ACPI P-states
esxc044.mydomain.com          ACPI C-states
esxc047.mydomain.com          ACPI C-states

As we can see from this output one of the host has no output. This means that it has power saving disabled in the BIOS. The ones with C states probably has a default setting of Balanced (gives poor performance) and the ones with P states have probably been manually configured to take advantage of cpu Turbo modes. For most workloads (and lowest latency) you will probably want to disable power saving in the BIOS and have a blank result here.

February 24, 2019

Quick boot and hba driver updates

Background
vSphere 6.7 introduced a new feature called Quick Boot that reboots ESXi of certain server models without going into POST. This can save quite a bit of time as many servers can spend several minutes during POST. In clusters with many servers this means we can save hours or even days during a year of updates and reboots.

Problem
When using vSAN it's very important that you follow the HCL in regards to using supported firmware+driver versions. We had a newly installed vSAN cluster with servers in a non-supported state. Because of this we used Update Manager to deploy a supported driver. The driver was installed, but the server never came up again. When we checked the console of the first host we saw the error message: "LoadESX in progress".
If we reset the power of the server it would complete POST and boot up with the new driver installed.

Solution
Disabling Quick Boot (from the Update Manager configuration) solved this problem and we were able to get all servers in our cluster up to date without any problems.

September 7, 2018

"Status of other host hardware objects" on HPE Gen10 servers

Problem
On HPE Gen10 servers we have observed several error messages in hardware monitoring and VMware vCenter reports these as a critical problem.

Device IO Module 4 NIC_Link_01P4
This problem has been reported on both vSphere 6.5 and 6.7.

Solution
Update 21Nov2018: This issue has been fixed in ILO version 1.37: https://vmoller.dk/index.php/2018/11/17/lom-warning-in-vmware-on-hpe-gen10-servers/

It turned out that the servers that had this problem were using ILO version 1.30 and 1.35 while the servers that did not have this problem were using ILO 1.20. After downgrading ILO to version 1.20 the problem was resolved. Hopefully this will be fixed in a future version.


Downgrading ILO is a new feature of ILO 5:
The components can be organized in to install sets and can be used to rollback/Patch faulty firmware
Note that while you can upgrade ILO directly with the Update Firmware functionality, you can't downgrade it the same way. In order to be able to downgrade ILO you must upload it to the ILO Repository first. Once it has been uploaded you can downgrade the ILO firmware.







After having downgraded ILO you will need to go into each of the ESXi host's hardware status and press the reset sensors button and everything will be fine.

June 15, 2018

LLDP not available on Intel X710 running ESXi 6.5U1

Problem
While setting up 6.5U1 on new HPE DL380 Gen10 servers we could not get LLDP working. We had the error message "Link Layer Discovery Protocol is not available on this physical network adapter."

It looks like the X710 card is doing LLDP in hardware, but I'm not sure why that could be a problem.


Solution
The solution to this problem is as follows:

  1. Upgrade firmware the firmware as provided in Service Pack for ProLiant (SPP) Version 2018.03.0, where the intel firmware version 6.0.1 is provided.
  2. The X710 driver needs to be on version 1.5.6 which is available on the HPE Custom Image for ESXi 6.5U1
  3. You also need to run the following command on each host:  esxcli system module parameters set -m i40en -p LLDP=0,0,0,0    where the number of zeros is the number of x710 interfaces in your system.
  4. Reboot