Yellow Bricks

by Duncan Epping

u3

Which vSAN Maintenance Mode option was used on a host?

· · 4 Comments

Last week there was a question on VMTN around maintenance mode, this customer wanted to find out which vSAN Maintenance Mode option was used while the host was placed in maintenance mode. The person who placed the host into maintenance wasn’t around, and I guess the customer wanted to know if data was moved from the host or not. They looked at the events and tasks, but unfortunately it wasn’t obvious from that vCenter section, next up were the logs. I also looked at the logs, and you would expect this info to be stored in hostd.log, but it wasn’t, so where is it? Well it actually is a configuration item, and you can dig it up here:

Pre ESXi 7.0 the info is stored in the “esx.conf” file, just grep for “hostDecommission”. Let me show you:

$ grep "/vsan/hostDecommission" /etc/vmware/esx.conf
/vsan/hostDecommissionVersion = "10"
/vsan/hostDecommissionState = "decom-state-decommissioned"
/vsan/hostDecommissionMode = "decom-mode-ensure-object-accessibility"

If the ESX is at 7.0 or later, just run the below config store command:

$ configstorecli config current get -c vsan -g system -k host_state
{
   "decom_mode": "ENSUREOBJECT_ACCESSIBILITY",
   "decom_state": "DECOMMISSIONED",
   "decom_version": 0
}

I hope that helps others who have a similar question!

vGPUs and vMotion, why the long stun times?

· · 2 Comments

Last week one of our engineers shared something which I found very interesting. I have been playing with Virtual Reality technology and NVIDIA vGPUs for 2 months now. One thing I noticed is that we (VMware) introduced support for vMotion in vSphere 6.7 and support for vMotion of multi vGPU VMs in vSphere 6.7 U3. In order to enable this, you need to set an advanced setting first. William Lam described this in his blog how to set this via Powershell or the UI. Now when you read the documentation there’s one thing that stands out, and that is the relatively high stun times for vGPU enabled VMs. Just as an example, here are a few potential stun times with various sized vGPU frame buffers:

  • 2GB – 16.5 seconds
  • 8GB – 61.3 seconds
  • 16GB – 100+ seconds (time out!)

This is all documented here for the various frame buffer sizes. Now there are a couple of things to know about this. First of all, the time mentioned was tested with 10GbE and the NVIDIA P40. This could be different for an RTX6000 or RTX8000 for instance. Secondly, they used a 10GbE NIC. If you use multi-NIC vMotion or for instance a 25GbE NIC than results may be different (times should be lower). But more importantly, the times mentioned assume the full frame buffer memory is consumed. If you have a 16GB frame buffer and only 2GB is consumed then, of course, the stun time would be lower than the above mentioned 100+ seconds.

Now, this doesn’t answer the question yet, why? Why on earth are these stun times this long? The vMotion process is described in this blog post by Niels in-depth, so I am not going to repeat it. It is also described in our Clustering Deep Dive book which you can download here for free. The key reason why with vMotion the “down time” (stun times) can be kept low is that vMotion uses a pre-copy process and tracks which memory pages are changed. In other words, when vMotion is initiated we copy memory pages to the destination host, and if a page has changed during that copy process we mark it as changed and copy it again. vMotion does this until the amount of memory that needs to be copied is extremely low and this would result in a seamless migration. Now here is the problem, it does this for VM memory. This isn’t possible for vGPUs unfortunately today.

Okay, so what does that mean? Well if you have a 16GB frame buffer and it is 100% consumed, the vMotion process will need to copy 16GB of frame buffer memory from the source to the destination host when the VM is stunned. Why when the VM is stunned? Well simply because that is the point in time where the frame buffer memory will not change! Hence the reason this could take a significant number of seconds unfortunately today. Definitely something to consider when planning on using vMotion on (multi) vGPU enabled VMs!

vCenter tasks time-out or ESX disconnects?

· ·

I just received an email from a fellow consultant about a customer which had vCenter tasks time-out every once in a while. At times also ESX hosts got disconnected for no apparent reason at all. He discovered the following article by Richard Blythe aka VMware Wolf: ESX disconnects randomly or when doing VI client tasks from VC, task randomly timeout after a long idle time. Richard created a list of issues/errors that might be related to this issue:

  • ESX disconnects randomly from VirtualCenter
  • ESX disconnects when performing VI Client tasks from VirtualCenter.
  • Tasks randomly timeout after a long idle time
  • “An error occurred communicating to the remote host” pops up.

The article refers to an issue with vCenter Update 3 in combination with firewalls using state-ful inspection. The problem occurs because of SOAP timeouts, and this behavior did not exist in VC 2.0.x or 2.5 GA, as they used a different mechanism to communicate with ESX. The official KB article hasn’t been released yet but a temporary workaround has been published by Richard. If you run into any of the before mentioned issues head over to Richard’s website and try out the workaround until the fix or official KB article is released.

Scripted installs and nic teaming

· ·

As of ESX 3.5 it was impossible to add an additional NIC to a team as active without resorting to editing the esx.conf file:

# Active and standby setup and maxActive from 1 to 2
mv /etc/vmware/esx.conf /tmp/esx.conf.bak
/bin/sed -e ’s/net\/vswitch\/child\[0001\]\/teamPolicy\/maxActive = \”1\”/net\/vswitch\/child\[0001\]\/teamPolicy\/maxActive = \”2\”/g’ /tmp/esx.conf.bak >> /etc/vmware/esx.conf

So as you can see, a “sed” command changed the maxActive from 1 to 2. But I rather not use these kinds of solutions, editing the esx.conf that is. As of ESX 3.5 U3 that’s not necessary anymore, VMware fixed this issue:

Network adapters lose bonding during scripted installation
The esxcfg-vswitch -L command now works as expected and with the same functionality as in 3.0.x.

During a scripted installation, the following two commands did not result in a bonded pair of active network adapters on virtual switch VS_VM1. Instead, vmnic3 became the active adapter and vmnic4 became the standby adapter.
esxcfg-vswitch -L vmnic3 VS_VM1
esxcfg-vswitch -L vmnic4 VS_VM1

So just use esxcfg-vswitch again and don’t edit the esx.conf anymore!

VMware HA implementations notes

· ·

As most of you know I work for VMware which means I receive a huge amount of email and documents with excellent technical info. Some I can’t blog about but some I can. Last week I received this VMware HA implementations notes document by Seva Semouchin. Seva is a Technical Account Manager based in Germany!

This document deals about VMware HA clusters and all the advanced options you have with the use cases for these options. And that’s what makes this document very useful, the use cases! So everyone dealing with HA clusters should definitely read this document. Thanks Seva for letting me distribute it, and thanks for a great document!