das.maskCleanShutdownEnabled is set to true by default

I had a couple of questions on the topic of das.maskCleanShutdownEnabled today. For those who have not read the other articles I wrote about this topic, this is in short what it does and why it was introduced and how I explained it in an email today:

When a virtual machine is powered off (or shut down) by a user a property is set to true named runtime.cleanPowerOff”. To vSphere HA this indicates that the virtual machine was powered off by a user and as such when a host fails it knows that for this virtual machine it doesn’t need to take action. By default this property is set to true. If for whatever reason the virtual machine is killed by ESXi than this property is set to false.

vSphere HA provides the ability to respond to a storage failure (PDL). When a PDL occurs it can kill a virtual machine and then restart the virtual machine. However, runtime.cleanPowerOff” default is “true” and vSphere HA cannot access the datastore (PDL remember) to change the property! So this means if the VM is killed after the PDL, then it won’t be restarted as HA assumes it was cleanly powered off.

This is where das.maskCleanShutdownEnabled comes in to play. By setting this to “true”, vSphere HA assumes that VM is not cleanly powered off. Only when you cleanly power it off the property is set. In other words, In a PDL situation it will now restart the VM even though the datastore was unavailable when the VM was killed!

Back to the original question, what is das.maskCleanShutdownEnabled set to in 5.1 and later? Do you need to set it manually? No you do not, by default it is set to true! So when you configure a cluster, be aware of this… Especially in a stretched cluster environment where a PDL scenario is not unlikely.

** do not forget to also set terminateVMonPDL described in this blog post if you want VMs to be automatically killed when a PDL occurs! **

vSphere HA and VMs per Datastore limit!

I felt I would need to get this out there, as it is not something many seem to be aware off . More and more people are starting to use storage solutions which offer 1 large shared datastore, examples are solutions like Virtual SAN, Tintri and Nutanix. I have seen various folks saying: unlimited number of VMs per datastore, but of course there are limits to everything! If you are planning to build a big cluster (HA enabled), keep in mind that per cluster your limit for a datastore is 2048 powered-on virtual machines! Say what? Yes that is right, per cluster you are limited to 2048 powered-on VMs on a single datastore. This is documented in the Max Config Guide of both vSphere 5.5 and vSphere 5.1. Please note it says datastore and not VMFS or NFS explicitly, this applies to both!

The reason for this today is the vSphere HA poweron list. I described that list in this article, in short: this list keeps track of the power-state of your virtual machines If you need more VMs in your cluster than 2048 you will need to create multiple datastores for now. (More details in the blog post) Do note that this is a known limitation and I have been told that the engineering team is researching a solution to this problem. Hopefully it will be in one of the upcoming releases.

vSphere HA advanced settings, the KB

I’ve posted about vSphere HA advanced settings various times in the past, and let me start by saying that you shouldn’t play around with them unless you have a requirement to do so. But if you do, there is a KB article which I can highly recommend as it lists all the known and lesser known advanced settings. I had the KB article updated with vSphere 5.5 advanced settings yesterday (Thanks KB team for being so responsive!) but it also applies to vSphere 5.0 and 5.1. Recommended read for those who want to get in to the nitty gritty details of vSphere HA.

http://kb.vmware.com/kb/2033250

With a single Datastore can I still use HA’s Datastore heartbeating?

I had a question last week around HA’s datastore heartbeating, the question was if datastore heartbeating still worked if you only have 1 datastore in your environment. I can understand where the question comes from as HA throws this error that you need to have 2 datastores at a minimum for HA datastore heartbeating to function correctly. I want to point out that even though HA says that 2 datastores is the minimum, even when only one datastore is available it will be used for heartbeat purposes. Yes this error will be there on your cluster, and yes you can suppress it using “das.ignoreInsufficientHbDatastore“. I figured others might be hitting the same error and have the same question so why not document it?!