by Duncan Epping
As I had this question many times by now I figured a short blog article wouldn’t hurt. Yes, the vSphere 4.1 HA/DRS Kindle copy is also free in other amazon shops, but just today… so pick it up now:
I wanted to point you guys to a white paper that I have worked on for the last months. This white paper was written in collaboration with Lee Dilworth, Ken Werneburg, Frank Denneman and Stuart Hardman. Thanks guys for taking time out of your busy schedule to work with me on this project! This white paper is about vSphere Metro Storage Cluster solutions (aka stretched clusters) and specifically looks at things from a VMware perspective. Enjoy!
I was on a holiday the last two weeks and slowly catching up on everything that happened. Some of you might think it wasn’t a lot, but in the world of cloud and virtualization it was. Not only was there a huge EUC launch event but also a new version of vCenter Infrastructure Navigator was released. Somehow it has been amazingly quiet around this product. Something I didn’t really understand, especially not after reading the release notes of version 1.1 of vCenter Infrastructure Navigator. Two things stood out:
For those who don’t know, Infrastructure Navigator is an application awareness plugin for vCenter Server. This enables you to get a better understanding of what is running on top of your virtual infrastructure. A lot of you may say, well why would I care? Think about DR for a second. What is the most challenging part of creating a DR Plan? Indeed, figuring out all dependencies. That is exactly where vCenter Infrastructure Navigator comes in to play as shown in the screenshot below, which I stole from Ben Scheerer. Ben wrote an excellent blog about some of the cool new features in vCenter Infrastructure Navigator, I am not going to repeat those just read his. It is worth it if you are serious about providing the best service to your (internal) customers!
I have had this question multiple times by now so I figured I would write a short blog post about it. The question is if you can share datastores across clusters with vSphere 5.0 and HA enabled. This question comes from the fact that HA has a new feature called “datastore heartbeating” and uses the datastore as a communication mechanism.
The answer is short and sweet: Yes.
For each cluster a folder is created. The folder structure is as follows:
/<root of datastore>/.vSphere-HA/<cluster-specific-directory>/
The “cluster specific directory” is based on the uuid of the vCenter Server, the MoID of the cluster, a random 8 char string and the name of the host running vCenter Server. So even if you use dozens of vCenter Servers there is no need to worry.
Each folder contains the files HA needs/uses as shown in the screenshot below. So no need to worry around sharing of datastores across clusters. Frank also wrote an article about this from a Storage DRS perspective. Make sure you read it!
PS: all these details can be found in our Clustering Deepdive book… find it on Amazon.
I had a question today around what the vSphere HA option advanced setting das.maskCleanShutdownEnabled is about. I described why it was introduced for Stretched Clusters but will give a short summary here:
Two advanced settings have been introduced in vSphere 5.0 Update 1 to enable HA to fail-over virtual machines which are located on datastores which are in a Permanent Device Loss state. This is very specific to stretchec cluster environments. The first setting is configured on a host level and is “disk.terminateVMOnPDLDefault”. This setting can be configured in /etc/vmware/settings and should be set to “True”. This setting ensures that a virtual machine is killed when the datastore it resides on is in a PDL state.
The second setting is a vSphere HA advanced setting called “das.maskCleanShutdownEnabled“. This setting is also not enabled by default and it will need to be set to “True”. This settings allows HA to trigger a restart response for a virtual machine which has been killed automatically due to a PDL condition. This setting allows HA to differentiate between a virtual machine which was killed due to the PDL state or a virtual machine which has been powered off by an administrator.
But why is “das.maskCleanShutdownEnabled” needed for HA? From a vSphere HA perspective there are two different types of “operations”. The first is a user initiated power-off (clean) and the other is a kill. When a virtual machine is powered off by a user, part of the process is setting the property “runtime.cleanPowerOff” to true.
Remember that when “disk.terminateVMOnPDLDefault” is configured your VMs will be killed when they issue I/O. This is where the problem arises, in a PDL scenario it is impossible to set “runtime.cleanPowerOff” as the datastore, and as such the vmx, is unreachable. As the property defaults to “true” vSphere HA will assume the VMs were cleanly powered off. This would result in vSphere HA not taking any action in a PDL scenario. By setting “das.maskCleanShutdownEnabled” to true, a scenario where all VMs are killed but never restarted can be avoided as you are telling vSphere HA to assume that all VMs are not shutdown in a cleanly matter. In that case vSphere HA will assume VMs are killed UNLESS the property is set.
If you have a stretched cluster environment, make sure to configure these settings accordingly!