Yellow Bricks

by Duncan Epping

ha

All host failed, how does HA respond?

· ·

I wrote an article about the scenario where all host fail, due to for instance a power outage, and how HA responds to it. I had a question today if this was still valid with vSphere 5.0. I figured it wouldn’t hurt to describe the steps that vSphere 5.0 takes.

  1. Power Outage, all hosts down
  2. Power on hosts
  3. Election process will be kicked off. Master will be elected.
  4. Master reads protected list
  5. Master initiates restarts for those VMs which were listed as protected but not running

Now the one thing I want to point out is that with vSphere 5.0 we will also track if the VM was cleanly powered off, as in initiated by the admin, or powered-off due to a failure/isolation. In the case they are cleanly powered off they will not be restarted, but in this scenario of course they are not cleanly powered off and as such the VMs will be powered on. The great thing about vSphere 5.0 is that you no longer need to know which hosts where your primary nodes so you can power these on first to ensure quick recovery… No, you can power on any host and HA will sort it out for you.

 

** Disclaimer: This article contains references to the words master and/or slave. I recognize these as exclusionary words. The words are used in this article for consistency because it’s currently the words that appear in the software, in the UI, and in the log files. When the software is updated to remove the words, this article will be updated to be in alignment. **

Managing resources with HA Admission Control?

· ·

Last week at VMworld and on the VMTN community I had a couple of questions around resource management and HA Admission Control. It appears people were using HA Admission Control for managing resources within their environment. In other words, the amount of VMs that HA would allow you to restart would be leading for managing resources. But is that what you should do?

If you look at how HA works and what HA is intended to do the answer in short is, No. Now the reason for this is that HA is all about getting your virtual machines up and running again. If you look at HA Admission Control in vSphere 5.0 you will quickly see that for instance the default value for CPU has been decreased from 256MHz to 32MHz, if no CPU reservations are specified that is. Now in many scenarios virtual machines will consume and demand more than that. Another thing to point out is that if no memory reservation is specified the memory overhead of the VM is used. These values are more than likely much lower than what your virtual machine currently consumes or demands. The thing to keep in mind is that these CPU and Memory values only represent what HA needs in order to power-on your virtual machines.

If you want to manage resources, avoid severe overcommitment, guarantee a certain experience you should start looking at the DRS statistics. You should start exploring tools like VC Ops, Cap IQ… Don’t (ab)use vSphere HA for this. It is not designed to solve this problem. One thing to think about though is maybe increasing the minimum value for slotsizes to avoid scenarios where environments are fully overloaded!? If you have a consolidation ratio in mind it should be fairly simple to figure out which value to use:

available memory esource per host / consolidation ratio = das.vmMemoryMinMB
or
available CPU esource per host / consolidation ratio = das.vmCpuMinMHz

I am not saying that you should do this, but I think it might not be a bad practice in environments where multiple people have access to vCenter and can deploy VMs. At least people will be triggered when you are running out of “slots” to start VMs.

HA & DRS appear disabled when a VM Storage Profile is enabled / disabled

· ·

A couple of weeks ago I was contacted about an issue with vSphere 5.0. When a VM Storage Profile was disabled / enabled it looked like HA and DRS got disabled as well. After diving in to the problem it appeared that both HA and DRS were still working. It seems that some how the vSphere Client thinks it is disabled. Again, although it looks like it is disabled it is still fully functioning… but indeed it is very confusing. A simple work around would be enabling HA/DRS. This problem has just been published in our Knowledge Base, and hopefully will be addressed soon.

http://kb.vmware.com/kb/2008203

Host Admission Control: Powering on a VM…

· ·

I was reading a whitepaper by VKernel and it mentioned the following “a failover host for these VMs requires sufficient idle resources“. In this whitepaper it is discussed how Monster VMs pose challenges for both HA and DRS. As I had a similar question last week at VMworld I figured I would post this. Also because it is fundamental to understand this with regards to HA. Now the thing is, I agree that there is no point in creating large VMs just because you can. Without a doubt do Monster VMs pose challenges with regards to managing resources. However I do want to point out that technically speaking the statement is incorrect.

To power-on a VM you need unreserved memory capacity! The unreserved memory capacity needs to be equal to the memory reservation of the VM and the memory overhead! In other words, if you set no memory reservation you can power-on multiple 96GB VMs on a 48GB host. Just because the memory overhead is much lower than 48GB of memory. Now this doesn’t mean it is a best practice, or this is something I would recommend, but it does mean that if you look at how HA handles a fail-overs it will accommodate the restart of these virtual machine. This also means that with regards to HA Admission Control, chances of not being able to power-on your virtual machine because of insufficient resources are fairly slim. I bet that if you over-commit to such an extent that a power-on operation is impossible you have a lot more challenges to begin with!

Frank Denneman wrote a nice article about this a while back, it explains perfectly what the impact is of a memory reservation.

vSphere 5 HA – Isolation Response which one to pick?

· ·

Last week I did an article about Datastore Heartbeating and the prevention of the Isolation Response being triggered. Apparently this was an eye-opener for some and I received a whole bunch of follow up questions through twitter and email. I figured it might be good to write-up my recommendations around the Isolation Response. Now I would like to stress that these are my recommendations based on my understanding of the product, not based on my understanding of your environment or SLA. When applying these recommendations always validate them against your requirements and constraints. Another thing I want to point out is that most of these details are part of our book, pick it up… the e-book is cheap.

First of all, I want to explain Isolation Response…

Isolation Response is the action HA triggers, per VM, when it is network isolated from the rest of your cluster. Now note the “per VM”, so a host will trigger the configured isolation response per VM, which could be either “power off” or “shutdown”. However before it will trigger the isolation response, and this is new in 5.0, the host will first validate if a master owns the datastore on which the VMs configuration files are stored. If that is not the case then the host will not trigger the isolation response.

Now lets assume for a second that the host has been network isolated but a master doesn’t own the datastore on which the VMs config files are stored, what happens? Nothing happens. Isolation response will not be triggered as the host knows that there is no master which can restart these VMs, in other words there is no point in powering down a VM when it cannot power it on. The host will of course periodically check if the datastore is claimed by a master.

There’s also a scenario where the complete datastore could be unavailable, in the case of a full network isolation and NFS / iSCSI backed storage for instance. In this scenario the host will power off the VM when it has detected another VM has acquired the lock on the VMDK. It will do this to prevent a so-called split brain scenario, as you don’t want to end up with two instances of your VM running in your environment. Keep in mind that in order to detect this lock the “isolation” on the storage layer needs to be resolved. It can only detect this when it has access to the datastore.

I guess there’s at least a couple of you thinking but what about the scenario where a master is network isolated? Well in that case the master will drop responsibility for those VMs and this will allow the newly elected master to claim them and take action if required.

I hope this clarifies things.

Now lets talk configuration settings. As part of the Isolation Response mechanism there are three ways HA could respond to a network isolation:

  1. Leave Powered On – no response at all, leave the VMs powered on when there’s a network isolation
  2. Shutdown VM – guest initiated shutdown, clean shutdown
  3. Power Off VM – hard stop, equivalent to power cord being pulled out

When to use “Leave Powered On”
This is the default option and more than likely the one that fits your organization best as it will work in most scenarios. When you have a Network Isolation event but retain access to your datastores HA will not respond and your virtual machines will keep running. If both your Network and Storage environment are isolated then HA will recognize this and power-off the VMs when it recognizes the lock on the VMDKs of the VMs have been acquired by other VMs to avoid a split brain scenario as explained above. Please note that in order to recognize the lock has been acquired by another host the “isolated” host will need to be able to access the device again. (The power-off won’t happen before the storage has returned!)

When to use “Shutdown VM”
It is recommend to use this option if it is likely that a host will retain access to the VM datastores when it becomes isolated and you wish HA to restart a VM when the isolation occurs. In this scenario, using shutdown allows the guest OS to shutdown in an orderly manner. Further, since datastore connectivity is likely retained during the isolation, it is unlikely that HA will shut down the VM unless there is a master available to restart it. Note that there is a time out period of 5 minutes by default. If the VM has not been gracefully shutdown after 5 minutes a “Power Off” will be initiated.

When to use “Power Off VM”
It is recommend to use this option if it is likely that a host will lose access to the VM datastores when it becomes isolated and you want HA to immediately restart a VM when this condition occurs. This is a hard stop in contrary to “Shutdown VM” which is a guest initiated shutdown and could take up to 5 minutes.

As stated, Leave Powered On is the default and fits most organizations as it prevents unnecessary responses to a Network Isolation but still takes action when the connection to your storage environment is lost at the same time.

 

** Disclaimer: This article contains references to the words master and/or slave. I recognize these as exclusionary words. The words are used in this article for consistency because it’s currently the words that appear in the software, in the UI, and in the log files. When the software is updated to remove the words, this article will be updated to be in alignment. **