Let me start by saying that in general HA should never be disabled. The later versions of vSphere have a neat option called “Enable Host Monitoring”. This option should be used for scheduled network maintenance. The difference between disabling host monitoring and disabling HA is that disabling host monitoring does not cause a full reconfiguration (see screenshot below) of HA and a new election process. Just the “host monitoring” functionality is disabled, which is what you want in this scenario.

Bilal asked multiple questions / made multiple statements in his article, I will respond to two of these specifically to explain the way HA handles failures/isolation:

In this case within 30 sec of the management network outage, each host would have declared itself isolated and wont attempt to restart any VMs like the primaries would in vSphere 5.

So why is this? As soon as a Master is isolated it will drop “ownership” of datastores on which VMs are running that are part of its cluster. Before the other hosts trigger the isolation response for a given VM they will validate if the datastore on which this VM is stored is “owned” by a master. In the case of a cluster wide isolation due to a network outage / maintenance the ownership would be dropped and this would result in HA not triggering the isolation response. This is a major change compared to vSphere 4.x and prior!

Now what happens when the network outage is over and the hosts are in a position to talk to each other? I have not been able to find documentation on whether an isolated host will enter an election (vSphere 4 or 5) ones the communication channel is open and bring the cluster back to life.

Lets focus on vSphere 5.0 as that seems most relevant. A host remains isolated until it observes HA network traffic, like for instance election messages OR it starts getting a response from an isolation address. Meaning that as long as the host is in “isolated state” it will continue to validate its isolation by pinging the isolation address. As soon as the isolation address responds it will initiate an election process or join an existing election process and the cluster will return to a normal state.

There’s absolutely no need to manually intervene. HA takes care of all of this for you.

Another thing to check if you experience this error is to see if you have jumbo frames enabled on the management network, since this interferes with HA communication.

This is document here in a note: http://kb.vmware.com/kb/2006729

To make it crystal clear: disable jumbo frames on your management network with vSphere 5.0 as there’s a problem with it! This problem is currently being investigated by the HA engineering team and will hopefully be resolved.

<Update> Just received an email that all the cases where we thought vSphere HA issues were caused by Jumbo Frames being enabled were actually caused by the fact that it was not configured correctly end-to-end. Please validate Jumbo Frame configuration on all levels when configuring. (Physical Switches, vSwitch, Portgroup, VMkernel etc)</Update>

Check it out, and let me know if you like it by leaving a comment… this is a trial for us as well, we might end up enabling it on all books in the future.

After an upgrade, under conditions we’re still investigating, an error is occurring when issuing a start request of the HA service on the upgraded host.  When that fails, HA then tries to re-install HA, and the re-install does nothing because the service is already there (and the right version) but we’re left without an HA service running.

This is the way to fix it if you are experiencing this issue. Now, if you do experience this issue please report it to VMware and submit log files as that will help the HA team fixing the problem.

1. Place host into Maintenance Mode
2. Take a copy of /opt/vmware/uninstallers/VMware-fdm-uninstall.sh (we copied to /tmp)
3. From the location you made a copy of the file, run the command (./VMware-fdm-uninstall.sh)
4. You should see a short pause before it gets back to the prompt (you’ll see why I mention this below)
5. Exit host out of Mainenance Mode and within the “Recent Tasks” area you should see the client being pulled from vCenter and installing

What happened was that the HA Isolation Response was configured to “Leave Powered On” and as both the Management Network and the iSCSI Network were isolated there was no “datastore heartbeating” and no “network heartbeating”. Because the datastores were unavailable the lock on the VMDKs expired and HA would restart the VMs. Although HA will power off the “ghosted VM” which lost the lock when it detects the lock cannot be re-acquired, this will happen when the lock cannot be re-acquired. This means that the time between when the restart happens and the isolation is solved the IP Address and the Mac Address of the VM will pop up on the network and as you can imagine this is not desired.

Relocating from cs-tkmt-h08, emc-vnx-fcoe to cs-tkmt-h05, emc-vnx-fcoe

In other words, the VM is relocated from my ESXi host cs-tkmt-h08 to cs-tkmt-h05. No need to worry about orphaned VMs and manually registering them against the remaining hosts… vSphere does it for you.

1. Power Outage, all hosts down
2. Power on hosts
3. Election process will be kicked off. Master will be elected.
4. Master reads protected list
5. Master initiates restarts for those VMs which were listed as protected but not running

Now the one thing I want to point out is that with vSphere 5.0 we will also track if the VM was cleanly powered off, as in initiated by the admin, or powered-off due to a failure/isolation. In the case they are cleanly powered off they will not be restarted, but in this scenario of course they are not cleanly powered off and as such the VMs will be powered on. The great thing about vSphere 5.0 is that you no longer need to know which hosts where your primary nodes so you can power these on first to ensure quick recovery… No, you can power on any host and HA will sort it out for you.

If you look at how HA works and what HA is intended to do the answer in short is, No. Now the reason for this is that HA is all about getting your virtual machines up and running again. If you look at HA Admission Control in vSphere 5.0 you will quickly see that for instance the default value for CPU has been decreased from 256MHz to 32MHz, if no CPU reservations are specified that is. Now in many scenarios virtual machines will consume and demand more than that. Another thing to point out is that if no memory reservation is specified the memory overhead of the VM is used. These values are more than likely much lower than what your virtual machine currently consumes or demands. The thing to keep in mind is that these CPU and Memory values only represent what HA needs in order to power-on your virtual machines.

If you want to manage resources, avoid severe overcommitment, guarantee a certain experience you should start looking at the DRS statistics. You should start exploring tools like VC Ops, Cap IQ… Don’t (ab)use vSphere HA for this. It is not designed to solve this problem. One thing to think about though is maybe increasing the minimum value for slotsizes to avoid scenarios where environments are fully overloaded!? If you have a consolidation ratio in mind it should be fairly simple to figure out which value to use:

available memory esource per host / consolidation ratio = das.vmMemoryMinMB
or
available CPU esource per host / consolidation ratio = das.vmCpuMinMHz

I am not saying that you should do this, but I think it might not be a bad practice in environments where multiple people have access to vCenter and can deploy VMs. At least people will be triggered when you are running out of “slots” to start VMs.

http://kb.vmware.com/kb/2008203

To power-on a VM you need unreserved memory capacity! The unreserved memory capacity needs to be equal to the memory reservation of the VM and the memory overhead! In other words, if you set no memory reservation you can power-on multiple 96GB VMs on a 48GB host. Just because the memory overhead is much lower than 48GB of memory. Now this doesn’t mean it is a best practice, or this is something I would recommend, but it does mean that if you look at how HA handles a fail-overs it will accommodate the restart of these virtual machine. This also means that with regards to HA Admission Control, chances of not being able to power-on your virtual machine because of insufficient resources are fairly slim. I bet that if you over-commit to such an extent that a power-on operation is impossible you have a lot more challenges to begin with!

Frank Denneman wrote a nice article about this a while back, it explains perfectly what the impact is of a memory reservation.