My posts on VMware High Availability(HA) have “historically” been my best read posts next to the “ESXi on USB” posts. I decided to rewrite the posts and create a page which is easier to maintain when functionality changes and a lot easier to find via Google and or the menu.

Everybody probably knows the basics of VMware HA so I’m not going to explain how to set it up or that is uses a heartbeat for monitoring outages or isolation.

However I do want to explain the different kind of nodes, roles, isolation response, gotchas, admission control/slots and some advanced settings. Here we go…

Primary and Secondary nodes

A VMware HA Cluster consists of nodes, primary and secondary nodes. Primary nodes hold cluster settings and all “node states” which are synchronized between primaries. Node states hold for instance resource usage information. In case that vCenter is not available the primary nodes will have a rough estimate of the resource occupation and can take this into account when a fail-over needs to occur. Secondary nodes send their state info to the primary nodes.

Nodes send a heartbeat to each other, which is the mechanism to detect possible outages. Primary nodes send heartbeats to primary nodes only. Secondary nodes send their heartbeats to primary nodes only. Nodes send out these heartbeats every second by default. However this is a changeable value: das.failuredetectioninterval. (Advanced Settings on your HA-Cluster)

The first 5 hosts that join the VMware HA cluster are automatically selected as primary nodes.  All the others are automatically selected as secondary nodes. When you do a reconfigure for HA the primary nodes and secondary nodes are selected again, this is at random. The vCenter client does not show which host is a primary and which is not. This however can be revealed from the Service Console:

cat /var/log/vmware/aam/aam_config_util_listnodes.log

Another method of showing the primary nodes is:

/opt/vmware/aam/bin/Cli (ftcli on earlier versions)
AAM> ln

A common misconception is that a re-election occurs when a primary node fails. This is not the case. The promotion of a secondary host only occurs when a primary host is either put in “Maintenance Mode”, disconnected from the cluster or removed from the cluster.

If all primary hosts fail simultaneously no HA initiated restart of the VMs will take place. HA needs at least one primary host to restart VMs. This is why you can only take four host failures in account when configuring HA. (Remember 5 primaries…)

This leads to a basic design rule in blade environments, divide hosts over all blade chassis and never exceed four hosts per chassis to avoid having all primary nodes in a single chassis.

The fail-over coordinator aka “active primary”

You will need at least one primary because the “fail-over coordinator” role will be assigned to this primary, this role is also described as “active primary”. I will use “fail-over coordinator” for now. The fail-over coordinator coordinates the restart of VMs on the remaining primary and secondary hosts. The coordinator takes restart priorities in account. Keep in mind, when two hosts fail at the same time it will handle the restart sequentially. In other words, restart the VMs of the first failed host (taking restart priorities in account) and then restart the VMs of the host that failed as second (again taking restart priorities in account). If the fail-over coordinator fails one of the other primaries will take over.

Isolation Response

Talking about HA initiated fail-overs; one of the settings everyone has looked into is the “isolation response”. The isolation response refers to the action that HA takes when the heartbeat network is isolated. Today there are three isolation responses, “power off”, “leave powered on” and “shut down”.

Up to ESX 3.5 U2 / vCenter 2.5U2 the default isolation response when creating a new cluster was “Power off”. As of ESX 3.5 U3 / vCenter 2.5 U3 the default isolation response is “leave powered on”. For vSphere ESX / vCenter 4.0 this has been changed to “Shut down”. Keep this in mind when installing a new environment, you might want to change the default depending on customer requirements.

Power off – When a network isolation occurs all VMs are powered off. It is a hard stop.

Shut down – When a network isolation occurs all VMs running on that host are shut down via VMware Tools. If this is not successful within 5 minutes a “power off” will be executed.

Leave powered on – When a network isolation occurs on the host the state of the VMs remains unchanged.

The question remains, which setting should I use? It depends. I personally prefer “Shut down” because I do not want to use a deprecated host and it will shut down your VMs cleanly. Many people prefer to use “Leave powered on” because it reduces the chances of a false positive. A false positive in this case is an isolated heartbeat network but a non-isolated VM network.

I guess most of you would like to know how HA knows if the host is isolated or completely unavailable when you have selected “leave powered on”.

HA actually does not know the difference. HA will try to restart the affected VMs in both cases. When the host has failed a restart will take place, but if a host is merely isolated the non-isolated hosts will not be able to restart the affected VMs. This is because of the VMDK file lock; no other host will be able to boot a VM when the files are locked. When a host fails this lock starves and a restart can occur.

The amount of retries is configurable as of vCenter 2.5 U4 with the advanced option “das.maxvmrestartcount”. The default value is 5. Pre vCenter 2.5 U4 HA would keep retrying forever which could lead to serious problems as described in the KB article.

The isolation response is a setting that needs to be taken into account when you create your design. For instance when using an iSCSI array choosing “leave powered on” as your default isolation response could lead to a split-brain situation because the disk lock times out when the iSCSI network fails. In this case the VM is being restarted while it is not being powered off on the original host. For iSCSI the preferred isolation response always is “Power off”.

Isolation gotcha

I thought this issue was something that was common knowledge but a recent blog article by Mike Laverick proved me wrong.

The default value for isolation/failure detection is 15 seconds. In other words the failed or isolated host will be declared dead by the other hosts in the HA cluster on the fifteenth second and a restart will be initiated by one of the primary hosts.

For now let’s assume the isolation response is “power off”. The “power off”(isolation response) will be initiated by the isolated host 2 seconds before the das.failuredetectiontime. A “power off” will be initiated on the thirteenth second and a restart will be initiated on the fifteenth second.

Does this mean that you can end up with your VMs being down and HA not restarting them?
Yes, when the heartbeat returns between the 13th and 15th second the “power off” could already have been initiated. The restart however will not be initiated because the heartbeat indicates that the host is not isolated anymore.

How can you avoid this?
Pick “Leave VM powered on” as an isolation response. Increasing the das.failuredetectiontime will also decrease the chances of running in to issues like these.

Slot sizes/Admission Control

This has always been a hot topic, HA and Slot sizes/Admission Control. One of the most extensive (Non-VMware) articles is by Chad Sakac aka Virtual Geek, but of course since then a couple of things has changed. Chad commented on this article and asked if I could address this topic, here you go Chad.

Lets start with the basics.

What’s HA admission control about? Why is it there? The “Availability Guide” states the following:

vCenter Server uses admission control to ensure that sufficient resources are available in a cluster to provide failover protection and to ensure that virtual machine resource reservations are respected.

To enable admission control we use a concept called “slots”. Slots dictate how many VMs can be started up before vCenter starts yelling “Out Of Resources”!! Normally each slot represents one VM and a slot is….

What is a slot?

A slot is a logical representation of the memory and CPU resources that satisfy the requirements for any powered-on virtual machine in the cluster.

In other words a slot size is the worst case CPU and Memory reservation scenario in a cluster. This directly leads to the first “gotcha”:

HA uses the highest CPU reservation of any given VM and the highest memory reservation of any given VM. If no reservations of higher than 256Mhz are set HA will use a default of 256Mhz for CPU and a default of 0MB+memory overhead for memory.

If VM1 has 2GHZ and 1024GB reserved and VM2 has 1GHZ and 2048GB reserved the slot size for memory will be 2048MB+memory overhead and the slot size for CPU will be 2GHZ.

Now how does HA calculate how many slots are available per host?

Of course we need to know what the slot size for memory and CPU is first. Then we divide the total available CPU resources of a host by the CPU slot size and the total available Memory Resources of a host by the memory slot size. This leaves us with a slot size for both memory and CPU. The most restrictive number is the amount of slots for this host. If you have 25 CPU slots but only 5 memory slots the amount of available slots for this host will be 5.

As you can see this can lead to very conservative consolidation ratios. With vSphere this is something that’s configurable. If you have just one VM with a really high reservation you can set the following advanced settings to lower the slot size being used during these calculations: das.slotCpuInMHz or das.slotMemInMB. To avoid not being able to power on the VM with high reservations these VM will take up multiple slots. Keep in mind that when you are low on resources this could mean that you are not able to power-on this high reservation VM as resources are fragmented throughout the cluster instead of located on a single host.

If you need to use a high reservation for either CPU or Memory these options (das.slotCpuInMHz or das.slotMemInMB) could definitely be useful, there is however something that you need to know. Check this diagram and see if you spot the problem, the das.slotMemInMB has been set to 1024MB.

Notice that the memory slotsize has been set to 1024MB. VM24 has a 4GB reservation set. Because of this VM24 spans 4 slots. As you might have noticed none of the hosts has 4 slots left. Although in total there are enough slots available; they are scattered and HA might not be able to actually boot VM24. Keep in mind that admission control does not take scattering of slots into account. It does count 4 slots for VM24, but it will not verify the amount of available slots per host.

Another issue that needs to be discussed is “Unbalanced clusters”. Unbalanced would for instance be a cluster with 5 hosts of which one contains substantially more memory than the others. What would happen to the total amount of slots in a cluster of the following specs:

Five hosts, each host has 16GB of memory except for one host(esx5) which has recently been added and has 32GB of memory.

One of the VMs in this cluster has 4CPUs and  4GB of memory, because there are no reservations set the memory overhead of 325MB is being used to calculate the memory slot sizes. (It’s more restrictive than the CPU slot size.)

This results in 50 slots for esx01, esx02, esx03 and esx04. However, esx05 will have 100 slots available. Although this sounds great admission control rules the host out with the most slots as it takes the worst case scenario into account. In other words; end result: 200 slot cluster.

With 5 hosts of 16GB, (5 x 50) – (1 x 50), the result would have been exactly the same. To make a long story short: balance your clusters when using admission control and think before you act!

Host Failures?

Now what happens if you set the number of allowed host failures to 1?
The host with the most slots will be taken out of the equation. If you have 8 hosts with 90 slots in total but 7 hosts each have 10 slots and one host 20 this single host will not be taken into account. Worst case scenario! In other words the 7 hosts should be able to provide enough resources for the cluster when a failure of the “20 slot” host occurs.

And of course if you set it to 2 the next host that will be taken out of the equation is the host with the second most slots and so on.

What more?

One thing worth mentioning, as Chad stated with vCenter 2.5 the number of vCPUs for any given VM was also taken in to account. This led to a very conservative and restrictive admission control. This behavior has been modified with vCenter 2.5 U2, the amount of vCPUs is not taken into account.

Avoiding Slot Size issues!

Can I avoid large HA slot sizes due to reservations without resorting to advanced settings? That’s the question I get almost daily. The answer used to be NO. HA uses reservations to calculate the slot size and there’s no way to tell HA to ignore them without using advanced settings pre-vSphere. So there is your answer: pre-vSphere.

With vSphere VMware introduced a percentage next to an amount of host failures. The percentage avoids the slot size issue as it does not use slots for admission control. So what does it use?

When you select a specific percentage that percentage of the total amount of resources will stay unused for HA purposes. First of all VMware HA will add up all available resources to see how much it has available. Then VMware HA will calculate how much resources are currently consumed by adding up all reservations of both memory and cpu for powered on virtual machines. For those virtual machines that do not have a reservation larger than 256Mhz a default of 256Mhz will be used for CPU and a default of 0MB+memory overhead will be used for Memory. (Amount of overhead per config type can be found on page 28 of the resource management guide.)

In other words:

((total amount of available resources – total reserved VM resources)/total amount of available resources)
Where total reserved VM resources include the default reservation of 256Mhz and the memory overhead of the VM.

Let’s use a diagram to make it a bit more clear:

Total cluster resources are 24Ghz(CPU) and 96GB(MEM). This would lead to the following calculations:

((24Ghz-(2Gz+1Ghz+256Mhz+4Ghz))/24Ghz) = 69 % available
((96GB-(1,1GB+114MB+626MB+3,2GB)/96GB= 85 % available

As you can see the amount of memory differs from the diagram. Even if a reservation has been set the amount of memory overhead is added to the reservation. For both metrics HA admission control will constantly check if the policy has been violated or not. When one of either two thresholds are reached, memory or CPU, admission control will disallow powering on any additional virtual machines.

Please keep in mind that if you have an unbalanced cluster(host with different CPU or memory resources) your percentage is equal or preferably larger than the percentage of resources provided by the largest host. This way you ensure that all virtual machines residing on this host can be restarted in case of a host failure.

What would I prefer?

It depends. Yes I know, that is the obvious answer but it actually does. There are three options and each have it’s own advantages and disadvantages. Here you go:

• Amount of host failures
  Pros: Fully automated, when a host is added to a cluster HA calculates how many slots are available.
  Cons: Can be very conservative and inflexible when reservations are used as the largest reservation dictates slot sizes.
• Percentage reserved
  Pros: Flexible. Although reservations have its effect on the amount of available resources it impacts the environment less.
  Cons: Manual calculations need to  be done when adding additional hosts in a cluster. Unbalanced clusters can be a problem when chosen percentage is too low.
• Designated failover host
  Pros: What you see is what you get.
  Cons: What you see is what you get.

Advanced settings

VMware HA is probably the feature with the most advanced settings. Although many of them are rarely used some of them are needed in specific situations or included in best practices documents. The most used and valuable advanced settings are described below:

das.failuredetectiontime – Amount of milliseconds, timeout time, for isolation response action (with a default of 15000 milliseconds). It’s a best practice to increase the value to 60000 when an active/standby Service Console setup is used. For a host with two Service Consoles and a secondary isolation address it’s a best practice to increase it to at least 20000. I would recommend to always increase it to at least 30000

das.isolationaddress[x] – IP address the ESX hosts uses to check on isolation when no heartbeats are received, where [x] = 1‐10. VMware HA will use the default gateway as an isolation address and the provided value as an additional checkpoint. I recommend to add an isolation address when a secondary service console is being used for redundancy purposes.

das.usedefaultisolationaddress – Value can be true or false and needs to be set in case the default gateway, which is the default isolation address, should not or cannot be used for this purpose. In other words, if the default gateway is a non-pingable address set the “das.isolationaddress” to a pingable address and disable the usage of the default gateway by setting this to “false”.

das.allowVmotionNetworks – Allows a NIC that is used for VMotion networks to be considered for VMware HA heartbeat usage. This permits a host (ESXi only) to have only one NIC configured for management and VMotion combined.

das.allowNetwork[x] – Enables the use of port group names to control the networks used for VMware HA, where [x] = 0 – ?. You can set the value to be ʺService Console 2ʺ or ʺManagement Networkʺ to use (only) the networks associated with those port group names in the networking configuration. These networks need to be compatible for HA to configure successful.

das.bypassNetCompatCheck – Disable the “compatible network” check for HA that was introduced with Update 2. Default value is “false”, setting it to “true” disables the check. This setting can be useful when nodes in a cluster are not in the same subnet.

das.ignoreRedundantNetWarning – Remove the error icon/message from your vCenter when you don’t have a redundant Service Console connection. Default value is “false”, setting it to “true” will disable the warning.