In part one I described what HA Admission Control is and in part two I will explain what your options are when admission control is enabled. Currently there are three admission control policies:
- Host failures cluster tolerates
- Percentage of cluster resources reserved as failover spare capacity
- Specify a failover host
Each of these work in a slightly different way. And lets start with “Specify a failover host” as it is the most simple one to explain. This admission control policy allows you to set aside 1 host that will only be used in case a fail-over needs to occur. This means that even if your cluster is overloaded DRS will not use it. In my opinion there aren’t many usecases for it, and unless you have very specific requirements I would avoid using it.
The most difficult one to explain is “Host failures cluster tolerates” but I am going to try to keep it simple. This admission control policy takes the worst case scenario in to account, and only the worst case scenario, and it does this by using “slots”. A slot is comprised of two components:
- Memory
- CPU
For memory it will take the largest reservation on any powered-on virtual machine in your cluster plus the memory overhead for this virtual machine. So if you have one virtual machine that has 24GB memory provisioned and 10GB out of that is reserved than the slot size for memory is ~10GB (reservation + memory overhead).
For CPU it will take the largest reservation on any powered-on virtual machine in your cluster, or it will use a default of 32MHz (5.0, pre 5.0 it was 256MHz) for the CPU slot size. If you have a virtual machine with 8 vCPUs assigned and a 2GHz reservation then the slot size will be 2GHz for CPU.
HA admission control will look at the total amount of resources and see how many “memory slots” there are by dividing the total amount of memory by the “memory slot size”. It will do the same for CPU. It will calculate this for each host. From the total amount of available memory and CPU slots it will take the worst case scenario again, so if you have 80 memory slots and 120 CPU slots then you can power on 80 VMs… well almost, cause the number of slots of the largest hosts is also subtracted. Meaning that if you have 5 hosts and each of those have 10 slots for memory and CPU instead of having 50 slots available in total you will end up with 40.
Simple right? So remember: reservations –> slot size –> worst case. Yes, a single large reservation could severely impact this algorithm!
So now what? Well this is where the third admission control policy comes in to play… “Percentage of cluster resources reserved as failover spare capacity”. This is not a difficult one to explain, but again misunderstood by many. First of all HA will add up all available resources to see how much it has available. It will now subtract the amount of resource specified for both memory and CPU. Then HA will calculate how much resources are currently reserved for both memory and CPU for powered-on virtual machines. For CPU, those virtual machines that do not have a reservation larger than 32Mhz a default of 32Mhz will be used. For memory a default of 0MB+memory overhead will be used if there is no reservation set. If a reservation is set for memory it will use the reservation+memory overhead.
That is it. Percentage based looks at “powered-on virtual machines” and its reservation or uses the above mentioned defaults. Nothing more than that. No. it doesn’t look at resource usage / consumption / active etc. It looks at reserved resources. Remember that!
What do I recommend? I always recommend using the percentage based admission control policy as it is the most flexible policy. It will do admission control on a per virtual machine reservation basis without the risk of skewing the numbers.
If you have any questions around this please don’t hesitate.
For those who still have not bought the book… do yourselves a favor and buy it. The book explains these with in depth include the impact of adding\removing hosts in a configured cluster.
However, the percentage based setting still confusing for me when calculating the adding a new host.
The percentage based setting is confusing for lots of people when trying to calculate what the figure should be when adding (or removing a host) from a cluster.
The Host based one is easier – you decide how many host failures you are willing to tollerate, which is fairly simple if all the hosts in your cluster are the same size (memory wise and compute wise).
With all hosts in the cluster the same size (which they generally should be), the Host based option I think is easier to understand. For ‘uneven’ hosts (like I have in my home lab), the percentage figure is easier to wrap your head around.
@Graham: I know, hence the reason I suggested a hybrid version of those two where the amount of hosts are specified but the slot algo is not used.
Hi Duncan,
Again a sweet article, thanks!
Most of the vSphere environments I’ve seen don’t have any reservations configured on the VMs. Does that mean that “Percentage of cluster resources reserved” is useless here? Because HA admission control will just calculate 0MB+memory overhead and it’ll result in a current memory capacity percentage that is way to large.
Also, are resource pools reservations taken into account?
same thing here…maybe for testing enviroments or I have not worked enough with vmware since none of my customer use reservations or limits so I always use percentage based
Hi Duncan, Thanks for the article. Though I did understand Admission Control and its configuration settings, Still need bit clarity on how it comes into play during real HA failover scenario ( I have read ur Ha deep dive book, but couldnt actually clarify these doubts in depth. Also tried to get this clarified with many of Vmware Admins, But surprisingly no one has got a clear answer for this)
Would be great if you could find some time to explain in detail for the following queries
There are 4 ESXi Hosts in the network and 4 VMs (Same CPU, RAM Reservation for all VMs) on each Host. Admission Control is policy is set to ‘Host failure cluster tolerates’ to 1. All the available 12 slots have been used by the powered ON VMs, except the 4 reserved slots for failover.
1A) What happens if 2 ESXi Hosts fails now? ( 2 * 4 VMs needs to fail over)
1B) Will HA restart only 4 VMs as it has only 4 slots available? And Restart of the remaining 4 VM fails?
Same Scenario, but Policy is set to ‘% of cluster resources reserved’ = 25%. All the available 75 % resources have been utilized by all the 16 VMs, except 25 % reserved for failover
2A) What happens if 2 ESXi Hosts fails now? ( 2 * 4 VMs needs to fail over)
2B) Will HA restart only 4 VMs as it consumes 25 % of resources? And Restart of the other 4 VM fails?
3) Does HA check the VM reservation (or any other factor) at the time of restart ?
4) HA only restart a VM if the Host could guarantee the reserved resources or restart Fails?
5) What if no VM reservations are set VM level ?
5A)What does HA takes into consideration when it has to restart VMs which has no reservation ?
5B)Will it guarantee the configured Resources for each VMs ?
5C)If not, How HA can restart 8 VMs (as per our eg) when it only has configured reserved resources for just 4 VM
5D)Will it share the reserved resources across 8 VMs and will not care about the resource crunch or is it about first come first serve
5E)Admission control doesn’t have any role at all in the event of HA failover ?
I answered them here: http://www.yellow-bricks.com/2012/07/03/answering-some-admission-control-questions/
Thanks a Ton Duncan 🙂 it answered more of the questions floating around on Admission control
It seems the default slot size is 256 MHz again in 5.1?
Old thread I know, Duncan. But tracking down details on this is aggravating, and I figured you may be the guy to confirm what I think I know! 🙂
Using the “percentage” approach on a heterogeneous cluster, the percentage gets applied to each host in the cluster individually, so that with a 10% setting, a host with 60GB of memory will reserve 6GB and one with 10GB will reserve 1GB. Now – if one host with the larger memory fails, and assuming all other hosts are max utilized except for their HA reservations, then … insufficient memory has been reserved to accommodate all of the workload from the failed host.
Is that correct?
So – calculating HA % settings for heterogeneous clusters is … trial and error at best?
it simply sets aside 10% of your total cluster capacity, that could be 2% from host-1 and 3% from host-3 and 4% from host-4 etc. If one host fails then HA will restart the failed VMs.
That 10% should match the number of resources of course your largest host provides.
Really- so it’s much smarter than I thought –
Just want to re-state so it’s clear we are saying the same thing:
instead of each host simply reserving that percentage of it’s own resources – somehow the HA functionality understands how to level across hosts with different resource capabilities? So hosts with more memory may reserve more than 10% (assuming that is the setting) to account for there being some hosts with less memory?
exactly… Also, if all hosts have only just a a couple of percent free and not 1 has sufficient resources to restart that large VM, than vSphere HA can do a “cluster defragmentation” through DRS to make room on a host for that VM to be restarted.