I had question on my HA Deepdive which I thought was worth answering in an article:
How does the active primary node decide where to restart failed VMs? Does it use a round-robin algorithm for selecting a host to start the VMs in restart priority order? What happens if the remaining nodes are imbalanced, especially without DRS enabled; are the nodes that have no spare capacity skipped? Or, does the active primary node restart VMs on the least busy host first, then the next busy host, etc?
Also, if VMs have no reservation for CPU or memory set, how does HA decide the number of VMs to restart on any one node? Is it possible that HA will restart too many VMs on one node so that performance is extremely poor until DRS move some VMs to other nodes?
In the past HA(pre 4.1) would consider the utilization of the Hosts and go through a check for every VM that needs to failover. It would fail the VM over the host with the most amount of available resources. Now from a “latency” perspective that is not the best approach as you can imagine. With latency meaning the time it takes to restart the VMs and the delay caused by hostd. Now type of delay can be cause by hostd? Well lets assume you have 1 host which is not doing a lot, this host would be the host that is selected for most failovers. Having 10 VMs (or more) starting in parallel will beat hostd severely.
So does HA use DRS to select which host to use for the restart? No it won’t, DRS happens on a vCenter level and HA happens on a host level…. But more things have changed. As of vSphere 4.1 virtual machines will be evenly distributed across hosts to lighten the load on the hostd service and to get quicker power-on results. HA then relies on DRS to redistribute the load later if required. This improvement results in faster restarts of the virtual machines and less stress on the ESX hosts.
So what if you are not using DRS? To put it bluntly, make sure you manual balance your environment to ensure HA doesn’t “overload” a single host… that is the only thing you can do for now. (by the way, all of this is included in the HA and DRS tech deepdive :-))
Mark Jones says
Manually balancing your environment is fine when you are dealing with just a few hosts but when you are dealing with hundreds of hosts with potentially thousands of virtual machines this is impossible.
If the infrastructure is right sized and the vm’s are placed intelligently up front based on observed utilization patterns then DRS activity is dramatically reduced. Any DRS operations should then be treated as an incident and a ticket raised.
Russell says
I’ll buy your book as soon as its available electronically. There just isn’t any point in buying printed technical books these days. For all we know these technologies could completely change in a year’s time.
That and I’m a traveling consultant and it’d take a large truck to carry around paper copies of the references I generally have around.
RE: Mark’s comments:
I definitely agree which is why at this point you want DRS enabled to handle these sorts of things. I don’t think Duncan was recommending this as a practice but more or less explaining that yes it is possible and here are the caveats (scalability being one of them)
I’d disagree with opening up an incident on every DRS operation though. On an HA failover event most definitely. If you’re worried about resource heavy applications stacking on the same system you can manage this with DRS rules and I’m sure eventually a number of 3rd party products will pop up to enable application performance based DRS as opposed to ESX load based DRS (VMTurbo is probably moving in this direction now).
We’re moving closer to that “golden datacenter” where servers just become dumb compute heads. If something is under-performing then yeah open up an incident and resolve it. Focus on service levels instead of metered CPU/memory/network/disk consumption.
Sean says
When you mention pre-4.1 vs 4.1, which behavior is used in an environment with vCenter 4.1 and ESX 4.0?
I’m assuming the 4.1 behavior since I believe the HA agents are installed via vCenter, but I’m hesitant to go too far with that assumption since I’ve never seen it clearly spelled out.
Duncan says
Couple of things here:
1) DRS Rocks, manually load balancing is something that will not work in ANY type of environment as you will always be running behind the facts.
2) An incident/ticket for DRS? I don’t think so… that will make your environment inflexible and incur a huge overhead
3) Digital, maybe early next year… maybe it won’t happen. The book is self-publishing which means we did everything ourselves including formatting. Formats like e-pub require a lot of reformatting unfortunately.
4) pre 4.1 is Pre vCenter 4.1 indeed.
Domenico Viggiani says
We currently have a bunch of licenses (coming from old ESX 3.5) including HA not not DRS. Thus, manually balancing is the only alternative possible!
Mark Jones says
All valid points and as Russell mentions it’s all about the SLA’s. Look forward to reading the book Duncan and wish anyone trying to manually load balance their environment the best of luck!
Nanthakumar says
HI
Is ther any thing like active primary node?, if yes, how does active primaty selected?, what is role of it? If active primary failes what happens next?
Duncan says
Did you read the HA deepdive page?
Nanthakumar says
Not yet. I will read and get back to you, Thanks