** for vSphere 5.0 check this update! **
Today I received a very valid question around a full cluster failure. What happens when all the hosts in a cluster go down and at some point return? Will the VMs be restarted and what do I need to have in place to ensure they will?
It seems to be an urban myth that you need to use “auto-start” for a full cluster failure. But as you might have noticed that won’t work when HA is enabled. So what will?
VMware HA
Is it really that simple? Yes it is! When a full cluster fails and nodes start powering up HA will restart the VMs. As you know HA (or to be precise the primary nodes) maintains the host states, which includes the status of all VMs on those hosts. When one of the primary nodes returns to duty it will trigger the restarts based on the last known state. Make sure you set the restart priority correct so that any VMs hosting “management apps” will be booted up first.
It can’t get any simpler than that can it!
PiroNet says
Hi Duncan, as you stated once one of the primaries is started up, it will also take over the fail-over coordinator roles to be able to coordinate the restart of the VMs based on the priorities if any…
However, if you don’t know what are your primaries, you might restart several secondaries before you finally start up a primary which will restart VMs. Thus it might be a good idea to keep track of your primaries and update that list whenever you put a host in maintenance mode, or disconnect it from the cluster or reconfigure your HA.
Rgds,
Didier
Duncan Epping says
Well, I guess you would always want to start up some of the secondaries first in any case to make sure you have resources available to power on VMs. But indeed for situations like these it would be good to have a list of primary/secondary nodes and boot them up in the required order.
you could use this script to dump the info on a regular basis: http://www.virtu-al.net/2009/10/28/powercli-listing-cluster-primary-ha-nodes/
Jeff Goldschrafe says
There is one very important exception to this, and that’s when you have an HA incident where isolation response kicks over the VMs on every node in the cluster, like in the event that you have a complete and full-out network failure. If HA powers everything off, it stays off, even as hosts reconnect to the cluster when the network comes back.
In a properly designed environment, this should never happen, but we’ve run across it in testing when all our hosts were hooked up to the same physical switch.
Duncan Epping says
Yes, hence the reason there is also an “HA Maintenance Mode” these days π
Brandon says
Thank god for HA maintenance mode. Was so annoying waiting for it to disable/enable back in the 3.5 days, especially with a lot of hosts.
Interesting scenarios I’ve never seen happen. I’m trying to think of a situation that this could happen. I guess if you lost power and the UPS and the backup generator all failed. Some places might not have that level of resiliency, but you should for having all your eggs in the VM basket. Maybe a network guy being incompetent could deploy a bad image to your network devices, all of them at the same time! That could do it too.
I guess the fact it could happen is all that matters, because that means it will.
B Long says
Actually this just happened to us during a Generator test the UPS decided it needed to step in, then promptly failed to maintain power and the entire DC shut off.
I was surprised when the ESX hosts were coming online that all of the VMs powered up within minutes.
Had we known this would happen I would have had the vCenter DB server and vCenter server and a DC set up with priority… Still it saved the DC team about 2 hours when getting things up and running. Yay VMware.
solgae says
What about DNS? When all hosts failed and it took out all your DNS VMs along with it, then HA would not work, whether you had anti-affinity or not. I guess I would probably have to turn on DNS server manually for HA to kick in? (Yes, I know about hosts file, but it gets cumbersome)
Probably another reason why you probably want an off-site DNS server or a backup physical DNS server locally.
Sharninder says
This is a neat tip. Sometimes I really wonder how many people really use VMware HA to the fullest. It is such a great piece of technology.
@Solgae: You should anyway have an offsite DNS since that is one of the pieces of infrastructure that the ESX/vCenter itself will depend on.
Chris says
If you have a network issue that causes an HA isolation event on all hosts in your cluster and you have HA configured to power off your VMs, HA will power off the VMs and the HA agent will be disabled. The HA agents will remain disabled until they can talk to vCenter which will reconfigure / start the HA agent on the hosts. Once HA is running it will start powering on your VMs. If vCenter is also a VM in this cluster and is set to power down, your VMs won’t be powered back on by HA until vCenter is up and running so it can reconfigure / start the HA agent on the hosts. Depending on your environment, it may make sense to set the vCenter VM’s isolation response to ‘Leave powered On’.
I don’t believe that DNS should play a role. I’m not 100% sure but I think the hosts will use DNS to do the initial HA configuration and then store the results in the FT_HOSTS file. Or maybe the hosts will try to use DNS but fall back to the FT_HOSTS file when necessary. Also, the hosts won’t need name resolution to the vCenter server since the IP of the vCenter server is stored in the vpxa.cfg file.
This is based on my testing with 4.1
Paul Geerlings says
I had the same situation a couple of weeks ago.
The entire datacenter went powerout, the UPS didnot work either…
The customers VI configuration and hardware infra was definitely not prepared for this.
When the power came backon shit started hitting the fan.
Here is what happened and some things to take into account;
IMPORTANT
Set the poweron options of the different hardware components in the right order, some cannot be controlled but the crucial often can.
The Switches;
The Ethernet and fiber switches usually come backonline after a powerrestore, there is normally no reason to keep them powered of.
The Enclosures:
The enclosures (in this case HP c7000 enclosures) start automatically there is no configuring there.
The Blades/Hosts;
The hosts (in this case HP blades) should be configured to NOT poweron after a powerrestore. In this case it was NOT set, causing the ESX hosts to come online without connection to the datastores. This wasnot really a problem the fact that the ESX servers were missing DNS (virtual) was.
The storage array;
The storage array (in this case a HP XP) is default set to NOT poweron after a powerrestore. This is best practice. Start it up and let it do all its checks before starting up your esx hosts.
VI infra;
If the storage array and the switches are back online you can begin starting up your esx hosts. Make sure you know what the primary HA hosts are.
DNS/AD;
Set the priority of your DNS and DC VMs, make sure they start first.
This is crucial when running these services virtual, HA needs DNS (still.., this is soon to change i was told in Kopenhagen, i think it was in some HA deepdive session ;-).
In our case the priority was not set and the blades were running long before the storage was.
The (longer) lack of DNS caused problems with HA on all the ESX hosts. When the storage array came back online HA did kickin, it began starting VMs, no problems there, the problems came when DRS had to do its work.
vCenter/Database;
Make sure your vCenter server is the next VM to start.
If it is NOT also running the vCenter Database, bring that up first.
HA works without vCenter, no problem there, DRS does need vCenter.
In our vSphere 4.0 version, HA and DRS still work independent of each other (this is changed in 4.1, the more reason to update).
Because of the mentioned HA problem, DRS was unable to move VMs causing HA to startup to many VMs on the primary host (over 30 vms on a 16GB Host). The customer did not configure a slotsize, I think they forgot to read your HA deepdive :-).
The customer also had no startup priority set on the vCenter VM, also no higher shares. This would have been nice in this case because the entire infra was very unresponsive.
I finally disabled HA causing DRS to be able to move VMS arround.
We ended up rebooting all the VMs because of all kinds of problems, this time in the right order.
Denis Baturin says
Duncan,
HA restarts VMs based on the last known state. What if all VMs were shuted down gracefully by powerchute or UPS shutdown script… VMs won`t start!!!
Duncan Epping says
Really? Of course they wonβt. That is not what HA is supposed to do is it, so no need to shout at me.
When you have a script that powers down all your VMs make sure you also have a script that powers them up,
Denis Baturin says
vSphere already has builtin script to start virtual machines and cool interfaces to manage it. π
/sbin/vmware-autostart.sh
Denis Baturin says
Continued at
http://communities.vmware.com/message/1667819#1667819
BV says
Duncan: Can you please clarify 4.1 HA’s dependencies on DNS? After reading this thread, I’m still not completely clear about the circumstances under which 4.1 HA cannot do its job due to a DNS issue. Some of the comments imply that DNS doesn’t matter in the all hosts failed scenario, other comments (specifically, Paul’s) indicate that it does.
I’m primarily interested in obtaining a better understanding of HA’s expected response when all hosts in a 4.1 cluster simultaneously fail (power loss), with a 100% virtualized AD/DNS(AD-integrated zones)infrastructure running on that failed cluster, and all cluster nodes are simultaneously powered back on when power is restored. Will DNS dependencies prevent HA from restarting VMs or is all node-to-node communication using FT_HOSTS in this situation?
Duncan says
Well 4.1 or lower in some cases still has a dependency on DNS, I don’t know all corner case scenarios from the top of my head so can’t really provide that answer, you will need to test it.
BV says
Thanks for your reply. Dimostratzione it is then. I’ll setup some lab experiments to explore this scenario as soon as possible.
It seems reasonable to think that all of HA’s DNS dependencies would be documented somewhere. Prior to reading this thread, I was under the impression that DNS was only “required” during initial HA setup. Based on the comments in this thread and some of my additional research on this subject, there still seems to be enough community misconception about this topic (in a 4.1 and lower context) to warrant some definitive clarification from VMware (product docs, KB article, etc.). If such clarification is already out there, I’ve missed it thus far. To be clear, I’m specifically referring to clarification about HA’s DNS dependencies during the failover process, not initial HA configuration.
Even in the context of info about DNS dependencies being removed from vSphere 5’s HA architecture, I haven’t been able to track down a deepdive-level comparison to HA’s DNS dependencies in 4.1 and lower.
The 4.1 HA/DRS Technical Deepdive book doesn’t address this particular subject matter in deepdive fashion. Seems like a worthy technical deepdive topic. Addendum? π
BV says
Edit: Dimostrazione
If I’m going to use that word, I guess I should spell it correctly. Leonardo would not be pleased. π
Peter says
Hi,
I have a customer who is currently facing the issue where a HA isolation event occured on both hosts in his cluster due to a powercut and fault switches. The vcenter is a vm on one of thoes 2 esx hosts and as all vms were powered down…
For the moment when i log onto both esx hosts i get the message that they are isolated from the network.
meanwhile i managed to get the vm with vcenter on it running again.
i restarted vpxa & mgmt services on the esx but still no reply from both hosts.
currently all vms are running but both hosts cannot be managed via vcenter as they are not responding/disconnected.
what would you recommend to get this fixed ?
thanks in advance.
regards.
Duncan Epping says
Is the network up and running fully? Because it should be possible to ping either or at least the gateway for a fail-over to occur!
Peter says
Hi Duncan,
Thanks for your quick reply!
Network is back up again as before.
The gateway can be pinged again.
vm’s on the shared storage can be pinged,physical computers & network devices can be pinged.
so i think network is fully functional.
hariprajan says
HI Duncan ,
I have not worked on vSphere 4.0 and now working on vSphere 5.5 (Learning Stage). It will be great some more lights in to HA – DNS dependency we were having with till 4 series and now with IP.
Regards
Hari Rajan
Omphile Tshipa says
Just to get get clarity, i have a 3 node cluster. all were power-off because of power failure. when power was restored, only 2 nodes came up. unfortunately the node that didn’t come had several VMs including Vcenter. is that whats supposed to happen? why couldn’t the 2 hosts takeover the VMs when they came up?