On Friday I got a question about the “Leave powered on” setting for HA. This setting is used for the Isolation Response. In other words, what does HA needs to do when a network isolation is detected. The question was pretty straight forward:”What happens when a host is isolated from the network or a host dies completely?”.
This question was asked because the default setting changed in ESX 3.5 / VirtualCenter 2.5 from “Power off vm” to “Leave powered on”. The old value was simple, what ever happens the VM will be powered off and restarted on a different host.
The new default value is real simple:
When a host network isolation occurs the VM’s remain running.
Because of the VMDK file lock(Look into VMFS Distributed lock management if you want to know more about this feature) no other host will be able to boot the VM’s. Yes indeed, they will try to boot them because these hosts don’t know the “missing” host is isolated, it could be completely dead. So if all network connections are dead, including the vSwitch for the VM’s these VM’s will not be restarted on a different host. They will only be restarted if for whatever reason also the storage connection fails and the VMDK file locks are timed out.
This also means that when using for instance iSCSI you should never ever set this setting to “Leave powered on” cause this will cause a split brain scenario for sure. (VM’s will be restarted on a new host because the lock timed out, while the VM’s also are still running on the original host.)
So what happens when a host is completely dead, yes you’ve probably guesed it by now the VMDK file locks times out and the VM’s are restarted on another host.
And although this all seems very obvious, it’s still an often overlooked setting. There’s no real right or wrong when it comes down to this setting. It’s just what your prefer. I would prefer a short down time over running VM’s on a downgraded host, so my advise would be set it to “Power off VM”.
As an addition I would love to see a “HA heartbeat” tick-box on vSwitches, which you could use to see if the VM’s are isolated from the network or not.
“So what happens when a host is completely dead, yes you’ve probably guest it by now the VMDK file locks times out and the VM’s are restarted on another host.”
I’ll let you off with a warning this time. Next time, you may not be so lucky.
-The Grammar Police.
🙂
p.s., thanks for bringing clarification to that setting.
I’m not sure I follow why it’s particularly important to power down the VM’s when using iSCSI. Shouldn’t it be the same risk as a FC environment?
Cause when it’s set to “leave powered on” with iscsi there’s a huge chance of a split-brain. iscsi uses the service console, usually it has an own service console. when both service consoles are isolated so is your iscsi san. when the vm’s remain running, but the locks are timed out cause the iscsi connection is dropped the vm’s will be started on a different host, but they are also still running on the “isolated host”, you can imagine that when the network returns on the isolated host it can and will produce all sorts of problems.
Get the JDR Websites SEO package and we will spam websites like this to get you the best rankings in Google for any keyword you want. Our SEO dosn’t work, but as long as you pay us we don’t care. Contact us now and see how we can do this for you.
Hello cheers for this entry. That is extremely refreshing.
Nice post.Thank you for taking the time to publish this information very useful!
Thank you so much for your opinion on Leave powered on… » Yellow Bricks , I totally agree with you. It is nice to see a fresh outlook on this and I look forward to more.