Yellow Bricks

by Duncan Epping

ha

HA Retries

· ·

I am working on a special project at the moment. I noticed something I never realized before and wanted to share it with you guys. When a restart of a VM fails VMware HA will retry this.

The amount of VMware HA retries is configurable as of vCenter 2.5 U4 with the advanced option “das.maxvmrestartcount”. The default value is 5. Pre vCenter 2.5 U4 HA would keep retrying forever which could lead to serious problems as described in KB article 1009625 where multiple virtual machines would be registered on multiple hosts simultaneously leading to a confusing and inconsistent state.(http://kb.vmware.com/kb/1009625)

Important to note is that HA will try to start the virtual machine on each of the hosts in the affected cluster, if this is unsuccessful the restart count will be increased by 1. In other words, if a cluster contains 32 hosts HA will try to start the virtual machine on all hosts and count it as a single try. Something I definitely never realized and something that definitely is worth knowing.

VMware FT/HA, we need your feedback!

· ·

Yesterday I received an email from product management asking if I could plug the VMware FT/HA survey like I did with the VMware Snapshot survey. The VMware Snapshot survey literally had hundreds of responses with so much valuable feedback that they decided to re-open this survey and give you guys, my readers, the chance to speak your mind on VMware FT/HA!

Please take the time to fill out the survey and help VMware to improve FT and HA! It will only take a couple of minutes: http://www.surveymethods.com/EndUser.aspx?AD89E5F7ABEEFDF8

This survey will collect information relevant to VMware Fault Tolerance and VMware HA, a new solution that shipped with VMware vSphere 4. VMware Fault Tolerance provides 100% uptime for a VM in the course of an ESX host failure. This feature will ensure your VM continues running without data loss or noticeable downtime should you encounter a single host failure. As we continue to advance this solution, we hope you can provide us information about your current deployment that will help us design Fault Tolerance and HA with your most missions critical VMs in mind.

Cool new HA feature coming up to prevent a split brain situation!

· ·

I already knew this was coming up but wasn’t allowed to talk about it. As it is out in the open on the VMTN community I guess I can talk about it as well.

One of the most common issues experienced with VMware HA is a split brain situation. Although currently undocumented, vSphere has a detection mechanism for these situations. Even more important the upcoming release ESX 4.0 Update 2 will also automatically prevent it!

First let me explain what a split brain scenario is, lets start with describing the situation which is most commonly encountered:

4 Hosts – iSCSI / NFS based storage – Isolation response: leave powered on

When one of the hosts is completely isolated, including the Storage Network, the following will happen:

Host ESX001 is completely isolated including the storage network(remember iSCSI/NFS based storage!) but the VMs will not be powered off because the isolation response is set to “leave powered on”. After 15 seconds the remaining, non isolated, hosts will try to restart the VMs. Because of the fact that the iSCSI/NFS network is also isolated the lock on the VMDK will time out and the remaining hosts will be able to boot up the VMs. When ESX001 returns from isolation it will still have the VMX Processes running in memory. This is when you will see a “ping-pong” effect within vCenter, in other words VMs flipping back and forth between ESX001 and any of the other hosts.

As of version 4.0 ESX(i) detects that the lock on the VMDK has been lost and issues a question if the VM should be powered off or not. Please note that you will(currently) only see this question if you directly connect to the ESX host. Below you can find a screenshot of this question.

With ESX 4 update 2 the question will be auto-answered though and the VM will be powered off to avoid the ping-pong effect and a split brain scenario! How cool is that…

Network loss after HA initiated failover

· ·

I had a discussion with one of my readers last week and just read this post on the VMTN community which triggered this article.

When you create a highly available environment take into account that you will need to have enough vSwitch ports available when a failover needs to occur. By default a vSwitch will be created with 56 ports and in general this is sufficient for most environments. However when two of your hosts fail in a 10 host cluster you might end up with 60 or more VMs running on a single host. If this would happen several VMs would not have a vSwitch port assigned.

The most commonly used command when creating an automated build procedure probably is:

esxcfg-vswitch -a vSwitch1

This would result in a vSwitch named “vSwitch1” with the default amount of 56 ports. Now it is just as easy to set it up with 128 ports for instance:

esxcfg-vswitch -a vSwitch1:128

Always design for a worst case scenario. Also be aware of the overhead, some ports are reserved for internal usage. You might want to factor in some additional ports for this reason as for instance in the example above you will have 120 ports available for your VMs and not the 128 you specified.

Overhauling the HA deepdive section

· ·

I’ve been working on an overhauled version of the HA Deepdive Page. I’ve been adding “basic design principles” which hopefully you find useful.Here’s an example of what they look like:

Basic design principle: For iSCSI the preferred isolation response is always “Power off” to avoid a possible split brain scenario.

Another bit I’ve added is the following:

Please keep in mind that if you have an unbalanced cluster(host with different CPU or memory resources) your percentage is equal or preferably larger than the percentage of resources provided by the largest host. This way you ensure that all virtual machines residing on this host can be restarted in case of a host failure. Another thing to keep in mind is as there are no slots which HA uses resources might be fragmented throughout the cluster. Make sure you have at least a host with enough available capacity to boot the largest VM (reservation CPU/MEM). Also make sure you select the highest restart priority for this VM(of course depending on the SLA) to ensure it will be able to boot.)

What I’m discussing here is the impact of selecting a “Percentage” instead of the amount of host failures for your HA cluster. Although you might have enough spare resources left on your total cluster, the reservation of a single VM might cause it not to boot when resources are fragmented. Make sure these VMs are the first to boot up when disaster strikes by using restart priorities.

I created a diagram which makes it more obvious I think. So you have 5 hosts, each with roughly 76% memory usage. A host fails and all VMs will need to failover. One of those VMs has a 4GB memory reservation, as you can imagine failing over this particular VM will be impossible.