This document deals about VMware HA clusters and all the advanced options you have with the use cases for these options. And that’s what makes this document very useful, the use cases! So everyone dealing with HA clusters should definitely read this document. Thanks Seva for letting me distribute it, and thanks for a great document!

1. Configuration of host IP address is inconsistent on host : address resolved to Host misconfigured. IP address of not found on local interfaces
2. cmd addnode failed for primary node: Internal AAM Error – agent could not start

So the first error(1.) was reported by esxhost01 and the second(2.) by esxhost02.

Let’s start with esxhost01.

So this customer had a VMotion and Management portgroup on two seperate vSwitches. This error seems to indicate that during the configuration HA is using the VMotion portgroup. These hosts have been added to VC with the management portgroup IP(IP+Name also in dns). So how do I make sure that HA isn’t using the VMotion network for HA, it’s easy go to your cluster and open up the advanced options for HA and add the following key with the value false:

• das.allowVmotionNetworks=false

In other words, don’t use the VMotion network for the HA heartbeat. The weird thing in this case is that it shouldn’t use the VMotion network by default so there seems to be a glitch here…

So now for the second problem.

The HA(AAM) agent could not start. So just to make sure that the USB key wasn’t corrupt the key was recreated. But still this error occurred. As some of you might now, that if you want to use HA with a disk less server you will need to create a userworld swap on the SAN. (Read this KB for more info on that one…) So just to make sure that the swap wasn’t causing this problem the directory was cleaned out and and HA was reconfigured. When the directory was emptied the HA agent installed without any problem at all…

• When reinstalling ESXi or when strange HA errors occur clean up the userworld swap!

Thanks goes out to Remco for providing me with the additional details!

How does a primary and / or secondary get selected?

• The first 5 hosts that join the VMware HA cluster are automatically selected as “primary nodes”
• All the others are automatically selected as “secondary nodes”
• When you do a reconfigure for HA the primary nodes and secondary nodes are selected again, this is random

What’s up with these primaries and secondaries?

• Primary nodes hold cluster settings and all node states which are synced between primaries
• Secondary nodes send their state info(resource occupation) to the primary nodes
• Nodes send heartbeats to each other, primary nodes send heartbeats to primary nodes only and secondary also only to primary. And they do this every second. (Which is a changeable value: das.failuredetectioninterval)

So what if a primary node fails, will a secondary be promoted?

• No, there will only be a new primary appointed when the failed one is removed from the cluster. A secondary will be promoted to primary at random.

But what if all my primary nodes fail?

• This is an unaddressed issue, that’s the reason why you can only account for 4 host failures within a cluster! There needs to be at least one primary!

So when does the gateway come in play?

• Actually the gateway, which is the default “isolation address”, will only be used when an isolation has occurred. So when the AAM client thinks it’s isolated it will check the isolation addresses.

So if anyone has a question just drop it here and I’ll try to answer it and update the above list…

So in order to prevent this you could change the location of the VPXD log files very easily. Edit “vpxd.cfg”. It’s located here: %AllUsersProfile%\Application Data\VMware\VMware VirtualCenter\.

Add the following lines in the “<config>” section and change the path accordingly:

<log>
<directory>c:\VC_Logs</directory>
</log>

And for ESXi, you should be looking here “/var/run/vmware/aam/” by the way.

But for some of you 5 minutes(300 seconds) might just not be long enough, or if you have an ultra fast environment 5 minutes might just be to damn long….

So what can you do to shorten or extent… It’s easy, open up you HA cluster settings and click on advanced options and add the following:

das.isolationShutdownTimeout – values in seconds, default is 300

I’ve also updated my HA advanced settings blog! If anyone has more advanced settings that aren’t on the list let me know!

The following are outtakes of the article over at DABCC:

VMware’s HA is heavily dependent on DNS or alternatively hosts entries being in place. The VMware implementation is based on the Legato Automated Availability Management (AAM), in fact some of us will recall that it used to place those logs into /opt/LGTOaam512/logs/ (since 3.5 this has been moved /opt/vmware/aam).

VMware’s HA uses the network to establish a heartbeat between all the ESX Hosts participating, So practically, what does this mean to the poor bloke who has to support the servers? If you network has a bit of a flap (personally I always blame the Network guysJ), your servers will implement an “isolation response”, the default server response will shut down your Virtual Machine to release the shared storage locks, this will allow the machine to be restarted on another host, this of course may not be desirable if the server is busy doing something, i.e. you may cause corruption or other issues with the Application/Database. In other words it won’t perform a clean shutdown. This is configurable such that you can keep the machines powered on, but this isn’t recommended in the case of NAS or iSCSI (as they are also network dependant) and you may end up with a split-brain situation.

There is now also experimental support for component level HA, i.e. if a Virtual Machine fails, then VMware will try to restart it.

1. As of ESX 3.5 U2 High Availability doesn’t heavily lean on DNS anymore, it gets its hostname and ip info from VirtualCenter.
2. ESX 3.5 U2 gives you the possibility to cleanly shutdown a VM in case of an isolation.
3. Normally one would indeed provide it’s SC with redundancy, and preferably via two separate switches to avoid the problems you are describing.
4. Virtual Machine High Availability isn’t experimental anymore as of ESX 3.5 U2.

The error message that appears:

This product has expired. Be sure that your host machine’s date and time are set correctly.
There is a more recent version available at the VMware web site: http://www.vmware.com/info?id=4.
————–
Module License Power on failed

In short, the workaround is simple just set the date back and you will be able to power on the VM’s again, it would be smart to set the time to correct value again as soon as you started the VM. As soon as I know more about the new 3.5 U2 update I’ll let you guys know!

And a nice work around from the VMTN forum:

Find the host where a VM is located
run ‘ vmware-cmd -l ‘ to list the vms.
issue the commands:
service ntpd stop
date -s 08/01/2008
vmware-cmd /vmfs/volumes/vm path/vmname.vmx start
service ntpd start

So let’s start with a little history, back in the days “pre 3.5 U2″ one could configure a cluster on multiple IP subnets without facing any problems, but this could lead into start up or execution problems. HA would configure without checking if the network on every host was compatible with the other, but in ESX 3.5 U2 this procedure changed. As of this version during the configuration of HA the host checks it’s network for compatability with the first configured HA host. So if the first host has the following configuration:
esx01 – 192.168.1.11 – 255.255.255.0.

And the second host has the following configuration:
esx02- 192.168.2.11 – 255.255.255.0

HA will fail with the following error:
“HA Agent on <hostname> in cluster <clustername> in <datacenter> has an error Incompatible HA Networks: Host has network(s) that don’t exist on cluster members: <ip address>: Cluster has network(s) missing on host: <ip address>: Consider using Advanced Cluster Settings das.allowNetwork to control network usage”

With the advance option “das.allowNetwork” you can rule out certain Service Consoles for HA usage. In other words, if you have several SC’s on each ESX host you can specify which SC should be used for HA. HA, as most of you probably know, uses the SC network for it’s heartbeat.

A couple of examples when to use das.allowNetwork and when not to use it:

1. So what should you do to get things running when you’ve used different subnets for each host?
   Well that’s an easy one… get all your Service Consoles on the same subnet. Even if’s a routable vlan, HA will just calculate the network and if the values mismatch than configuring HA will fail!
2. So what if I’ve got multiple SC’s and some are on different subnets but there’s at least one on each host on the same subnet with the same portgroup name?
   Here’s where das.allowNetwork comes in to place, set this option in the “Advanced Option” section of the HA tab and specify which SC should be used for HA. The ones that aren’t specified will not be used.

There have been numerous people asking if this “network compatibility” check could be disabled. At this moment the answer is no and there’s now workaround at this moment. When using das.allowNetwork specify the first as “das.allowNetwork0″ the second as “das.allowNetwork1″ etc.

More info can be found here:

KB1006541

KB1006606

Resource Management PDF

Open this link to the PDF or click on the picture below.