I’ve seen a lot misinterpretations of the new advanced HA option “das.allowNetwork”. What is this option for and when should I use it or shouldn’t is the most asked question.
So let’s start with a little history, back in the days “pre 3.5 U2″ one could configure a cluster on multiple IP subnets without facing any problems, but this could lead into start up or execution problems. HA would configure without checking if the network on every host was compatible with the other, but in ESX 3.5 U2 this procedure changed. As of this version during the configuration of HA the host checks it’s network for compatability with the first configured HA host. So if the first host has the following configuration:
esx01 – 192.168.1.11 – 255.255.255.0.
And the second host has the following configuration:
esx02- 192.168.2.11 – 255.255.255.0
HA will fail with the following error:
“HA Agent on <hostname> in cluster <clustername> in <datacenter> has an error Incompatible HA Networks: Host has network(s) that don’t exist on cluster members: <ip address>: Cluster has network(s) missing on host: <ip address>: Consider using Advanced Cluster Settings das.allowNetwork to control network usage”
With the advance option “das.allowNetwork” you can rule out certain Service Consoles for HA usage. In other words, if you have several SC’s on each ESX host you can specify which SC should be used for HA. HA, as most of you probably know, uses the SC network for it’s heartbeat.
A couple of examples when to use das.allowNetwork and when not to use it:
- So what should you do to get things running when you’ve used different subnets for each host?
Well that’s an easy one… get all your Service Consoles on the same subnet. Even if’s a routable vlan, HA will just calculate the network and if the values mismatch than configuring HA will fail! - So what if I’ve got multiple SC’s and some are on different subnets but there’s at least one on each host on the same subnet with the same portgroup name?
Here’s where das.allowNetwork comes in to place, set this option in the “Advanced Option” section of the HA tab and specify which SC should be used for HA. The ones that aren’t specified will not be used.
There have been numerous people asking if this “network compatibility” check could be disabled. At this moment the answer is no and there’s now workaround at this moment. When using das.allowNetwork specify the first as “das.allowNetwork0″ the second as “das.allowNetwork1″ etc.
More info can be found here:
vSphere 4.0 Quick Start Guide
I was asked:
If a host has one service console network with default label Service Console, is it ok to change to something_else? As far as I know, why not! Go change it. It’s just a label!
But given a scenario like below, it might be simple to do as below
5 Hosts – one SC network, same subnet
6th Hosts – one SC network, different subnet
7th Hosts – one SC network, different subnet
Instead of adding “Service Console 2″ on 6th and 7th host, changing
“Service Console” to “Service Console 2″ on 5 hosts, I suggest to
1) Keep “Service Console” as it is on 5 hosts
2) Add new SC portgroup on 6th and 7th and change the label “Service Console 2″
3) Move “Service Console” in 6th and 7th to same subnet as 1 – 5.
That is, if 6th and 7th still need to stay in their original subnet.
Note: Updating /etc/hosts file and /etc/resolv.conf might be necessary for 6th and 7th hosts.
Does adding
das.allowNetwork1 “Service Console”
das.allowNetwork2 “Service Console 2″
Have the same effect of having Service Console redundancy and telling it to use multiple addresses to check for isolation?
Benjamin: Indeed, that would be the easiest! I would empty out the host files, as in get all the entries for the other hosts out of there. as of 2.5 U2 HA will get this info from the VC anyway.
Jaime: If you have several Service Consoles all will be used for HA. You will only need to specify these 2 SC’s if you’ve got a third for instance and want to rule that one out.
Would HA still use only the primary SC default gateway for the isolation address unless others are specified using das.isolationaddress?
Yes it will, as far as I know.
I’m not very happy with this explanation.
In my understanding of the matter:
1. Network compatibility check is a genuine part of HA cluster design. It wasn’t working in the previous versions, just because it wasn’t implemented.
2. During the formation of the HA cluster all service console network interfaces (all management network interfaces on ESXi) are being sorted into networks. HA treats each service console interface (each management interface on ESXi which is not used by VMotion, unless we have das.allowVmotionNetworks defined ) as cluster network interface (here the dependence on the das.isolationaddress must be checked). When HA discovers a network interffaqce which does not belong to any known cluster network id adds the new network. HA expects that all cluster members will have the same networks. Thus if a new host has an interface which does not belong top any existing xluster networks HA throws message about incompatible networks
3. Option das.allowNetwork allow to pick for cluster communcication only those networks, whose port group name matches the value of this parameter. It is not an intended use of das.allowNetwork but with this parameter we chan “mask out” service console interfaces which should not be used for cluster communications. Each parameter denotes one network. Thus das.allowNetwork allows to put in the same network interfaces on different subnets.
4. Do not use das.allowNetwork (without number) with das.allowNetwork2 (with number) this may prevent the cluster formation.
I need to verify some staments and then I can prepare document with some use cases.
5. DNS configuration (as well as /etc/hosts file) for an ESX host in HA cluster are obsolete. Each node gets the name resolution for others cluster members and keeps it locally. You may need DNS / /etc/hosts for time seerver configuration.
I will double check everything there and republish results
Best regards
Seva,
1) I agree, that’s what I stated… It’s new.
2) We’re saying the same but in a different way
3) I don’t agree on this one. According to Marc’s posts on the VMTN forum it isn’t possible.
4) I know you have to specify number, but I didn’t elaborate on it because that’s not what the article is about, but I will clarify it.
5) I don’t think DNS is obsolete. Maybe for HA it is. But when I use stuff like scripting you’ll need DNS to resolve the other hosts etc.
[...] – Disable the “compatible network” check for HA that was introduced with Update 2. Default value is “false”, setting it to “true” disables the [...]
So… I’ve run into an interesting “problem”. All of my SC’s are on the same subnet. 0 issues with esx 3.5. When I upgraded to vsphere4, it gives the error. Thing is, we’re using a class B instead of a class C. It looks to me like the code checking automatically assumes a class C and bombs out. Maybe it’s something else, but to me it looks like it’s an HA config issue.