I’ve seen a lot misinterpretations of the new advanced HA option “das.allowNetwork”. What is this option for and when should I use it or shouldn’t is the most asked question.
So let’s start with a little history, back in the days “pre 3.5 U2” one could configure a cluster on multiple IP subnets without facing any problems, but this could lead into start up or execution problems. HA would configure without checking if the network on every host was compatible with the other, but in ESX 3.5 U2 this procedure changed. As of this version during the configuration of HA the host checks it’s network for compatability with the first configured HA host. So if the first host has the following configuration:
esx01 – 192.168.1.11 – 255.255.255.0.
And the second host has the following configuration:
esx02- 192.168.2.11 – 255.255.255.0
HA will fail with the following error:
“HA Agent on <hostname> in cluster <clustername> in <datacenter> has an error Incompatible HA Networks: Host has network(s) that don’t exist on cluster members: <ip address>: Cluster has network(s) missing on host: <ip address>: Consider using Advanced Cluster Settings das.allowNetwork to control network usage”
With the advance option “das.allowNetwork” you can rule out certain Service Consoles for HA usage. In other words, if you have several SC’s on each ESX host you can specify which SC should be used for HA. HA, as most of you probably know, uses the SC network for it’s heartbeat.
A couple of examples when to use das.allowNetwork and when not to use it:
- So what should you do to get things running when you’ve used different subnets for each host?
Well that’s an easy one… get all your Service Consoles on the same subnet. Even if’s a routable vlan, HA will just calculate the network and if the values mismatch than configuring HA will fail! - So what if I’ve got multiple SC’s and some are on different subnets but there’s at least one on each host on the same subnet with the same portgroup name?
Here’s where das.allowNetwork comes in to place, set this option in the “Advanced Option” section of the HA tab and specify which SC should be used for HA. The ones that aren’t specified will not be used.
There have been numerous people asking if this “network compatibility” check could be disabled. At this moment the answer is no and there’s now workaround at this moment. When using das.allowNetwork specify the first as “das.allowNetwork0” the second as “das.allowNetwork1” etc.
More info can be found here:
I was asked:
If a host has one service console network with default label Service Console, is it ok to change to something_else? As far as I know, why not! Go change it. It’s just a label!
But given a scenario like below, it might be simple to do as below
5 Hosts – one SC network, same subnet
6th Hosts – one SC network, different subnet
7th Hosts – one SC network, different subnet
Instead of adding “Service Console 2” on 6th and 7th host, changing
“Service Console” to “Service Console 2” on 5 hosts, I suggest to
1) Keep “Service Console” as it is on 5 hosts
2) Add new SC portgroup on 6th and 7th and change the label “Service Console 2”
3) Move “Service Console” in 6th and 7th to same subnet as 1 – 5.
That is, if 6th and 7th still need to stay in their original subnet.
Note: Updating /etc/hosts file and /etc/resolv.conf might be necessary for 6th and 7th hosts.
Does adding
das.allowNetwork1 “Service Console”
das.allowNetwork2 “Service Console 2”
Have the same effect of having Service Console redundancy and telling it to use multiple addresses to check for isolation?
Benjamin: Indeed, that would be the easiest! I would empty out the host files, as in get all the entries for the other hosts out of there. as of 2.5 U2 HA will get this info from the VC anyway.
Jaime: If you have several Service Consoles all will be used for HA. You will only need to specify these 2 SC’s if you’ve got a third for instance and want to rule that one out.
Would HA still use only the primary SC default gateway for the isolation address unless others are specified using das.isolationaddress?
Yes it will, as far as I know.
I’m not very happy with this explanation.
In my understanding of the matter:
1. Network compatibility check is a genuine part of HA cluster design. It wasn’t working in the previous versions, just because it wasn’t implemented.
2. During the formation of the HA cluster all service console network interfaces (all management network interfaces on ESXi) are being sorted into networks. HA treats each service console interface (each management interface on ESXi which is not used by VMotion, unless we have das.allowVmotionNetworks defined ) as cluster network interface (here the dependence on the das.isolationaddress must be checked). When HA discovers a network interffaqce which does not belong to any known cluster network id adds the new network. HA expects that all cluster members will have the same networks. Thus if a new host has an interface which does not belong top any existing xluster networks HA throws message about incompatible networks
3. Option das.allowNetwork allow to pick for cluster communcication only those networks, whose port group name matches the value of this parameter. It is not an intended use of das.allowNetwork but with this parameter we chan “mask out” service console interfaces which should not be used for cluster communications. Each parameter denotes one network. Thus das.allowNetwork allows to put in the same network interfaces on different subnets.
4. Do not use das.allowNetwork (without number) with das.allowNetwork2 (with number) this may prevent the cluster formation.
I need to verify some staments and then I can prepare document with some use cases.
5. DNS configuration (as well as /etc/hosts file) for an ESX host in HA cluster are obsolete. Each node gets the name resolution for others cluster members and keeps it locally. You may need DNS / /etc/hosts for time seerver configuration.
I will double check everything there and republish results
Best regards
Seva,
1) I agree, that’s what I stated… It’s new.
2) We’re saying the same but in a different way
3) I don’t agree on this one. According to Marc’s posts on the VMTN forum it isn’t possible.
4) I know you have to specify number, but I didn’t elaborate on it because that’s not what the article is about, but I will clarify it.
5) I don’t think DNS is obsolete. Maybe for HA it is. But when I use stuff like scripting you’ll need DNS to resolve the other hosts etc.
So… I’ve run into an interesting “problem”. All of my SC’s are on the same subnet. 0 issues with esx 3.5. When I upgraded to vsphere4, it gives the error. Thing is, we’re using a class B instead of a class C. It looks to me like the code checking automatically assumes a class C and bombs out. Maybe it’s something else, but to me it looks like it’s an HA config issue.
Hi All, I know this is an old string but I haven’t found any answer on this yet. I have four hosts each with two vSwitchs (vSwitch0 and vSwitch1) Each has its own service console on the same subnet. (In Case the first vSwitch Fails) Where do I specify the secondary Service Console addresses? Do I add them all into the DAS.Allow setting??? Please help Im really stuck with this!
Thanks in Advance
You do not need to specify anything! But why did you use the same subnet for both? Having differen subnet adds more resiliency and you can use a second das.isolationaddress
I know it’s an old post, but it’s still valid.
When to use das.allowNetwork?
In my environment/case, if I have more than one Service Console. I have two Service Console, one is being used for management, which is a Tagged with vLAN ID and has got a vLAN Pingable Gateway.
The other Service Console I called it Backup Console. This Backup Console is being used only for iSCSI/NFS with a vMkernel Portgroups. This Backup Console in a separate vLAN and doesn’t reach by any other vLAN. Only iSCSI/NFS hosts or devices are reaching these Backup Console and vMkernel Portgroups.
Where to use das.allowNetwork?
I use it to separate/distinguish the vMotion network. So, the vMotion network will always use the first Service Console which is resides in the vSwitch0 and won’t use my Backup Console which resides in vSwitch1.
So, in das.allowNetwork0 = Service Console
Not to das.allowNetwork1 = Backup Console
Thanks for this explanation Duncan
Regards,
S.Hussain
I don’t understand what vMotion has to do with the Service Console? But anyway, I would assume that you would like HA to use both options for heartbeats as that would decrease chances of a false positive. In your environment I would as such not recommend using das.allowNetwork at all.
I am having similar situation which might be related. I am setting up my ESXi 4.1 Update 1 hosts with 2 Service Console (SC). Both of the SC are in different segments. The primary SC has a pingable gateway while the secondary SC doesn’t.
All my ESXi host and the vCenter has entries of the primary SC and the secondary SC in the hosts file.
When i try to simulate a network failure on the primary SC on one of the ESXi host(unplugged the cable from the NIC), the ESXi host becomes disconnected and unmanageable. I changed the hosts file on the vCenter to use the secondary SC to connect to the ESXi host. I did a right click and select connect, it works but fails during the HA config.
In this scenario, should i use das.allowNetwork to make the HA working again?
If I have 7 hosts in a cluster and I have multiple management network(different subnet) on the last host use it for p2v migration purpose , with this advance setting does it help to avoid HA giving error? cause now every time if the host rebooted it will try to re-configure the HA and giving the error it can’t reach to the rest of the management network.