“An ESX/ESXi host requires connectivity to vCenter Server to make vDS operations, such as powering on a VM to attach that VM’s network interface.”

“The issue is that if vCenter is a VM and changes hosts during a disaster (like a total power outage) and then is unable to grant itself a port to come back online.”

I figured the best way to debunk all these myths was to test it myself. I am confident that it is no problem, but I wanted to make sure that I could convince you. So what will I be testing?

• Network connectivity after Powering-on a VM which is connected to a VDS while vCenter is down.
• Network connectivity restore of vCenter attached to a VDS after a host failure.
• Network connectivity restore of vCenter attached to a VDS after HA has moved the VM to a different host and restarted it.

Before we start I think it is useful to rehash something, which is different types of portgroups which is described in more depth in this KB:

• Static binding - Port is immediately assigned and reserved for it when VM is connected to the dvPortgroup through vCenter. This happens during the provisioning of the virtual machine!
• Dynamic binding - Port is assigned to a virtual machine only when the virtual machine is powered on and its NIC is in a connected state. The Port is disconnected when the virtual machine is powered off or the virtual machine’s NIC is disconnected. (Deprecated in 5.0)
• Ephemeral binding - Port is created and assigned to a virtual machine when the virtual machine is powered on and its NIC is in a connected state. The Port is deleted when the virtual machine is powered off or the virtual machine’s NIC is disconnected. Ephemeral Port assignments can be made through ESX/ESXi as well as vCenter.

Hopefully this makes it clear straight away that their should be no problem at all, “Static Binding” is the default and even when vCenter is down a VM which has been provisioned before vCenter went down can easily be powered on and will have network access. I don’t mind spending some lab hours on this, so lets put this to a test. Lets use the defaults and see what the results are.

First I made sure all VMs were connected to a dvSwitch. I powered of a VM and checked the “Network settings and this is what it revealed… a port already assigned even when powered off:

This is not the only place you can see port assignments, you can verify it on the VDS’s “ports” tab:

Now lets test this, as that is ultimately what it is all about. First test, Network connectivity after Powering-on a VM which is connected to a VDS while vCenter is down:

• Connected VM to dvPortgroup with static binding (is the default and best practice)
• Power off VM
• Power off vCenter VM
• Connect vSphere Client to host
• Power on VM
• Ping VM –> Positive result
• You can even see on the command line that this VM uses its assigned port:

  esxcli network vswitch dvs vmware list

  Client: w2k8-001.eth0

  DVPortgroup ID: dvportgroup-516

  In Use: true

  Port ID: 137

Second test, Network connectivity restore of vCenter attached to a VDS after a host failure:

• Connected vCenter VM to dvPortgroup with static binding (is the default and best practice)
• Power off vCenter VM
• Connect vSphere Client to host
• Power on vCenter VM
• Ping vCenter VM –> Positive result

Third test, Network connectivity restore of vCenter attached to a VDS after HA has moved the VM to a different host and restarted it.

• Connected vCenter VM to dvPortgroup with static binding (is the default and best practice)
• Yanked the cable out of the ESXi host on which vCenter was running
• Opened a ping to the vCenter VM
• HA re-registered the vCenter VM on a different host and powered it on
• The re-register / power-on took roughly 45 – 60 seconds
• Ping vCenter VM –> Positive result

I hope this debunks some of those myths floating around. I am the first to admit that there are still challenges out there, these will hopefully be addressed soon, but I can assure you that your virtual machines will regain connection as soon as they are powered on through HA or manually… yes even when your vCenter Server is down.

I figured it should be possible to achieve this using a CNAME but also recalled something around vCenter screwing this up again. I tested it anyway and with success. This is what I did:

• Added both NFS servers to DNS
• Create a CNAME (DNS Alias) and pointed to the “active” NFS server
• I used the name “nasdr” to make it obvious what it is used for
• Created an NFS share (drtest) on the NFS server
• Mount the NFS export using vCenter or though the CLI
• esxcfg-nas -a -o nasdr -s /drtest drtest
• Check the UUID using vCenter or through the CLI
• ls -lah /vmfs/volumes
• example output:
  lrwxr-xr-x    1 root     root           17 Feb  6 10:56 drtest -> e9f77a89-7b01e9fd
• Created a virtual machine on the nfsdatastore
• Enabled replication to my “standby” NFS server
• I killed my “active” NFS server environment (after validating it had completed replication)
• Changed the CNAME to point to the secondary NFS server
• Unmounted the volume old volume
• esxcfg-nas -d drtest
• I did a vmkping to “nasdr” just to validate the destination IP had changed
• Rescanned my storage using “esxcfg-rescan -A”
• Mounted the new volume
• esxcfg-nas -a -o nasdr -s /drtest drtest
• Checked the UUID using the CLI
• ls -lah /vmfs/volumes
• example output:
  lrwxr-xr-x    1 root     root           17 Feb  6 13:09 drtest -> e9f77a89-7b01e9fd
• Powered on the virtual machine now running on the secondary NFS server

As you can see, both volumes had the exact same UUID. After the fail-over I could power-on the virtual machine. No need to re-register the virtual machines within vCenter first. Before I wanted to share it with the world I reached out to my friends at NetApp. Vaughn Stewart connected me with Peter Learmonth who validated my findings and actually pointed me to a blog article he wrote about this topic. I suggest to head-over to Peter’s article for more details on this.

But, just like Rob, you might not agree with this and rather have SDRS handling balancing on a file per file basis. That is possible and we documented this procedure in our book. I was under the impression that I blogged this, but just noticed that somehow I never did. Here is how you change the affinity rule for the current provisioned VMs in a datastore cluster:

1. Go to Datastores and Datastore Clusters
2. Right click a datastore cluster and select “edit settings”
3. Click “Virtual machine settings”
4. Deselect “Keep VMDKs together”
1. For virtual machines that need to stick together you can override the default by ticking the tick box next to the VM

Also check out this article by Frank about DRS/SDRS affinity rules, useful to know!

And the actually migrations being spawned:

Yes, this is the true value of Storage DRS… initial placement recommendations!

Cannot initialize property ‘vami.netmask0.VM_1′ since network ‘VM Network’ has no associated IP Pools configuration.

I figured this would be simple so I jumped back to “home” and went to the network section… Nothing around IP Pools. Even on a host or cluster layer there was nothing. Luckily my colleague Cormac jumped in and said check the “Datacenter” object, there should be an IP Pool tab there. He was right. Weird place and definitely something that needs to be improved. Anyway, configuring an IP Pool itself, now that I found it, was easy:

1. Click your Datacenter object
2. Go to the “IP Pools” tab
3. Click “Add”
4. Fill out the details:
1. Subnet: which network will be used and what is the mask? (You can use a subnet calculator if you don’t know…)
2. Enter the details of the gateway
3. Specify a range, the format is “10.1.1.10#10″, this would result in a range from 10.1.1.10 until 10.1.1.19 (10 addresses counting from .10)
4. Don’t forget to tick the “Enable IP Pool” check box
5. Click on the “Associations” Tab and associate it to a network!
6. Also, fill out the DNS and proxy details if and when required.
5. This is what it should look like:
   

It is as simple as that, but indeed not easy to find hence the reason I figured a short article was in place.

PS: Creating a range and enabling the “IP Pool” is not required. “Enable IP Pool” enables the use of the Range. In this example I had to use a range as I could only use a specific range of this subnet.

Let me start by saying that in general HA should never be disabled. The later versions of vSphere have a neat option called “Enable Host Monitoring”. This option should be used for scheduled network maintenance. The difference between disabling host monitoring and disabling HA is that disabling host monitoring does not cause a full reconfiguration (see screenshot below) of HA and a new election process. Just the “host monitoring” functionality is disabled, which is what you want in this scenario.

Bilal asked multiple questions / made multiple statements in his article, I will respond to two of these specifically to explain the way HA handles failures/isolation:

In this case within 30 sec of the management network outage, each host would have declared itself isolated and wont attempt to restart any VMs like the primaries would in vSphere 5.

So why is this? As soon as a Master is isolated it will drop “ownership” of datastores on which VMs are running that are part of its cluster. Before the other hosts trigger the isolation response for a given VM they will validate if the datastore on which this VM is stored is “owned” by a master. In the case of a cluster wide isolation due to a network outage / maintenance the ownership would be dropped and this would result in HA not triggering the isolation response. This is a major change compared to vSphere 4.x and prior!

Now what happens when the network outage is over and the hosts are in a position to talk to each other? I have not been able to find documentation on whether an isolated host will enter an election (vSphere 4 or 5) ones the communication channel is open and bring the cluster back to life.

Lets focus on vSphere 5.0 as that seems most relevant. A host remains isolated until it observes HA network traffic, like for instance election messages OR it starts getting a response from an isolation address. Meaning that as long as the host is in “isolated state” it will continue to validate its isolation by pinging the isolation address. As soon as the isolation address responds it will initiate an election process or join an existing election process and the cluster will return to a normal state.

There’s absolutely no need to manually intervene. HA takes care of all of this for you.

I am hoping each of you will select the top-10 blogs based on quality, longevity and frequency. (I personally find length of the article irrelevant, content is King!) I did want to list my top 10 articles over the last 12 months:

1. The vSphere 5.0 – HA Deepdive
2. Using vSphere Auto-Deploy in your home lab
3. Multiple-NIC vMotion in vSphere 5…
4. esxtop
5. vSphere 5.0: Storage vMotion and the Mirror Driver
6. vSphere 5.0: What has changed for VMFS?
7. HA Architecture Series (1 – 5)
8. “Hacking” Site Recovery Manager (SRM) / a Storage Array Adapter
9. ESXi 5.0 and Scripted Installs
10. vSphere 5.0 vMotion Enhancements

The voting is very straight forward and will only take 2 minutes of your time, all you have to do is select your Top 10 favourite VMware related virtualization blog sites and then sort them in your order of preference (ie: 1 – 10) – it’s as easy as that! Don’t wait any longer, cast your vote now!

• iBook – $ 9.99
• Nook – $ 9.99

We hope you will enjoy it!

Another thing to check if you experience this error is to see if you have jumbo frames enabled on the management network, since this interferes with HA communication.

This is document here in a note: http://kb.vmware.com/kb/2006729

To make it crystal clear: disable jumbo frames on your management network with vSphere 5.0 as there’s a problem with it! This problem is currently being investigated by the HA engineering team and will hopefully be resolved.

<Update> Just received an email that all the cases where we thought vSphere HA issues were caused by Jumbo Frames being enabled were actually caused by the fact that it was not configured correctly end-to-end. Please validate Jumbo Frame configuration on all levels when configuring. (Physical Switches, vSwitch, Portgroup, VMkernel etc)</Update>

But what about using vShield Edge? If you look at how vShield Edge is used in a vCloud Director environment, mainly NAT’ing and Firewall functionality, you could use it in exactly the same way for your VMs in a DR enabled environment. I know there are many Apps out there which don’t use hardcoded IP adresses and which are simple to re-IP. But for those who are not, why not just leverage vShield Edge… NAT the VMs and when there is a DR event just swap out the NAT pool and update DNS. On the “inside” nothing will change… and the application will continue to work fine. On the outside things will change, but this is an “easy” fix with a lot less risk than re-IP’ing that whole multi-tier application.

I wonder how some of you out in the field do this today.