Yellow Bricks

by Duncan Epping

ha

No Jumbo frames on your Management Network! (Updated!)

· ·

I was just reading some of the comments posted today and Marc Sevigny, one of the vSphere HA developers, pointed out something which I did not know. I figured this is probably something that many are not aware of so I copied and pasted his comment:

Another thing to check if you experience this error is to see if you have jumbo frames enabled on the management network, since this interferes with HA communication.

This is document here in a note: http://kb.vmware.com/kb/2006729

To make it crystal clear: disable jumbo frames on your management network with vSphere 5.0 as there’s a problem with it! This problem is currently being investigated by the HA engineering team and will hopefully be resolved.

<Update> Just received an email that all the cases where we thought vSphere HA issues were caused by Jumbo Frames being enabled were actually caused by the fact that it was not configured correctly end-to-end. Please validate Jumbo Frame configuration on all levels when configuring. (Physical Switches, vSwitch, Portgroup, VMkernel etc)</Update>

vSphere 4.1 HA/DRS Deepdive available in the Kindle Lending Library!

· ·

For those who hadn’t seen it yet, the “vSphere 4.1 HA/DRS Deepdive” is available in the Kindle Lending Library! The Kindle Owners’ Lending Library is a collection of books that Amazon Prime members who own a kindle can borrow once a month, with no due dates. (Yes you will need a Kindle Prime subscription!) If you have a Kindle or are using the Kindle App check it out:

Check it out, and let me know if you like it by leaving a comment… this is a trial for us as well, we might end up enabling it on all books in the future.

vSphere HA Waiting for cluster election to complete Operation timed out?

· ·

I noticed this thread on the VMTN communtity which discussed a time-out during a cluster election process. The one thing all scenarios described in the topic is that they upgraded from 4.1 to 5.0 or 5.0 base to a higher patch level. Marc Sevigny posted in the same thread that it is a known issue which the HA team is currently investigating…

After an upgrade, under conditions we’re still investigating, an error is occurring when issuing a start request of the HA service on the upgraded host.  When that fails, HA then tries to re-install HA, and the re-install does nothing because the service is already there (and the right version) but we’re left without an HA service running.

This is the way to fix it if you are experiencing this issue. Now, if you do experience this issue please report it to VMware and submit log files as that will help the HA team fixing the problem.

  1. Place host into Maintenance Mode
  2. Take a copy of /opt/vmware/uninstallers/VMware-fdm-uninstall.sh (we copied to /tmp)
  3. From the location you made a copy of the file, run the command (./VMware-fdm-uninstall.sh)
  4. You should see a short pause before it gets back to the prompt (you’ll see why I mention this below)
  5. Exit host out of Mainenance Mode and within the “Recent Tasks” area you should see the client being pulled from vCenter and installing

vSphere HA Isolation response when using IP Storage

· ·

I had a question from one of my colleagues last week about the vSphere HA Isolation Response and IP Storage. His customer had an ISCSI storage infrastructure (applies to NFS also) and recently implemented a new vSphere environment. When one of the hosts was isolated virtual machines were restarted and users started reporting strange problems.

What happened was that the vSphere HA Isolation Response was configured to “Leave Powered On” and as both the Management Network and the iSCSI Network were isolated there was no “datastore heartbeating” and no “network heartbeating”. Because the datastores were unavailable the lock on the VMDKs expired (virtual disk files) and HA would be able to restart the VMs.

Now please note that HA/ESXi will power-off (or kill actually) the “ghosted VM” (the host which runs the VMs that has lost network connection) when it detects the locks cannot be re-acquired. It still means that the time between when the restart happens and the time  when the isolation event is resolved potentially the IP Address and the Mac Address of the VM will pop up on the network. Of course this will only happen when your virtual machine network isn’t isolated, and as you can imagine this is not desired.

When you are running IP based storage, it is highly (!!) recommend to configure the isolation response to: power-off! For more details on configuring the isolation response please read this article which lists the best practices / recommendations.

What happens to powered off VMs when a host fails?

· ·

I had the question today what happens to a powered off VM when the host they are registered against fails? This customer always has multiple powered off VMs and was afraid their VMs would show up as orphaned. I was pretty confident that the VM would be re-registered against one of the remaining hosts in the cluster, but I validated it just in case and this is what the events section of the VM shows:

Relocating from cs-tkmt-h08, emc-vnx-fcoe to cs-tkmt-h05, emc-vnx-fcoe

In other words, the VM is relocated from my ESXi host cs-tkmt-h08 to cs-tkmt-h05. No need to worry about orphaned VMs and manually registering them against the remaining hosts… vSphere does it for you.