Yellow Bricks

by Duncan Epping

vds

Can I use a custom TCP/IP stack for the vSAN Network?

· ·

I got this question today if you can use a custom TCP/IP stack for the vSAN Network, and I had this before. This question is usually asked by customers who have a stretched cluster as the stretched cluster forces them to create different routes in an L3 network. Especially with a large number of hosts this is cumbersome and error prone, unless you have everything automated end to end, but we are not all William Lam or Alan Renouf right 🙂

The answer is unfortunately: No, we do not support custom TCP/IP stacks for vSAN and there’s no vSAN TCP/IP stack either. The default TCP/IP stack has to be used, this is documented on storagehub.vmware.com in the excellent vSAN Networking guide which my colleagues Paudie and Cormac wrote. Of course for things like vMotion you can use the vMotion TCP/IP stack while using vSAN, that is fully supported.

Currently, vSphere does not include a dedicated TCP/IP stack for the vSAN traffic service nor the supportability for the creation of custom vSAN TCP/IP stack. To ensure vSAN traffic in Layer 3 network topologies leaves over the vSAN VMkernel network interface, administrators need to add the vSAN VMkernel network interface to the default TCP/IP Stack and define static routes for all of the vSAN cluster members.

vSphere 5.1 networking enhancements

· ·

There are many networking enhancement in vSphere 5.1 but I want to call out a couple specifically. The reason for this is that there have been many discussions on this blog about “hybrid VSS / VDS” environments as many were not comfortable with running everything on a VDS. Although the risks were minimal I could understand where people were coming from. So what’s new in this space?

  1. Management network rollback and recovery
  2. VDS config backup and restore
  3. Network health check

Management Network rollback / recovery says it all I guess. I for whatever reason you made changes that will result in your host not being able to connect to vCenter then this change will not be committed. Even more importantly, if you ever end up in the situation where your host is not able to connect to the network while using a VDS you can now reconfigure it through the DCUI (Network Restore Options). I played around with it, and I think it is a huge enhancement. I don’t see a reason to go hybrid any longer… go full VDS!

Another often heard complaint was around export/import of the VDS config or backup/restore. With vSphere 5.1 this ability is now added. Not only can you save the VDS config and use it for new VDS’s but you can of course also use this feature for backup purposes (see screenshot below). Another cool feature is that if you made a change to a portgroup that was not what you intended you can actually roll it back.

Last but not least is the “Network Health Check” option. I particularly like this feature as I’ve been in the situation many times in the past that changes were made on a physical level and people forgot to inform me about it. This will allow you to quickly identify when things changed and that will make the discussion with your networking colleagues a lot easier. In this release three things are checked:

  • VLAN
  • MTU
  • Network adapter teaming

These checks will be done every minute, and is done by sending probing packets on the VDS uplinks. If for whatever reason these probing packets fail it could indicate that the config of the physical components have changed. Nice right… I am not going to reveal any more secrets as I am guessing Venky will be writing some deepdive stuff soon.

In the mean while, for more details around what’s new I would like to refer to the great what’s new paper that Venky Deshpande wrote: What’s New for Network in vSphere 5.1.

KB article about SvMotion / VDS / HA problem republished with script to mitigate!

· ·

Just a second ago the GSS/KB team republished the KB article that explain the vSphere 5.0 problem around SvMotion / vDS / HA. I wrote about this problem various times and would like to refer to that for more details. What I want to point out here though is that the KB article now has a script attached which will help preventing problems until a full fixed is released. This script is basically the script that William Lam wrote, but it has been fully tested and vetted by VMware. For those running vSphere 5.0 and using SvMotion on VMs attached to distributed switches I urge you to use the script. I expect that the PowerCLI version will also be attached soon.

http://kb.vmware.com/kb/2013639

Migrating to a VDS switch when using etherchannels

· ·

Last week at PEX I had a discussion around migrating to a Distributed Switch (VDS) and some of the challenges one of our partners faced. During their migration they ran in to a lot of network problems, which made them decide to change back to a regular vSwitch. They were eager to start using the VDS but could not take the risk again to run in to the problems they faced.

I decided to grab a piece of paper and we quickly drew out the current implemented architecture at this customer site and discussed the steps the customer took to migrate. The steps described were exactly as the steps documented here and there was absolutely nothing wrong with it. At least not at first sight…. When we dived in to their architecture a bit more a crucial keyword popped up, ether channels. Why is this a problem? Well look at this process for a minute:

  • Create Distributed vSwitch
  • Create dvPortgroups
  • Remove vmnic from vSwitch0
  • Add vmnic to dvSwitch0
  • Move virtual machines to dvSwitch0 port group

Just imagine you are using an ether channel and traffic is being load balanced, but now you have one “leg” of the ether channel ending up in dvSwitch0 and one in vSwitch0. Yes not a pretty sight indeed. In this scenario the migration path would need to be:

  1. Create Distributed vSwitch
  2. Create dvPortgroups
  3. Remove  all the ports from the etherchannel configuration on the physical switch
  4. Change vSwitch load balancing from “IP Hash” to “Virtual Port ID”
  5. Remove vmnic from vSwitch0
  6. Add vmnic to dvSwitch0
  7. Move virtual machines to dvSwitch0 port group

For the second vmnic only steps 5 and 6 (and any subsequent NICs) would need to be repeated. After this the dvPor group can be configured to use “IP-hash” load balancing and the physical switch ports can be added to the etherchannel configuration again. You can repeat this for additional portgroups and VMkernel NICs.

I do want to point out, that I am personally not a huge fan of etherchannel configurations in virtual environments. One of the reason is the complexity which often leads to problems when things are misconfigured. An example of when things can go wrong is displayed above. If you don’t have any direct requirements to use IP-hash… use Load Based Teaming on your VDS instead, believe me it will make your life easier in the long run!

 

Distributed vSwitches and vCenter outage, what’s the deal?

· ·

Recently my colleague Venky Deshpande released  a whitepaper around VDS Best Practices. This white paper describes various architectural options when adopting a VDS only strategy. A strategy of which I can see the benefits. On Facebook multiple people made comments around why this would be a bad practice instead of a best practice, here are some of the comments:

“An ESX/ESXi host requires connectivity to vCenter Server to make vDS operations, such as powering on a VM to attach that VM’s network interface.”

“The issue is that if vCenter is a VM and changes hosts during a disaster (like a total power outage) and then is unable to grant itself a port to come back online.”

I figured the best way to debunk all these myths was to test it myself. I am confident that it is no problem, but I wanted to make sure that I could convince you. So what will I be testing?

  • Network connectivity after Powering-on a VM which is connected to a VDS while vCenter is down.
  • Network connectivity restore of vCenter attached to a VDS after a host failure.
  • Network connectivity restore of vCenter attached to a VDS after HA has moved the VM to a different host and restarted it.

Before we start I think it is useful to rehash something, which is different types of portgroups which is described in more depth in this KB:

  • Static binding – Port is immediately assigned and reserved for it when VM is connected to the dvPortgroup through vCenter. This happens during the provisioning of the virtual machine!
  • Dynamic binding – Port is assigned to a virtual machine only when the virtual machine is powered on and its NIC is in a connected state. The Port is disconnected when the virtual machine is powered off or the virtual machine’s NIC is disconnected. (Deprecated in 5.0)
  • Ephemeral binding – Port is created and assigned to a virtual machine when the virtual machine is powered on and its NIC is in a connected state. The Port is deleted when the virtual machine is powered off or the virtual machine’s NIC is disconnected. Ephemeral Port assignments can be made through ESX/ESXi as well as vCenter.

Hopefully this makes it clear straight away that their should be no problem at all, “Static Binding” is the default and even when vCenter is down a VM which has been provisioned before vCenter went down can easily be powered on and will have network access. I don’t mind spending some lab hours on this, so lets put this to a test. Lets use the defaults and see what the results are.

First I made sure all VMs were connected to a dvSwitch. I powered of a VM and checked the “Network settings and this is what it revealed… a port already assigned even when powered off:

This is not the only place you can see port assignments, you can verify it on the VDS’s “ports” tab:

Now lets test this, as that is ultimately what it is all about. First test, Network connectivity after Powering-on a VM which is connected to a VDS while vCenter is down:

  • Connected VM to dvPortgroup with static binding (is the default and best practice)
  • Power off VM
  • Power off vCenter VM
  • Connect vSphere Client to host
  • Power on VM
  • Ping VM –> Positive result
  • You can even see on the command line that this VM uses its assigned port: 
    esxcli network vswitch dvs vmware list
    Client: w2k8-001.eth0
    DVPortgroup ID: dvportgroup-516
    In Use: true
    Port ID: 137

Second test, Network connectivity restore of vCenter attached to a VDS after a host failure:

  • Connected vCenter VM to dvPortgroup with static binding (is the default and best practice)
  • Power off vCenter VM
  • Connect vSphere Client to host
  • Power on vCenter VM
  • Ping vCenter VM –> Positive result

Third test, Network connectivity restore of vCenter attached to a VDS after HA has moved the VM to a different host and restarted it.

  • Connected vCenter VM to dvPortgroup with static binding (is the default and best practice)
  • Yanked the cable out of the ESXi host on which vCenter was running
  • Opened a ping to the vCenter VM
  • HA re-registered the vCenter VM on a different host and powered it on
    • The re-register / power-on took roughly 45 – 60 seconds
  • Ping vCenter VM –> Positive result

I hope this debunks some of those myths floating around. I am the first to admit that there are still challenges out there, these will hopefully be addressed soon, but I can assure you that your virtual machines will regain connection as soon as they are powered on through HA or manually… yes even when your vCenter Server is down.