It is complex process as we need vMotion to able to handle mixes of 10GbE and 1GbE NICs.

When we start the process we will check, from the vCenter side, each host and determine the total combined pool of bandwidth available for vMotion. In other words, if you have 2x1GbE NICs and 1x10GbE NIC, then that host has a pool of 12GbE worth of bandwidth. We will do the same for the source and the destination host. Then, we will walk down each host’s list of vMotion vmknics, pairing off NICs until we’ve exhausted the bandwidth pool.

There are many combinations possible, but lets discuss a few just to provide a better idea of how this works:

• If the source host has 1x1GbE NIC and the dest 1x1GbE NIC, we’ll open one connection between the these two hosts.
• If the source has 3x1GbE NICs and the destination 1x10GbE NIC, then we’ll open one connection from each source-side 1GbE NIC to the destination’s 10GbE NIC – so a total of three socket connections all to the dest’s single 10GbE NIC.
• If the source has 15x1GbE NICs and the destination 1x10GbE NIC and 5x1GbE NICs, then we’ll direct the first 10 source-side 1GbE NICs to connect to the dest’s 10GbE NIC, then the remaining pair of 5 1GbE vmknics will connect to each other – 15 connections in all.

Keep in mind that if the hosts are mismatched, we will create connections between vmknics until one of the sides is “depleted”. In other words if the source has 2 x 1GbE and the destination 1 x 1GbE only 1 connection would be opened.

How do you setup multi-NIC vMotion? I had this question 3 times in the past couple of days during workshops so I figured it was worth explaining how to do this. It is fairly straight forward to be honest and it is more or less similar to how you would setup iSCSI with multiple vmknic’s. More or less as there is one distinct difference.

You will need to bind each VMkernel Interface (vmknic) to a physical NIC. In other words:

• Create a VMkernel Interface and give it the name “vMotion-01″
• Go to the settings of this Portgroup and configure 1 physical NIC-port as active and all others as “standby” (see the screenshot below for an example)
• Create a second VMkernel Interface and give it the name “vMotion-02″
• Go to the settings of this Portgroup and configure a different NIC-port as active and all others as “standby”
• and so on…

Now when you will initiate a vMotion multiple NIC ports can be used. Keep in mind that even when you vMotion just 1 virtual machine both links will be used. Also, if you don’t have dedicated links for vMotion you might want to consider using Network I/O Control. vMotion can saturate a link and at least when you’ve set up Network I/O Control and assigned the right amount of shares each type of traffic will get what it has been assigned.

For a video on how to do this:

<update: dvSwitch details below>

For people using dvSwitches it is fairly straight forward: You will need to create two dvPortgroups. These portgroup will need to have the “active/standby” setup (Teaming and Failover section). After that you will need to create two Virtual Adapters and bind each of these to a specific dvPortgroup.

And again the video on how to set this up:

There’s a new component that is part of vMotion which is only enabled with Enterprise Plus and that components is what we call ‘Metro vMotion’.  This feature enables you to safely vMotion a virtual machine across a link of up to 10 miliseconds RTT. The technique used is common practice in networking and a bit more in-depth described here.

In the case of vMotion the standard socket buffer size is around 0.5MB.  Assuming a 1GbE network (or 125MBps) then bandwidth delay product dictates that we could support roughly 5ms RTT delay without a noticeable bandwidth impact.  With the “Metro vMotion” feature, we’ll dynamically resize the socket buffers based on the observed RTT over the vMotion network.  So, if you have 10ms delay, the socket buffers will be resized to 1.25MB, allowing full 125MBps throughput.  Without “Metro vMotion”, over the same 10ms link, you would get around 50MBps throughput.

Is that cool or what?

There are some fundamental changes when it comes to vMotion scalability and performance in vSphere 5.0. Most of these changes have one common goal: being able to vMotion ANY type of workload. It doesn’t matter if you have a virtual machine with 32GB of memory that is rapidly changing memory pages any more with the the following enhancements:

• Multi-NIC vMotion support
• Stun During Page Send (SDPS)

Multi-NIC vMotion Support

One of the most substantial and visible changes is multi-NIC vMotion capabilities. vMotion is now capable of using multiple NICs concurrently to decrease the amount of time a vMotion takes. That means that even a single vMotion can leverage all of the configured vMotion NICs. Prior vSphere 5.0, only a single NIC was used for a vMotion enabled VMkernel. Enabling multiple NICs for your vMotion enabled VMkernel’s will remove some of the constraints from a bandwidth/throughput perspective that are associated with large and memory active virtual machines. The following list shows the currently supported maximum number of NICs for multi-NIC vMotion:

• 1GbE – 16 NICs supported
• 10GbE – 4 NICs supported

It is important to realize that in the case of 10GbE interfaces, it is only possible to use the full bandwidth when the server is equipped with the latest PCI Express busses. Ensure that your server hardware is capable of taking full advantage of these capabilities when this is a requirement.

Stun During Page Send

A couple of months back I described this cool vSphere 4.1 vMotion enhancement called Quick Resume and now it is replaced with Stun During Page Send, or also often referred to as “Slowdown During Page Send” is a feature that “slowsd own” the vCPU of the virtual machine that is being vMotioned. Simply said, vMotion will track the rate at which the guest pages are changed, or as the engineers prefer to call it, “dirtied”. The rate at which this occurs is compared to the vMotion transmission rate. If the rate at which the pages are dirtied exceeds the transmission rate, the source vCPUs will be placed in a sleep state to decrease the rate at which pages are dirtied and to allow the vMotion process to complete. It is good to know that the vCPUs will only be put to sleep for a few milliseconds at a time at most. SDPS injects frequent, tiny sleeps, disrupting the virtual machine’s workload just enough to guarantee vMotion can keep up with the memory page change rate to allow for a successful and non-disruptive completion of the process. You could say that, thanks to SDPS, you can vMotion any type of workload regardless of how aggressive it is.

It is important to realize that SDPS only slows down a virtual machine in the cases where the memory page change rate would have previously caused a vMotion to fail.

This technology is also what enables the increase in accepted latency for long distance vMotion. Pre-vSphere 5.0, the maximum supported latency for vMotion was 5ms. As you can imagine, this restricted many customers from enabling cross-site clusters. As of vSphere 5.0, the maximum supported latency has been doubled to 10ms for environments using Enterprise Plus. This should allow more customers to enable DRS between sites when all the required infrastructure components are available like, for instance, shared storage.

Selection Resource Pools?

I guess we have all cursed when we had to manually migrate VMs around and accidentally selected the wrong Resource Pool. This operational “problem” has finally been resolved and I am very happy about it. As of 5.0 the “source” resource pool will automatically be selected. Of course it is still possible to override this and to select a different resource pool but in most cases “next – next – finish” will be just fine.

Quick Resume enables the source virtual machines to be stunned while starting the destination virtual machine before all pages have copied. However, as the virtual machine is already running at the destination it could possibly attempt to touch (read or write) a page which hasn’t been copied yet. In that case Quick Resume requests the page from the source to allow the guest to complete the action while continuously copying the remaining memory pages until all pages are migrated. But what if the network would fail at that point, wouldn’t you end up with a destination virtual machine which cannot access certain memory pages anymore as they are “living” remotely? Just like Storage IO Control, vMotion leverages shared storage. A special file would be created in the case Quick Resume is used and this file is basically used as a backup buffer. In the case the network would fail this file would allow for the migration to complete. This file is typically in the order of just a couple MBs. Besides being used as a buffer for transferring the memory pages it also enables bi-directional communication between the two hosts allowing the vMotion to complete as though the network hadn’t failed. Is that cool or what?

The typical question that arises immediately  is if this will impact performance? It is good to realize that without Quick Resume vMotioning large memory active virtual machines would be difficult. The switch-over time could potentially be too large and lead to temporary loss of connection with the virtual machine. Although Quick Resume will impact performance when pages that are not copied yet are accessed, the benefits of being able to vMotion very large virtual machines with minimal impact by far outweigh this temporary increase of memory access time.

There is so many cool features and enhancements in vSphere that I just keep being amazed.

Now this might be because the story isn’t as exciting as we hope it  will be or because no one took the time to document it. In my opinion however VMotion is still one of the key features VMware offers but even more important it is what revolutionized the IT world. I think it is a great part of VMware history and probably the turning point  for the company. For me personally VMotion literally is what made me decide, years ago, to adopt virtualization and I am certain this goes for many others.

At VMworld I asked around who was mainly responsible for VMotion back in the days but no one really had a clear answer until I bumped into Kit Colbert. Kit, who was still an intern back then, worked closely with the person who originally developed VMotion. I decided to contact the engineer and asked him if he was willing to share the story as there are a million myths floating around.

Before I reveal the real story about how VMotion came to life I want to thank Mike Nelson for revolutionizing the world of IT and taking the time to share this with me and allowing me to share it with the rest of the world. Here is the true story of VMotion:

A bunch of us at VMware came from academia where process migration was popular but never worked in any mainstream OS because it had too many external dependencies to take care of.  The VMware platform on the other  hand provided the ability to encapsulate all of the state of a virtual machine.  This was proven with checkpointing; where we were able to checkpoint a virtual machine, copy the state to another host, and then resume it.  It was an obvious next step that if we could checkpoint to disk and resume on another machine that we should be able to checkpoint over the network to another machine and resume.

During the design phase for what would later become Virtual Center a couple of us discussed the notion of virtual machine migration.  I took the lead and wrote up some design notes. I managed to extract myself from the mainline development of ESX 2.0 and I decided to go off and build a virtual machine migration prototype.  I was able to build a prototype fairly quickly because we already had checkpointing support.  However, of course there was a lot more work done by myself and others to turn the prototype into a high quality product.

I needed something to demo it so I used the pinball application on Windows.  The only interactive app I had on my virtual machines was pinball.  I had two machines side by side each with a display. I would start pinball on a virtual machine on one physical machine.  Then I would start the migration and keep playing pinball. When the pre-copy of memory was done it would pause for a second and then resume on the other machine. I would then keep playing pinball on the other machine.

That’s the VMotion story. Basically VMware had built the underlying technology that made VMotion possible.  All it required was someone to take the time to exploit this technology and build VMotion.

-Mike

The funny thing is that although this might have been the obvious next step for VMware engineering it is something that “shocked” many of us. Most of us will still remember the first time they heard about VMotion or remember it being demoed, and as I stated it is the feature that convinced me to adopt virtualization at large scale, or better said it is responsible for me ending up here! In my case the demo was fairly “simple” as we VMotioned a Windows VM, however we had an RDP session open to the VM and of course we were convinced the session would be dropped. I think we did the actual VMotion more than 10 times as we couldn’t believe it actually worked.

Now I am not the only one who was flabbergasted by this great piece of technology of course hence the reason I reached out to a couple of the well known bloggers and asked if they could tell their VMotion story/confessions…

Chad Sakac, virtualgeek.typepad.com

For me, while I remember being amazed from a generic bland use case, the “this is going to change everything” moment occured for me in 2007.

If vmotion is about non-disruptive workload mobility (an amazing concept), where things get crazy cool for me are scenarioes and definitions of “workload” and “mobility” are stretched.

In early 2007, I was in the basement of my house playing with early prototypes of the Celerra VSA running on ESX whiteboxes. It was one of those now-common “russian doll” scenarios where the host powering the VSA was in turn being supported by an iSCSI LUN being presented by the VSA, which in turn supported other VMs. While intellectually obvious that VMotion **should** work, it was never the less amazing to see in it action, with no dropped connection under load.

At that moment, I realized that the workload could be as broad a definition as I wanted, including full blown stacks normally associated with “hardware” like arrays. It was also an “aha” that this could transform a million use cases not normally associated with a server workload.

Ironic side note – the next day, I was showing that concept in the boardroom during a discussion why all our stacks needed to be encapsulated and virtualized. Turns out they were already working on it

That all said – those “aha moments” happen constantly. Another example – this time more recently – was about stretching the definition of “mobility”. It was during the run-up to VMworld 2010, when we were doing the demo work for the VERY long distance vMotion scenarios with early prototypes of VPLEX Geo. As we dialed up the latency between the ESX hosts on the network and storage – I was very curious to where it would blow up. When it made it past 44ms RTT (for math/physics folks – that’s the latency equivalent of 13,000km at the speed of light!), it was a “wow” moment (BTW, it blew up at 80ms ) I need to point out here that it completely violates the VMware support position (and for many, many good reasons – one “it worked in this narrow case in the lab” does not equal “works in the real world”), so don’t try this at home.

BUT it highlighted how, over time, the idea of non-disruptive workload mobility over what TODAY are consider crazy distances, network, and storage configs will tomorrow be considered normal.

vMotion and svMotion never cease to amaze me.

Nothing less than expected of course, some crazy scenario and as Chad states it isn’t supported but it definitely shows the potential of the technology!

Frank Denneman, frankdenneman.nl

During our VCDX sessions in Copenhagen we spoke about  things in your life you would always remember. My reply was ; Seeing  Return of the Jedi in the cinema, the falling of the Berlin wall, 9/11, Pim Fortuyn murder and witnessing vMotion in action for the first time.

I clearly remember my colleague screaming through the wall that separated our office. “Frank do you really want to see something cool?” As an MS exchange admin/architect responsible for a global spanning exchange infrastructure nothing really could impress me those days but  giving him the benefit of the doubt I walked over. Peter sitting there grinning like a madman, offered me a seat, because he thought it was better to sit down. He opened a dos prompt, triggered a continuous ping and showed the virtual infrastructure explaining the current location of the virtual machine. As he started to migrate the virtual machine he instructed me to keep tracking the continuous ping, after the one ping loss he explained the virtual machine was up and running on the other host and to prove me, he powered-down the ESX host. I just leaped out of my seat, said some words I cannot repeat online and was basically sold. I think we  migrated the virtual machine all day long, inviting anyone who passed by our office to see the best show on earth. No explanation needed of course, but from that point I was hooked on virtualization and the rest is history.

I still enjoy explaining people the technology of vMotion and it still ranks in my book as one of the most-kick-ass technologies available today. As Mendel explained in the keynote of VMworld 2006 demonstrating recording an execution stream (now FT), we have the technology and the platform available to do anything we want, the problem is we still haven’t reached the boundaries of our creativity, I fully concur and I think we still haven’t reached the full potential of vMotion.  Heck, I’m off to my lab just to vMotion a bunch of virtual machines.

Can I thank Peter for introducing Frank to the wonderful world of virtualization?

Mike Laverick, rtfm-ed.co.uk

My first VMotion was  demo of media server being moved from one ESX hosts to another – with the buffering switched off. I forget now what movie clip was being shown to the desktops – I think it might have been a Men In Black trailer. Anyway, nothing flickered and nothing stopped – the video just kept on playing without a hick-up.

At that point my mind began to race. I was thinking initially about hardware maintenance. But quickly (this in in ESX2) days began to think of moving VMs around to improve performance, and possibility of moving VMs across large distances. At the time I told my Microsoft chums all about this, and they were very skeptical. Virtualization, they (mis)informed me, was going to be a flash in the pan, and that VMotion was some kind of toy – of course, in a Road to Damascus way now HyperV supports “Live Migrate” its an integral part of virtualization. In truth when I started to demo VMotion to my students occasionally I felt like I was show-boating. This was in the vCenter 1.x days. But in some respects there’s no harm in showboating. It allowed me to demonstrate to students how far ahead VMware was against the competition, and what a visionary are company VMware is. It certainly added to my credibility to have a technology that was so easy to setup (so long as you meet the basic pre-requisites) and the great thing about VMware and the courses is that you didn’t have to “hard sell” the product sold itself.

On a more humorous note I’ve seen all kind of wacky VMotion setups. I once had two PIII servers with a shared DEC JBOD with SCSI personality cables (circa NT4 Clustering configuration) just to get the shared storage running. I managed to get VMotion working with 1997 era equipment. I’ve also been asked by student – who had laser line-of-site connectivity between two buildings – if he could VMotion between them. I laughed and said as long as he could  meet the pre-requisites there would be no reason why not. Although it would definitely be unsupported. Then I smiled and said, if he ever got VMotion working – I would come round in my Dr Evil outfit to explain – VMotion – with laser.

As I already stated, but re-enforced by Mike… VMotion changed the world, and the fact that both Microsoft and Citrix copied the feature definitely supports that claim… now I am wondering if the VMotion across “laser line-of-site connectivity” actually worked or not!

Scott Lowe, blog.scottlowe.org

I remember when I first started testing vMotion (then VMotion, of course). I was absolutely sure that it had to be a trick–surely you can’t move a running workload from 1 physical server to another! I performed my first vMotion with just a standard Windows 2000 server build. It worked as expected. So I tried a Citrix Metaframe server with users logged in. It worked, too. Then I tried a file server while copying files to and from the server. Again, it worked. SSH? Worked. Telnet? Worked. Media server with clients streaming content? Web server while users were accessing pages and downloading files? Active Directory? Solaris? Linux? Everything worked. At this point, after days–even weeks–of unsuccessfully trying to make it fail, I was sold. I was officially hooked on virtualization with VMware.

Thanks for the invitation to share memories about vMotion!

It appears that all top bloggers got hooked on virtualization when they witnessed a VMotion… As I stated at the beginning of this post; VMotion revolutionized the world of IT and I would like to thank VMware and especially Mike Nelson for this great gift! I also like to thank Scott, Mike, Frank and Chad for sharing their stories and I bet many of you are currently having flashbacks of when you first witnessed a VMotion.

Now I remember when this used to be part of the VMware documentation but that requirement is nowhere to be found anywhere. I even have a memory of documentation of the previous versions stating that it was “recommended” to have layer-2 adjacency but even that is nowhere to be found. The only reference I could find was an article by Scott Lowe where Paul Pindell from F5 chips in and debunks the myth, but as Paul is not a VMware spokes person it is not definitive in my opinion. Scott also just published a rectification of his article after we discussed this myth a couple of times over the last week.

So what are the current Networking Requirements around vMotion according to VMware’s documentation?

• On each host, configure a VMkernel port group for vMotion
• Ensure that virtual machines have access to the same subnets on source and destination hosts
• Ensure that the network labels used for virtual machine port groups are consistent across hosts

Now that got me thinking, why would it even be a requirement? As far as I know vMotion is all layer three today, and besides that the vmkernel interface even has the option to specify a gateway. On top of that vMotion does not check if the source vmkernel interface is on the same subnet as the destination interface, so why would we care?

Now that makes me wonder where this myth is coming from… Have we all assumed L2 adjacency was a requirement? Have the requirements changed over time? Has the best practice changed?

Well one of those is easy to answer; no the best practice hasn’t changed. Minimize the amount of hops needed to reduce latency, is and always will be, a best practice. Will vMotion work when your vmkernels are in two different subnets, yes it will. Is it supported? No it is not as it has not explicitly gone through VMware’s QA process. However, I have had several discussions with engineering and they promised me a more conclusive statement will be added to our documentation and the KB in order to avoid any misunderstanding.

Hopefully this will debunk this myth that has been floating around for long enough once and for all. As stated, it will work it just hasn’t gone through QA and as such cannot be supported by VMware at this point in time. I am confident though that over time this statement will change to increase flexibility.

References:

• Integrating VMs into the Cisco Datacenter Architecture (ESX 2.5)
• Deploying BIG-IP to enable LD vMotion
• vMotion Practicality
• http://kb.vmware.com/kb/1002662

The following procedure is used to form a set of recommendations to correct the imbalanced cluster:

While (load imbalance metric > threshold) {
move = GetBestMove();
  If no good migration is found:
    stop;
  Else:
    Add move to the list of recommendations;
    Update cluster to the state after the move is added;
}

Step by step in plain English:

While the cluster is imbalanced (Current host load standard deviation > Target host load standard deviation) select a VM to migrate based on specific criteria and simulate a move and recompute the “Current host load standard deviation” and add to the migration recommendation list. If the cluster is still imbalanced(Current host load standard deviation > Target host load standard deviation) repeat procedure.

Now how does DRS select the best VM to move? DRS uses the following procedure:

GetBestMove() {
  For each VM v:
    For each host h that is not Source Host:
      If h is lightly loaded compared to Source Host:
      If Cost Benefit and Risk Analysis accepted
      simulate move v to h
      measure new cluster-wide load imbalance metric as g
  Return move v that gives least cluster-wide imbalance g.
}

Again in plain English:

For each VM check if a VMotion to each of the hosts which are less utilized than source host would result in a less imbalanced cluster and meets the Cost Benefit and Risk Analysis criteria. Compare the outcome of all tried combinations(VM<->Host) and return the VMotion that results in the least cluster imbalance.

This should result in a migration which gives the most improvement in terms of cluster balance, in other words: most bang for the buck! This is the reason why usually the larger VMs are moved as they will most likely decrease “Current host load standard deviation” the most. If it’s not enough to balance the cluster within the given threshold the “GetBestMove” gets executed again by the procedure which is used to form a set of recommendations.

Now the next question would be what does “Cost Benefit” and “Risk Analysis” consist of and why are we doing this?

First of all we want to avoid a constant stream of VMotions and this will be done by weighing costs vs benefits vs risks. These consists of:

• Cost benefit
  Cost: CPU reserved during migration on t he target host
  Cost: Memory consumed by shadow VM during VMotion on the target host
  Cost: VM “downtime” during the VMotion
  Benefit: More resources available on source host due to migration
  Benefit: More resources for migrated VM as it moves to a less utilized host
  Benefit: Cluster Balance
• Risk Analysis
  Stable vs unstable workload of the VM (historic info used)

Based on these consideration a cost-benefit-risk metric will be calculated and if this has an acceptable value the VM will be consider for migration.

I will consolidate both post in a single blog page today to make it easier to find!

This article will focus on two primary DRS  functions:

1. Load balancing VMs due to imbalanced Cluster
2. VM Placement when booting

I will not be focusing on Resource Pools at all as I feel that there are already more than enough articles which explain these. The Resource Management Guide also contains a wealth of info on resource pools and this should be your starting place!

Load Balancing

First of all VMware DRS evaluates your cluster every 5 minutes. If there’s an imbalance in load it will reorganize your cluster, with the help of VMotion, to create an evenly balanced cluster again. So how does it detect an imbalanced Cluster? First of all let’s start with a screenshot:

fig 1

There are three major elements here:

1. Migration Threshold
2. Target host load standard deviation
3. Current host load standard deviation

Keep in mind that when you change the “Migration Threshold” the value of the “Target host load standard deviation” will also change. In other words the Migration Threshold dictates how much the cluster can be “imbalanced”. There also appears to be a direct relationship between the amount of hosts in a cluster and the “Target host load standard deviation”. However, I haven’t found any reference to support this observation. (Two host cluster with threshold set to three has a THLSD of 0.2, a three host cluster has a THLSD of 0.163.) As said every 5 minutes DRS will calculate the sum of the resource entitlements of all virtual machines on a single host and divides that number by the capacity of the host:

sum(expected VM loads) / (capacity of host)

The result of all hosts will then be used to compute an average and the standard deviation. (Which effectively is the “Current host load standard deviation” you see in the screenshot(fig1).) I’m not going to explain what a standard deviation is as it’s explained extensively on Wiki.

If the environment is imbalanced and the Current host load standard deviation exceeds the value of the “Target host load standard deviation” DRS will either recommend migrations or perform migrations depending on the chosen setting.

Every migration recommendation will get a priority rating. This priority rating is based on the Current host load standard deviation. The actual algorithm being used to determine this is described in this KB article. I needed to read the article 134 times before I actually understood what they were trying to explain so I will use an example based on the info shown in the screenshot(fig1). Just to make sure it’s absolutely clear, LoadImbalanceMetric is the Current host load standard deviation value and ceil is basically a “round up”. The formula mentioned in the KB article followed by an example based on the screenshot(fig1):

6 - ceil(LoadImbalanceMetric / 0.1 * sqrt(NumberOfHostsInCluster))

6 - ceil(0.022 / 0.1 * sqrt(3))

This would result in a priority level of 5 for the migration recommendation if the cluster was imbalanced.

The only question left for me is how does DRS decide which VM it will VMotion… If anyone knows, feel free to chip in. I’ve already emailed the developers and when I receive a reply I will add it to this article and create a seperate article about the change so that it stands out.

VM Placement

The placement of a VM when being powered on is as you know part of DRS. DRS analyzes the cluster based on the algorithm described in “Load Balancing”. The question of course is for the VM which is being powered on what kind of values does DRS work with? Here’s the catch, DRS assumes that 100% of the provisioned resources for this VM will be used. DRS does not take limits or reservations into account. Just like HA, DRS has got “admission control”. If DRS can’t guarantee the full 100% of the resources provisioned for this VM can be used it will VMotion VMs away so that it can power on this single VM. If however there are not enough resources available it will not power on this VM.

That’s it for now… Like I said earlier, if you have more indepth details feel free to chip in as this is a grey area for most people.