<edit>KB article has been published, including the video I shot</edit>
How do you setup multi-NIC vMotion? I had this question 3 times in the past couple of days during workshops so I figured it was worth explaining how to do this. It is fairly straight forward to be honest and it is more or less similar to how you would setup iSCSI with multiple vmknic’s. More or less as there is one distinct difference.
You will need to bind each VMkernel Interface (vmknic) to a physical NIC. In other words:
- Create a VMkernel Interface and give it the name “vMotion-01″
- Go to the settings of this Portgroup and configure 1 physical NIC-port as active and all others as “standby” (see the screenshot below for an example)
- Create a second VMkernel Interface and give it the name “vMotion-02″
- Go to the settings of this Portgroup and configure a different NIC-port as active and all others as “standby”
- and so on…
Now when you will initiate a vMotion multiple NIC ports can be used. Keep in mind that even when you vMotion just 1 virtual machine both links will be used. Also, if you don’t have dedicated links for vMotion you might want to consider using Network I/O Control. vMotion can saturate a link and at least when you’ve set up Network I/O Control and assigned the right amount of shares each type of traffic will get what it has been assigned.
For a video on how to do this:
<update: dvSwitch details below>
For people using dvSwitches it is fairly straight forward: You will need to create two dvPortgroups. These portgroup will need to have the “active/standby” setup (Teaming and Failover section). After that you will need to create two Virtual Adapters and bind each of these to a specific dvPortgroup.
And again the video on how to set this up:
Hello Duncan. Thanks for this great Guide to multi-nic vmotion in vSphere5. But for me a Question still exists… How should the VMkernel Nics configured at IP Side? IPs on the same Subnet for vmknic0 and vmknic1 ?
They can be on the same subnet, and that is probably easiest as well.
If they’re on the same subnet, which port on server 1 connects to which port on server 2 ?
I’d put them in different subnets so I can arrange for each network to stay on 1 switch. Otherwise vMotion might be taxing the inter-switch links unnecessarily.
Then this is not an option as you cannot predict which link it takes. But is the “cost” really that high for the average VM?
Hi Duncan,
Thanks alot! Could you maybe also explain how this works with a dvs, i got a dedicated vds with 2 physical nics..
You will need to create two dvPortgroups. These portgroup will need to have the “active/standby” setup (Teaming and Failover section). After that you will need to create two Virtual Adapters and bind each of these to a specific dvPortgroup.
I am confused. How is this different than just having 1 porgroup with 2 nics that are setup as teaming??
multiple VMkernels = multiple initiators for the traffic.
I think I see. I just thought a Teaming portgroup did the same thing.
If we have multiple VMkernel port groups with multiple nic per port group for teaming, would this improve vmotion comparing if having the same port group without teaming?
If we have multiple VMkernel port groups with multiple nic per port group for teaming, would this improve vmotion comparing if having the same number of port groups without teaming?
No it will not, but teaming would be preferred for resiliency
You also need to (if its a Cisco switch) create and etherchannel and hardcore it to on correct?
no you don’t. there are no requirements for the physical swithc.
two vmkernel nics + two physical nic ports is all you need.
Thank you! Great info
Incase this helps anyone, below is how I scripted Multi-NIC vMotion based on Duncan’s post above in the %firstboot section of my ks.cfg. I’m pretty sure it correct…
dcc
–begin cut & paste of %firstboot / ks.cfg –
# add vSwitch1 for vMotion (128 = 120 usable ports)
esxcli network vswitch standard add –ports 128 –vswitch-name vSwitch1
# attach vmnics to vSwitch1
esxcli network vswitch standard uplink add –uplink-name vmnic1 –vswitch-name vSwitch1
esxcli network vswitch standard uplink add –uplink-name vmnic5 –vswitch-name vSwitch1
# configure vMotion portgroup
esxcli network vswitch standard portgroup add –portgroup-name “vMotion01″ –vswitch-name vSwitch1
esxcli network vswitch standard portgroup add –portgroup-name “vMotion02″ –vswitch-name vSwitch1
# configure active uplinks for vSwitch1
esxcli network vswitch standard policy failover set –active-uplinks vmnic1,vmnic5 –vswitch-name vSwitch1
# configure active uplinks for multi-vnmic vMotion
esxcli network vswitch standard portgroup policy failover set –active-uplinks vmnic1 –standby-uplinks vmnic5 –portgroup-name vMotion01
esxcli network vswitch standard portgroup policy failover set –active-uplinks vmnic5 –standby-uplinks vmnic1 –portgroup-name vMotion02
# configure failure detection + load balancing for vSwitch1
esxcli network vswitch standard policy failover set –failback yes –failure-detection link –load-balancing iphash –notify-switches yes –vswitch-name vSwitch1
# configure failure detection + load balancing for multi-vnmic vMotion (use portid)
esxcli network vswitch standard portgroup policy failover set –failback yes –failure-detection link –load-balancing portid –notify-switches yes –portgroup-name vMotion01
esxcli network vswitch standard portgroup policy failover set –failback yes –failure-detection link –load-balancing portid –notify-switches yes –portgroup-name vMotion02
# configure vmkernel interface for vMotion traffic
esxcli network ip interface add –interface-name vmk1 –portgroup-name “vMotion01″
esxcli network ip interface ipv4 set –interface-name vmk1 –ipv4 172.16.3.101 –netmask 255.255.255.0 –type static
esxcli network ip interface add –interface-name vmk2 –portgroup-name “vMotion02″
esxcli network ip interface ipv4 set –interface-name vmk2 –ipv4 172.16.3.102 –netmask 255.255.255.0 –type static
vim-cmd hostsvc/vmotion/vnic_set vmk1
vim-cmd hostsvc/vmotion/vnic_set vmk2
–end cut & paste of %firstboot / ks.cfg –
Great article
Is there a limit how many VMkernel dvPortgroups you can create?
Example:
Converged adapter with 8 virtual vmnics created = Max 64 simultaneous vmotion?
True or false
sorry just read the answer in clustering deepdive:
1GbE = 16 nics supported
10GbE = 4 nics supported
Does 16nics for 1gbps means e.g. 2 nics for 16 vmkernel port groups with active passive, or 8 vmkernel port groups with active passive.
Hi Ducan
taking 1Gbps for this enquiry.
16 nics are supported for vmotion. does it means 16 active nics or number of nics used irregardless its active or passive ie a vmkernel port group setup wiht active passive so maximum of 8 port groups.
Thanks.
I tried to implement this configuration and have a problem. If I enable vMotion on one VMkernel port, it is automatically disabled on the other one. I would assume that I need both VMkernel ports with vMotion enabled. How do I manage that?
In addition to my previous post, I will show you the PowerCLI script that I used to create the vSwitch.
[sourcecode language="powershell"]
$VMHost = Get-VMhost MyHost.MyDomain.com
$VMotionIP1 = ’192.168.0.1′
$VMotionIP2 = ’192.168.0.2′
$VMotionSubnetMask = ’255.255.255.0′
$VMotionVLan = 0
$vSwitch2Name = ‘vSwitch2′
$vSwitch2Nics = ‘vmnic0′,’vmnic1′
$VMotionPortGroupName1 = ‘vMotion-01′
$VMotionPortGroupName2 = ‘vMotion-02′
# Configure virtual switch for VMotion
$vswitch = New-VirtualSwitch -VMHost $VMHost -Name $vSwitch2Name -Nic $vSwitch2Nics
$vportgroup1 = New-VirtualPortGroup -VirtualSwitch $vswitch -Name $VMotionPortGroupName1 -VLanId $VMotionVLan
$vportgroup2 = New-VirtualPortGroup -VirtualSwitch $vswitch -Name $VMotionPortGroupName2 -VLanId $VMotionVLan
$vNic1 = New-VMHostNetworkAdapter -VMHost $vmHost -VirtualSwitch $vswitch -PortGroup $vportgroup1 -IP $VMotionIP1 -SubnetMask $VMotionSubnetMask -VMotionEnabled:$true
$vNic2 = New-VMHostNetworkAdapter -VMHost $vmHost -VirtualSwitch $vswitch -PortGroup $vportgroup2 -IP $VMotionIP2 -SubnetMask $VMotionSubnetMask -VMotionEnabled:$true
$vportgroup1 | Get-NicTeamingPolicy | Set-NicTeamingPolicy -MakeNicActive $vSwitch2Nics[0] -MakeNicStandBy $vSwitch2Nics[1]
$vportgroup2 | Get-NicTeamingPolicy | Set-NicTeamingPolicy -MakeNicActive $vSwitch2Nics[1] -MakeNicStandBy $vSwitch2Nics[0]
[/sourcecode]
I think I solved the problem my self.
My first test was with an ESXi 4.1U1 server. There you can have only one vMotion enabled portgroup.
I tested it again with an ESXi 5.0 server. There you can have more than one vMotion enabled portgroups.
This is a nice new feature of vSphere 5. It gives us another good reason to migrate soon.
This is “vSphere 5″ only indeed…. Hence the reason I mentioned “vSphere 5″ in the title
Great article, can’t wait to be able to test and implement….Just curious on where this is within the vSphere 5 documentation, as I’m having trouble finding it.
Somehow it was left out. I have asked the Doc Team to document it and have created two videos for the KB. KBs should be online soon hopefully and Doc change will happen with the next update probably,
I’ve been doing some testing with vSphere 5.0 in my lab environment, specifically related to vMotion speeds.
The test I did consisted of two identical HP blades in the same chassis, all which had 4 x 1Gb physical links to the VSS core and two HBAs into the storage fabric. I tested the speed of vMotion, back and forth from one host to another, marking down the time it took each one, and taking an average as I increased the number of VMkernals (physical links). It should also be noted the VM, nor the host, were under any kind of load.
1 VMkernal: 21.2 seconds (5 tests)
2 VMkernals: 14.4 seconds (5 tests)
3 VMkernals: 13 seconds (5 tests)
4 VMkernals: 11.8 seconds (5 tests)
As you can see, as you add multiple VMkernals enabled for vMotion, the time it takes for a vMotion to complete is considerably less as you add multiple initiators for the traffic. Additionally, just moving to vSphere 5 (from 4.1) dropped the normal vMotion time almost in half from what we have now going across a single 1 Gb link.
I’m pleasantly surprised at the vMotion speed on 5 so far. I’ll be doing this same test again while the VM is under high load to compare again.
previously the suggestion was to have one vSwitch with two pNICs and have one active for management traffic(standby for vMotion) and one active for vMotion (standby for management).
how do you suggest setting up for multiple NIC vMotion?
do you still recommend one vSwitch with multiple vMotion vmkernels and one for management traffic, with the each NIC in standby mode for the other service?
thanks
Graham
I would like to know the answer to this one as well.
If you have two pNIC’s, is it okay to have Multiple-NIC vMotion setup and management traffic on the same vSwitch?
I would not share the Management NIC with the vMotion NICs to be honest. Although chances are slim it would interfere I would not take the risk.
Here is a link to the video and KB artibles
http://www.youtube.com/watch?v=n-XBof_K-b0
http://kb.vmware.com/kb/2007467
Multiple-NIC vMotion in vSphere 5 with SQL Server
http://www.youtube.com/emcprovensolutions#p/u/1/X8AqMhdz3OE
thanks for the links, but it was more of a design question rather than a how-to question.
anyone able to advise?
thanks
Graham
can anyone offer some help?
thanks
My answer was : don’t share the NICs between vMotion and Management, if you do use Network IO Control to guarantee bandwidth to your management network.
Thanks for the reply Duncan, and sorry I missed the first reply.
I have 4 x GbE Nics to use between mgmt and vmotion.
I could set up two vswitches. One for mgmt with two nics, one active and one for failover in standby. Second vswitch for vmotion with two active vmkernel ports.
I thought it may be better to set up one vswitch. 3 vmkernel ports for vmotion and one for mgmt traffic each using one nic. I could then allow the vmotion ports to carry mgmt traffic for failover in standby mode. Under normal conditions this would allow one extra vmkernel port for vmotion.
The only time both traffic would go over the same nic is if the primary mgmt nic failed.
Is this not a good idea, and have I missed something?
Thanks
Graham
I know this reply is a little late, but I just enabled this configuration in my environment, and I saw some guge performance gains (almost 2x as fast).
My setup is 2 identical Dell R910s, going to two Dell 5424 switches, with vMotion traffic in a private VLAN, with a 2 port LAG in between the switches. Each R910 has 2 physical NICs dedicated for vMotion (one going to each switch). As per the article, one VMK port per NIC.
My results:
Windows Server 08R2 VM1 – 4GB RAM:
-2 vMotion Tests (host A to B, then host B back to A. With 1 VMkernel port, the vMotions took 45 and 43 seconds each. With 2 VMkernel ports, the vMotions took 28 and 26 seconds each.
Windows Server 08R2 VM2 – 12GB RAM:
-2 vMotion Tests (host A to B, then host B back to A. With 1 MVkernel port, the vMotions both took 2:05. With 2 VMkernel ports, the vMotions took 1:12 and 1:04 each.
That’s a 45% speed improvement in what took me 5 mins to configure. I already use iSCSI, so this particular way of configuring and binding multiple vmk nics to physical NICs was very familiar.
Thanks Duncan!
I’m doing some testing with this and am seeing traffic only use one NIC. I only have one vSphere 5 host. Do both sides of the vMotion have to vSphere 5 for this to work?
Hallo
I cannot se both vmkernel ports on the same subnet, when i set vmotion dont work, vmkping not work cause second vmotion port doesnt route (vmk1):
~ # esxcfg-route -l
VMkernel Routes:
Network Netmask Gateway Interface
172.16.130.0 255.255.255.0 Local Subnet vmk2
172.16.131.0 255.255.255.0 Local Subnet vmk4
172.16.132.0 255.255.255.0 Local Subnet vmk0
192.168.21.0 255.255.255.0 Local Subnet vmk3
default 0.0.0.0 192.168.21.1 vmk3
~ # esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 VMotion IPv4 172.16.132.202 255.255.255.0 172.16.132.255 00:50:56:70:15:9a 1500 65535 true STATIC
vmk1 VMotion2 IPv4 172.16.132.22 255.255.255.0 172.16.132.255 00:50:56:7d:b2:9a 1500 65535 true STATIC
vmk2 SCSI IPv4 172.16.130.22 255.255.255.0 172.16.130.255 00:50:56:78:65:76 1500 65535 true STATIC
vmk3 Management Network IPv4 192.168.21.102 255.255.255.0 192.168.21.255 00:50:56:46:45:b6 1500 65535 true STATIC
vmk4 SCSI2 IPv4 172.16.131.22 255.255.255.0 172.16.131.255 00:50:56:7a:34:43 1500 65535 true STATIC
~ # esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 64 4 64 9000 vmnic0,vmnic1
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 64 7 64 1500 vmnic2,vmnic3
PortGroup Name VLAN ID Used Ports Uplinks
SCSI2 0 1 vmnic3
SCSI 0 1 vmnic2
VMotion2 0 1 vmnic3,vmnic2
VMotion 0 1 vmnic2,vmnic3
vSphere 4 on ESXi had an issue where if two IPs were assigned to a NIC team, and those IPs were in the same VLAN, a unicast storm would result: all NICs on all hosts that were on the vMotion network would flood.
In this configuration, if one of the NIC teams were to engage its secondary NIC, primary NIC failed, and thus placed the second IP on same NIC as the other Team was already using, does a similar unicast storm result when a vmotion is initiated on both NIC Teams?
Does vmotion on multiple nics work on esxi4 with port channeling? what policy do I need to set on the vswitch?
I do not know, I have not tried. My comment was based on the IPs used in the video. Referring to KB 1013077 that should result in different NICs being used for inbound and outbound traffic, and from experience, will cause a unicast storm.
If I had to guess for vSphere 4, create a vSwitch with two NICs. Add two vmotion ports. Make the first port active on the first adapter and standby on the second, and the second port active on the second adapter and standby on the first. Use one VLAN, but two IP address spaces, such as 10.x.y.z on the first port, and 192.x.y.z on the second port. Make sure no other kernel port is on either of the two IP subnets or the rules from KB 1013077 kick in.
Don’t know if that would work, or if its supported….let us know what your port counters say when you try it.
abit loss here. may i know what are the advantages of using multi-nic vmotion compared to a vmkernal with 2 nic teaming on load balance? will we be looking at better throughput?
What do you folks think about this for the 6-NIC setup with multiple vMotion? Our storage is fiber channel in this situation. Worst case scenario is if the quad PCIe card goes, then vMotion and Mgmt is on the same single pNIC. However, with NIOC vMotion bandwidth can be limited to ensure Mgmt traffic flows well.
OnBoard vmnic0
OnBoard vmnic1
PCIe vmnic2
PCIe vmnic3
PCIe vmnic4
PCIe vmnic5
vSwitch0
Mgmt
vmnic0 Primary
vmnic4 Standby
vmnic5 Standby
vMotion-1
vmnic0 Standby
vmnic4 Primary
vmnic5 Standby
vMotion-2
vmnic0 Standby
vmnic4 Standby
vmnic5 Primary
vSwitch1
LAN-1
vmnic1 Active
vmnic2 Active
vmnic3 Active
LAN-2
vmnic1 Active
vmnic2 Active
vmnic3 Active
so vMotion will never flow through nic0 (unless nic1 and 2 fail)? Why not add another vMotion network and use nic0 as well since you will be using NIOC?
EDIT: previous post had an error
so vMotion will never flow through nic0 (unless nic4 and 5 fail)? Why not add another vMotion network and use nic0 as well since you will be using NIOC? Just saying.
With your setup vMotion will use 2 nics, you could use 3 and since you are going to use NIOC, might as well use all links for blazing vMotion.. NIOC will make sure your mgmt stays afloat.
Will this concept work on other vmkernal ports like management and FT?
I tried it for FT and noticed it will only allow one port group enabled for FT and disables the other port group.
FYI. If you are having trouble getting this to work, the bug in KB 2008144 might be your reason. I had to use the no Failback workaround. This bug usually only manifests itself after a reboot of the host. The nics are bound correctly after things are initially setup, its only after a reboot do they all bind to just one nic. This can be witnessed by doing an ESXTOP and entering N to see the nics, you will see that they same NIC is bound to both VMK ports. Note: this bug also effects iSCSI port binding. Took 3 weeks and 3 VMware cases to identify this Bug.
@Vision: No it does not work for FT in 5.0
Duncan,
Any idea if Multi NIC vMotion is possible on a Nexus 1000v? I reached out to Cisco to see if they have any information but I figured I would ask here as well.
@Kalen: don’t know to be honest… Never tested and never seen any info.
So am I right to assume that when you run this configuration you lose redundancy on your vMotion.
I have 2 nics strictly for vMotion, each to a seperate switch.
No, you would normally have 2 vmkernel’s for vmotion. that is where the redundancy comes in!
Wonder if I have something configured wrong then. Whenever a link is down the vmotion fails not connecting to the IP address of the down card.
I have to remove the vmkernal in order to get vmotion to work (of course just over the 1 link) If both links are up it works great and much faster.
I think I got it….In the picture all the Nic Teaming Policy exceptions are checked, but in the video they are unchecked. Mine were unchecked and didn’t seem to work unless both links were live.
Craig ran into the same issue this Friday afternoon – just upgraded from 4.1 to 5 to discover the vswitch bugs – and there are many.
NOTE the KB you refer to is wrong!! Spent over an hour on the phone with tech , they stated that the KB is NOT the prefered method – they further stated that each nic port group needs to be assigned to a unique / seperate vswitch…….
this is supposed to be fixed in SP1, but that isn’t due out for a couple of months!!!
So many people / organisations have been caught out by this and it hasn’t been properly advertised. The upgrade QC process wasn’t followed so v5 is properly broken in a very fundamental way. If you haven’t upgraded yet, I’d hold fire until SP1 has ben released, save yourself the hassle.