I have seen this question popping up in multiple places now, vMotion over VXLAN is it supported? I googled it and nothing turned up, so I figured I would write a short statement:
In vSphere 5.1 (and earlier) vMotion over VXLAN is not supported.
This statement might change in the future, it could be that in the next version vMotion traffic over a VXLAN wire will be supported, but with the current release it is not. Do note that vMotioning virtual machines which are attached to a VXLAN network is supported.
The next question people ask typically is, will it work? Yes it probably will, but again… it is not supported. Keep that in mind when you are designing a multi-site environment and want to use VXLAN.
Could we please change that into “vMotion over VXLAN is stupid and unnecessary”. The question is very close to “can we run FCoE over VXLAN?”
vMotion is an exchange of data between hypervisor kernels, VXLAN is a VM networking solution. Nuff said.
We could, but not sure people would be able to find it through google then :-).
I could see a scenario where a network could be more or less built around VXLAN. Take, for example, my home lab network built on 20Gb Infiniband, Cisco Nexus 1000v, and you guessed it: VXLAN. As an overlay technology that relies on multicast (something IB does support) instead of broadcasts (something IB doesn’t support), VXLAN could be used to actually virtualize Ethernet over an IB fabric! While solutions such as EoIB already exist today they have zero support within vSphere and a require separate physical appliance (at least today they do even though the gateway functionality could simply be implemented in software). On the flip side, IPoIB support in vSphere is already there and requires no additional appliance to get up and running. When you start to factor in the fact that Mellanox already has EoIB and IPoIB offload capabilities built into their HCAs, you start to wonder whether or not they could also extend hardware offloading to VXLAN with a new driver and possibly a hook into the vDS/N1Kv (at least I do). This also prompts the question: Do we already have all the hardware necessary to create high performance software defined networks today, and all we’re missing is that little bit of software to define it?
I’m not sure I agree it’s “stupid and unnecessary” Many folks haven’t invested in SRM yet, still have an n-tier approach to solution deployments, and are still using stretched clusters to satisfy DR and in some cases HA requirements. Perhaps my understanding of the technology isn’t deep enough yet, but it seems to me there are still many great reasons to ask the question.
vMotion runs over TCP, and the hypervisor hosts don’t have to be in the same subnet. You never needed L2 connectivity for vMotion traffic (although the official recommendation might have been to use one a while ago), just for VMs.
As I said above … 😉
Said another way – folks should be asking a better question, like: Can I use VXLAN and vMotion together to meet my business requirements. vMotion is a feature that requires L3 transport with specific technical requirements, VXLAN is another feature requiring L3 transport with specific technical requirements. While you could potentially run vMotion across a VXLAN backed network, ensure you examine your requirements thoroughly to ensure you have a real need it addresses.
Do note, that from a support standpoint routed vMotion traffic is ruled out. Yes it will work fine, been there done that… but officially it is not supported today.
It would be nice if we could configure additional routes on ESXi hosts more easily. I know in our environments, and I think it’s a common practice elsewhere, we isolate vMotion interfaces on a different subnet than the mgmt interface. Without mucking with the routing table on every host, I don’t think L3 vMotion would work. We might be able add the routes manually in the cli, or via startup scripts, but scaling that across many hosts is non-trivial and could be hard to troubleshoot.
Hi,
VxLan provide VM to VM communication solution , if we would like to have long distance VMotion we can use complementary solution like Cisco OTV and LISP. the first will take care of the layer 2 between the site and the 2nd will take care the routing problem to the correct datacenter base on /32 ip address.
to read more about this you can look at the following CVD:
http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DCI/4.0/EMC/EMC.pdf
As Ivan pointed out, several folks in the past had asked me, can VXLAN support vMotion. My answer was, well, it is only the VMs that care about L2-ajacency (for doing all kinds of crazy things), not the vMotion network that vSphere host uses for transferring VM’s state during vMotion . All that vMotion network (which is hypervisor facing) cares about is IP (L3) connectivity and you don’t really need VXLAN though technically it will work as VMotion network is a just a DVPort in VDS just like any other DVPort used by VMs.
For those who don’t realize, Mallik was one of the key people on VXLAN development. Welcome and thanks for commenting Mallik, much appreciated and looking forward to hear/see what you are working on now. (http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-00)
Thanks Duncan.
But I have to say that VMware should make this clear!
I can point to at least 3 VMworld presentation (2 from VMware Emloyees) + VTSP 5 official Training Material that specifically position vMotion WITH VXLAN!!!
I wonder if they just meant you can now vMotion your virtual machines to places where you couldn’t before due to the reasons VXLAN exists 😉 I can see how that would be confusing to customers.