I received a question last week about higher latency thresholds for vMotion… A rumor was floating around that vMotion would support RTT latency up to 10 miliseconds instead of 5. (RTT=Round Trip Time) Well this is partially true. With vSphere 5.0 Enterprise Plus this is true. With any of the versions below Enterprise Plus the supported limit is 5 miliseconds RTT. Is there a technical reason for this?
There’s a new component that is part of vMotion which is only enabled with Enterprise Plus and that components is what we call ‘Metro vMotion’. This feature enables you to safely vMotion a virtual machine across a link of up to 10 miliseconds RTT. The technique used is common practice in networking and a bit more in-depth described here.
In the case of vMotion the standard socket buffer size is around 0.5MB. Assuming a 1GbE network (or 125MBps) then bandwidth delay product dictates that we could support roughly 5ms RTT delay without a noticeable bandwidth impact. With the “Metro vMotion” feature, we’ll dynamically resize the socket buffers based on the observed RTT over the vMotion network. So, if you have 10ms delay, the socket buffers will be resized to 1.25MB, allowing full 125MBps throughput. Without “Metro vMotion”, over the same 10ms link, you would get around 50MBps throughput.
Is that cool or what?
Maybe a stupid question: Is there any technical reason why the latency could not be over 10ms
No it could easily be higher, this is probably what has been tested today.
While it is a cool feature, it sucks that it’s available only in the more expensive edition.
Businesses that don’t have as much money to spend on the Ent+ probably also don’t have as much money to spend on the highest quality link between their datacenters. They are the ones that would benefit the most from allowing higher latencies, get screwed. While those with more money probably don’t even need higher latency tolerance.
I know it is cool to complain about licensing and packaging today but if you can afford multiple sites with an array either spanning both sites or being presented to both sites surely you can spend some extra money on vSphere licenses.
Keep in mind that innovation doesn’t come cheap.
I know it’s cool to try to justify stupid licensing requirements but the guy has a point. This “feature” is only allowing you to VMotion across a link with latency, nothing else. Had VMware included some storage replication features within the Metro vMotion feature, then you’d have an argument.
Keep in mind you have to buy a license to get the vmotion feature, it’s not free. Insinuating that he’s cheap because he doesn’t want to pay 2x for the same functionality makes you look like an arrogant ass.
I am not saying that he is cheap those are your words. I am only saying that it is part of Enterprise+ and that people who have the type of environments requiring these techniques more than likely already use enterprise+ or should be using it. I might have phrased it poorly but no reason to start name calling.
If you feel there is something wrong with licensing and packaging take it up with your VMware Sales representative. I am not the right person for that and neither is it the focus of my blog. My focus is technology.
VMware changed licensing. Now we can migrate to 5.0 as soon as it is available https://www.vmware.com/files/pdf/vsphere_pricing.pdf.
Duncan, is there a date when VI5 will be available for download?
I cannot comment on that
With all the cash you saved by virtualizing and consolidating down your servers you should be able to afford it 😉
Is the footprint of esxi 5.0 still maintains 32 MB size? They have added many functionalities like firewall features etc. I suspect they cannot make it within 32 MB. Your thoughts?
not sure how relevant your question is with regards to this article, but no it is no 32MB but 100something. Don’t know the exact number.
32 MB was for ESXi 3.5, and it was around 64 MB in ESXi 4.x, and is about 144 MB in 5.0.
I suppose this is to allow the new feature ‘Long Distance vMotion’ ? Is it correct that vmotion now works across logical datacenter as well (probably not 100% related to this post, sorry).
Correct!
Cool – Thanks for the info. Despite the debate about cost and such this should be added to the feature list for Enterprise + so customers have more incentive to upgrade.
Duncan,
this isn’t exactly “innovation” as I could do the exact same thing with Hyper-V + VPLEX up to ~50ms RTT… Granted, that’s an uber-expensive way of looking at it but Hyper-V is much less sensitive to RTT in general than VMware. Wonder if I could do the same with KVM?
-dave
Yes I can do a vMotion across a link with a 500ms latency as well Dave. But these enhancements just like Stun During Page Send are about Memory Page Changes and the rate in which they do. It is not like any other technology can magically copy pages across a link in a pace faster than the network allows.
We have made sure to test our vMotions with our most difficult workloads across this latency to guarantee a good experience no matter the workload. For any larger latency, you will have to worry about a lot more than latency. So it is worthless to have vMotion support for 50ms without the supporting technologies to be able to make that experience worth while. Anyway, we have done our homework on this at VMware :)…
just to peggyback on Mike’s question and perhaps out of topics, but if somebody can make me a favor and answer to that “out of topic question”, I will really really enjoy the answer ..
Why in hell would you want to vmotion accros physical site ?
A lot of customers have sites which are a “metro” distance from each other (e.g. NY to NJ, etc.). For such customers they would like to treat the two sites as one logical site and be elastic between them. They purchase special storage that can span the two sites and also special network hw to be able to stretch their L2 network across. Along with dedicated bandwidth and all.
I’ve seen these types of configurations often in Germany (NetApp metro clusters mostly) and also in the Netherlands using netapp / lefthand / EMC.
The prefer a single site for disaster avoidance. if a site fails they can do HA between sites. and if an issue is about to occur they can vMotion everything to the other site.
thanks Duncan. Is this part of Ent Plus only? Because the official page (http://www.vmware.com/products/vsphere/buy/editions_comparison.html) does not state it.
BTW, cool feature!
Yes, E+ only
For reference, see VMware KB2005202 as it explicitly states that this is for Ent+ (if anyone needs documentation or written statement). 😉
Steve how about some professional courtesy to Duncan who has written books and provides a great deal of good info to the rest of us.
Just so you know that in vSphere 4.x Advanced there was no DRS functionality but VMWare has dropped Advanced in 5.x and gave those companies licensed with Advanced a free upgrade to Enterprise. Things filter down as versions progress which is great and you have to remember things cost money.
I’m sure you will find you can’t buy the same features on a top of the range Mercedes in a bottom of the range Ford. Yes it would be nice but if you want them pay for them.
Duncan great work mate.
Steve keep your emotion in check because nobody likes a smart ass or an asshole
Excellent blog Duncan, I got asked the question, what’s Metro vMotion, I answered and then thought ‘hang on check yellow-bricks”, we then got a load more info.
Also a message to Steve, See Olivers comment, he was spot on.
Because of the work that Duncan does, Yellow Bricks is always my first port of call.
I completely agree with Oliver on this one, Steve I think most of us are here because we have a certain respect and admiration to the level of work Duncan has been doing all these years .
It’s easier to sit in a corner and type some slang’s and make yourself look cool ( sarcasm), However the right attitude was yet again shown by Duncan by guiding you into right direction , My intention is not to post something irrelevant in this thread , as this not You Tube / Face Book, It’s just the respect to the Man, which drives me to make sure people like Steve avoid posting on technical forums .