• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Yellow Bricks

by Duncan Epping

  • Home
  • ESXTOP
  • Stickers/Shirts
  • Privacy Policy
  • About
  • Show Search
Hide Search

vSphere 5 – Metro vMotion

Duncan Epping · Aug 3, 2011 ·

I received a question last week about higher latency thresholds for vMotion… A rumor was floating around that vMotion would support RTT latency up to 10 miliseconds instead of 5. (RTT=Round Trip Time) Well this is partially true. With vSphere 5.0 Enterprise Plus this is true. With any of the versions below Enterprise Plus the supported limit is 5 miliseconds RTT. Is there a technical reason for this?

There’s a new component that is part of vMotion which is only enabled with Enterprise Plus and that components is what we call ‘Metro vMotion’.  This feature enables you to safely vMotion a virtual machine across a link of up to 10 miliseconds RTT. The technique used is common practice in networking and a bit more in-depth described here.

In the case of vMotion the standard socket buffer size is around 0.5MB.  Assuming a 1GbE network (or 125MBps) then bandwidth delay product dictates that we could support roughly 5ms RTT delay without a noticeable bandwidth impact.  With the “Metro vMotion” feature, we’ll dynamically resize the socket buffers based on the observed RTT over the vMotion network.  So, if you have 10ms delay, the socket buffers will be resized to 1.25MB, allowing full 125MBps throughput.  Without “Metro vMotion”, over the same 10ms link, you would get around 50MBps throughput.

Is that cool or what?

Share it:

  • Tweet

Related

BC-DR, cloud, Management & Automation 5, 5.0, metro, vmotion, vSphere

Reader Interactions

Comments

  1. Pascal says

    3 August, 2011 at 14:56

    Maybe a stupid question: Is there any technical reason why the latency could not be over 10ms

    • Duncan Epping says

      3 August, 2011 at 22:12

      No it could easily be higher, this is probably what has been tested today.

  2. Mxx says

    3 August, 2011 at 15:12

    While it is a cool feature, it sucks that it’s available only in the more expensive edition.
    Businesses that don’t have as much money to spend on the Ent+ probably also don’t have as much money to spend on the highest quality link between their datacenters. They are the ones that would benefit the most from allowing higher latencies, get screwed. While those with more money probably don’t even need higher latency tolerance.

    • Duncan Epping says

      3 August, 2011 at 16:26

      I know it is cool to complain about licensing and packaging today but if you can afford multiple sites with an array either spanning both sites or being presented to both sites surely you can spend some extra money on vSphere licenses.

      Keep in mind that innovation doesn’t come cheap.

      • Steve says

        3 August, 2011 at 19:35

        I know it’s cool to try to justify stupid licensing requirements but the guy has a point. This “feature” is only allowing you to VMotion across a link with latency, nothing else. Had VMware included some storage replication features within the Metro vMotion feature, then you’d have an argument.

        Keep in mind you have to buy a license to get the vmotion feature, it’s not free. Insinuating that he’s cheap because he doesn’t want to pay 2x for the same functionality makes you look like an arrogant ass.

        • Duncan Epping says

          3 August, 2011 at 22:12

          I am not saying that he is cheap those are your words. I am only saying that it is part of Enterprise+ and that people who have the type of environments requiring these techniques more than likely already use enterprise+ or should be using it. I might have phrased it poorly but no reason to start name calling.

          If you feel there is something wrong with licensing and packaging take it up with your VMware Sales representative. I am not the right person for that and neither is it the focus of my blog. My focus is technology.

          • Hendrik says

            4 August, 2011 at 13:35

            VMware changed licensing. Now we can migrate to 5.0 as soon as it is available https://www.vmware.com/files/pdf/vsphere_pricing.pdf.

            Duncan, is there a date when VI5 will be available for download?

          • Duncan Epping says

            4 August, 2011 at 14:11

            I cannot comment on that

        • FH says

          1 September, 2011 at 03:13

          With all the cash you saved by virtualizing and consolidating down your servers you should be able to afford it 😉

  3. Vinod Pant says

    3 August, 2011 at 17:09

    Is the footprint of esxi 5.0 still maintains 32 MB size? They have added many functionalities like firewall features etc. I suspect they cannot make it within 32 MB. Your thoughts?

    • Duncan says

      3 August, 2011 at 19:00

      not sure how relevant your question is with regards to this article, but no it is no 32MB but 100something. Don’t know the exact number.

      • Rickard Nobel says

        3 August, 2011 at 19:41

        32 MB was for ESXi 3.5, and it was around 64 MB in ESXi 4.x, and is about 144 MB in 5.0.

  4. Mike says

    4 August, 2011 at 18:48

    I suppose this is to allow the new feature ‘Long Distance vMotion’ ? Is it correct that vmotion now works across logical datacenter as well (probably not 100% related to this post, sorry).

    • Duncan Epping says

      5 August, 2011 at 00:02

      Correct!

  5. VMFan says

    4 August, 2011 at 22:58

    Cool – Thanks for the info. Despite the debate about cost and such this should be added to the feature list for Enterprise + so customers have more incentive to upgrade.

  6. Dave Graham says

    5 August, 2011 at 02:12

    Duncan,

    this isn’t exactly “innovation” as I could do the exact same thing with Hyper-V + VPLEX up to ~50ms RTT… Granted, that’s an uber-expensive way of looking at it but Hyper-V is much less sensitive to RTT in general than VMware. Wonder if I could do the same with KVM?

    -dave

    • Duncan Epping says

      5 August, 2011 at 07:43

      Yes I can do a vMotion across a link with a 500ms latency as well Dave. But these enhancements just like Stun During Page Send are about Memory Page Changes and the rate in which they do. It is not like any other technology can magically copy pages across a link in a pace faster than the network allows.

    • dilpreet says

      12 August, 2011 at 07:56

      We have made sure to test our vMotions with our most difficult workloads across this latency to guarantee a good experience no matter the workload. For any larger latency, you will have to worry about a lot more than latency. So it is worthless to have vMotion support for 50ms without the supporting technologies to be able to make that experience worth while. Anyway, we have done our homework on this at VMware :)…

  7. Dan Voyer says

    8 August, 2011 at 16:26

    just to peggyback on Mike’s question and perhaps out of topics, but if somebody can make me a favor and answer to that “out of topic question”, I will really really enjoy the answer ..

    Why in hell would you want to vmotion accros physical site ?

    • dilpreet says

      12 August, 2011 at 07:50

      A lot of customers have sites which are a “metro” distance from each other (e.g. NY to NJ, etc.). For such customers they would like to treat the two sites as one logical site and be elastic between them. They purchase special storage that can span the two sites and also special network hw to be able to stretch their L2 network across. Along with dedicated bandwidth and all.

    • Duncan Epping says

      12 August, 2011 at 08:15

      I’ve seen these types of configurations often in Germany (NetApp metro clusters mostly) and also in the Netherlands using netapp / lefthand / EMC.

      The prefer a single site for disaster avoidance. if a site fails they can do HA between sites. and if an issue is about to occur they can vMotion everything to the other site.

  8. iwan says

    30 August, 2011 at 13:37

    thanks Duncan. Is this part of Ent Plus only? Because the official page (http://www.vmware.com/products/vsphere/buy/editions_comparison.html) does not state it.

    BTW, cool feature!

    • Duncan Epping says

      30 August, 2011 at 15:46

      Yes, E+ only

    • Linus B. says

      4 November, 2011 at 01:30

      For reference, see VMware KB2005202 as it explicitly states that this is for Ent+ (if anyone needs documentation or written statement). 😉

  9. OliverK says

    7 December, 2011 at 02:36

    Steve how about some professional courtesy to Duncan who has written books and provides a great deal of good info to the rest of us.

    Just so you know that in vSphere 4.x Advanced there was no DRS functionality but VMWare has dropped Advanced in 5.x and gave those companies licensed with Advanced a free upgrade to Enterprise. Things filter down as versions progress which is great and you have to remember things cost money.

    I’m sure you will find you can’t buy the same features on a top of the range Mercedes in a bottom of the range Ford. Yes it would be nice but if you want them pay for them.

    Duncan great work mate.

    Steve keep your emotion in check because nobody likes a smart ass or an asshole

  10. Bryan O'Connor says

    22 June, 2012 at 12:16

    Excellent blog Duncan, I got asked the question, what’s Metro vMotion, I answered and then thought ‘hang on check yellow-bricks”, we then got a load more info.

    Also a message to Steve, See Olivers comment, he was spot on.

    Because of the work that Duncan does, Yellow Bricks is always my first port of call.

  11. Arjun says

    13 February, 2013 at 02:46

    I completely agree with Oliver on this one, Steve I think most of us are here because we have a certain respect and admiration to the level of work Duncan has been doing all these years .

    It’s easier to sit in a corner and type some slang’s and make yourself look cool ( sarcasm), However the right attitude was yet again shown by Duncan by guiding you into right direction , My intention is not to post something irrelevant in this thread , as this not You Tube / Face Book, It’s just the respect to the Man, which drives me to make sure people like Steve avoid posting on technical forums .

Primary Sidebar

About the author

Duncan Epping is a Chief Technologist in the Office of CTO of the Cloud Platform BU at VMware. He is a VCDX (# 007), the author of the "vSAN Deep Dive", the “vSphere Clustering Technical Deep Dive” series, and the host of the "Unexplored Territory" podcast.

Upcoming Events

Feb 9th – Irish VMUG
Feb 23rd – Swiss VMUG
March 7th – Dutch VMUG
May 24th – VMUG Poland
June 1st – VMUG Belgium

Recommended Reads

Sponsors

Want to support Yellow-Bricks? Buy an advert!

Advertisements

Copyright Yellow-Bricks.com © 2023 · Log in