There is something I always wanted to know and that is how VMotion (yes I am using the old school name on purpose) came to life. After some research on the internet and even on the internal websites I noticed that there are hardly any details to be found.
Now this might be because the story isn’t as exciting as we hope it will be or because no one took the time to document it. In my opinion however VMotion is still one of the key features VMware offers but even more important it is what revolutionized the IT world. I think it is a great part of VMware history and probably the turning point for the company. For me personally VMotion literally is what made me decide, years ago, to adopt virtualization and I am certain this goes for many others.
At VMworld I asked around who was mainly responsible for VMotion back in the days but no one really had a clear answer until I bumped into Kit Colbert. Kit, who was still an intern back then, worked closely with the person who originally developed VMotion. I decided to contact the engineer and asked him if he was willing to share the story as there are a million myths floating around.
Before I reveal the real story about how VMotion came to life I want to thank Mike Nelson for revolutionizing the world of IT and taking the time to share this with me and allowing me to share it with the rest of the world. Here is the true story of VMotion:
A bunch of us at VMware came from academia where process migration was popular but never worked in any mainstream OS because it had too many external dependencies to take care of. The VMware platform on the other hand provided the ability to encapsulate all of the state of a virtual machine. This was proven with checkpointing; where we were able to checkpoint a virtual machine, copy the state to another host, and then resume it. It was an obvious next step that if we could checkpoint to disk and resume on another machine that we should be able to checkpoint over the network to another machine and resume.
During the design phase for what would later become Virtual Center a couple of us discussed the notion of virtual machine migration. I took the lead and wrote up some design notes. I managed to extract myself from the mainline development of ESX 2.0 and I decided to go off and build a virtual machine migration prototype. I was able to build a prototype fairly quickly because we already had checkpointing support. However, of course there was a lot more work done by myself and others to turn the prototype into a high quality product.
I needed something to demo it so I used the pinball application on Windows. The only interactive app I had on my virtual machines was pinball. I had two machines side by side each with a display. I would start pinball on a virtual machine on one physical machine. Then I would start the migration and keep playing pinball. When the pre-copy of memory was done it would pause for a second and then resume on the other machine. I would then keep playing pinball on the other machine.
That’s the VMotion story. Basically VMware had built the underlying technology that made VMotion possible. All it required was someone to take the time to exploit this technology and build VMotion.
The funny thing is that although this might have been the obvious next step for VMware engineering it is something that “shocked” many of us. Most of us will still remember the first time they heard about VMotion or remember it being demoed, and as I stated it is the feature that convinced me to adopt virtualization at large scale, or better said it is responsible for me ending up here! In my case the demo was fairly “simple” as we VMotioned a Windows VM, however we had an RDP session open to the VM and of course we were convinced the session would be dropped. I think we did the actual VMotion more than 10 times as we couldn’t believe it actually worked.
Now I am not the only one who was flabbergasted by this great piece of technology of course hence the reason I reached out to a couple of the well known bloggers and asked if they could tell their VMotion story/confessions…
Cormac Hogan, comachogan.com
So for me, it happened back around 2004 when I was still at EMC. I was part of a team that provided customer support for Unix and Linux platform. I had seen ESX (might have been 2.0) when someone said that I needed to see this new vMotion feature. I didn’t really get it when he said that he had just moved all the VMs from host A to host B. But when he then flicked the power button on host A, and when I saw that all the VMs were still running, then it sunk in. That was then I knew I needed to work for this company!
Truly eye opening experience, which resulted in a career change. How cool is that 🙂
Chad Sakac, virtualgeek.typepad.com
For me, while I remember being amazed from a generic bland use case, the “this is going to change everything” moment occured for me in 2007.
If vmotion is about non-disruptive workload mobility (an amazing concept), where things get crazy cool for me are scenarioes and definitions of “workload” and “mobility” are stretched.
In early 2007, I was in the basement of my house playing with early prototypes of the Celerra VSA running on ESX whiteboxes. It was one of those now-common “russian doll” scenarios where the host powering the VSA was in turn being supported by an iSCSI LUN being presented by the VSA, which in turn supported other VMs. While intellectually obvious that VMotion **should** work, it was never the less amazing to see in it action, with no dropped connection under load.
At that moment, I realized that the workload could be as broad a definition as I wanted, including full blown stacks normally associated with “hardware” like arrays. It was also an “aha” that this could transform a million use cases not normally associated with a server workload.
Ironic side note – the next day, I was showing that concept in the boardroom during a discussion why all our stacks needed to be encapsulated and virtualized. Turns out they were already working on it 🙂
That all said – those “aha moments” happen constantly. Another example – this time more recently – was about stretching the definition of “mobility”. It was during the run-up to VMworld 2010, when we were doing the demo work for the VERY long distance vMotion scenarios with early prototypes of VPLEX Geo. As we dialed up the latency between the ESX hosts on the network and storage – I was very curious to where it would blow up. When it made it past 44ms RTT (for math/physics folks – that’s the latency equivalent of 13,000km at the speed of light!), it was a “wow” moment (BTW, it blew up at 80ms :-)) I need to point out here that it completely violates the VMware support position (and for many, many good reasons – one “it worked in this narrow case in the lab” does not equal “works in the real world”), so don’t try this at home.
BUT it highlighted how, over time, the idea of non-disruptive workload mobility over what TODAY are consider crazy distances, network, and storage configs will tomorrow be considered normal.
vMotion and svMotion never cease to amaze me.
Nothing less than expected of course, some crazy scenario and as Chad states it isn’t supported but it definitely shows the potential of the technology!
Frank Denneman, frankdenneman.nl
During our VCDX sessions in Copenhagen we spoke about things in your life you would always remember. My reply was ; Seeing Return of the Jedi in the cinema, the falling of the Berlin wall, 9/11, Pim Fortuyn murder and witnessing vMotion in action for the first time.
I clearly remember my colleague screaming through the wall that separated our office. “Frank do you really want to see something cool?” As an MS exchange admin/architect responsible for a global spanning exchange infrastructure nothing really could impress me those days but giving him the benefit of the doubt I walked over. Peter sitting there grinning like a madman, offered me a seat, because he thought it was better to sit down. He opened a dos prompt, triggered a continuous ping and showed the virtual infrastructure explaining the current location of the virtual machine. As he started to migrate the virtual machine he instructed me to keep tracking the continuous ping, after the one ping loss he explained the virtual machine was up and running on the other host and to prove me, he powered-down the ESX host. I just leaped out of my seat, said some words I cannot repeat online and was basically sold. I think we migrated the virtual machine all day long, inviting anyone who passed by our office to see the best show on earth. No explanation needed of course, but from that point I was hooked on virtualization and the rest is history.
I still enjoy explaining people the technology of vMotion and it still ranks in my book as one of the most-kick-ass technologies available today. As Mendel explained in the keynote of VMworld 2006 demonstrating recording an execution stream (now FT), we have the technology and the platform available to do anything we want, the problem is we still haven’t reached the boundaries of our creativity, I fully concur and I think we still haven’t reached the full potential of vMotion. Heck, I’m off to my lab just to vMotion a bunch of virtual machines.
Can I thank Peter for introducing Frank to the wonderful world of virtualization?
Mike Laverick, rtfm-ed.co.uk
My first VMotion was demo of media server being moved from one ESX hosts to another – with the buffering switched off. I forget now what movie clip was being shown to the desktops – I think it might have been a Men In Black trailer. Anyway, nothing flickered and nothing stopped – the video just kept on playing without a hick-up.
At that point my mind began to race. I was thinking initially about hardware maintenance. But quickly (this in in ESX2) days began to think of moving VMs around to improve performance, and possibility of moving VMs across large distances. At the time I told my Microsoft chums all about this, and they were very skeptical. Virtualization, they (mis)informed me, was going to be a flash in the pan, and that VMotion was some kind of toy – of course, in a Road to Damascus way now HyperV supports “Live Migrate” its an integral part of virtualization. In truth when I started to demo VMotion to my students occasionally I felt like I was show-boating. This was in the vCenter 1.x days. But in some respects there’s no harm in showboating. It allowed me to demonstrate to students how far ahead VMware was against the competition, and what a visionary are company VMware is. It certainly added to my credibility to have a technology that was so easy to setup (so long as you meet the basic pre-requisites) and the great thing about VMware and the courses is that you didn’t have to “hard sell” the product sold itself.
On a more humorous note I’ve seen all kind of wacky VMotion setups. I once had two PIII servers with a shared DEC JBOD with SCSI personality cables (circa NT4 Clustering configuration) just to get the shared storage running. I managed to get VMotion working with 1997 era equipment. I’ve also been asked by student – who had laser line-of-site connectivity between two buildings – if he could VMotion between them. I laughed and said as long as he could meet the pre-requisites there would be no reason why not. Although it would definitely be unsupported. Then I smiled and said, if he ever got VMotion working – I would come round in my Dr Evil outfit to explain – VMotion – with laser.
As I already stated, but re-enforced by Mike… VMotion changed the world, and the fact that both Microsoft and Citrix copied the feature definitely supports that claim… now I am wondering if the VMotion across “laser line-of-site connectivity” actually worked or not!
Scott Lowe, blog.scottlowe.org
I remember when I first started testing vMotion (then VMotion, of course). I was absolutely sure that it had to be a trick–surely you can’t move a running workload from 1 physical server to another! I performed my first vMotion with just a standard Windows 2000 server build. It worked as expected. So I tried a Citrix Metaframe server with users logged in. It worked, too. Then I tried a file server while copying files to and from the server. Again, it worked. SSH? Worked. Telnet? Worked. Media server with clients streaming content? Web server while users were accessing pages and downloading files? Active Directory? Solaris? Linux? Everything worked. At this point, after days–even weeks–of unsuccessfully trying to make it fail, I was sold. I was officially hooked on virtualization with VMware.
Thanks for the invitation to share memories about vMotion!
It appears that all top bloggers got hooked on virtualization when they witnessed a VMotion… As I stated at the beginning of this post; VMotion revolutionized the world of IT and I would like to thank VMware and especially Mike Nelson for this great gift! I also like to thank Scott, Mike, Frank and Chad for sharing their stories and I bet many of you are currently having flashbacks of when you first witnessed a VMotion.
A great historical read Duncan.
My story is similar to Mike Laverick’s, except we were showing a Terminator II trailer in our boardroom.
Christian Mohn says
A long time ago, in a virtual infrastructure long since upgraded I saw my first live VMotion. I still use a it as a live demo, while pinging the guest, whenever I try to explain and show how it works to someone in our company. VMotion has indeed revolutionized IT Operations for many of us, and if you think about it, it has really set the standard for how many other products work and will work in the future. When everything, from servers, apps and even server apps are virtualized, we’ll see, for instance, moving Oracle or Microsoft SQL server instances between servers in the same way. Why not? VMotion has proven, time and time again, that it works extremely well.
Julian Wood says
VMotion is one of the the great enablers for the cloud. Without it we couldn’t be talking about an adaptive infrastructure.
Can you even put a figure on what VMotion has saved in downtime for maintenance / upgrades / migrations never mind the performance possibilities of moving machines around.
Absolutely stoked by this wonderful feel-good story. Seeing such wonderful technique in action is almost epic. These techniques makes the IT enthusiast in us exist!
Thank you very much Duncan for a great read. This really made me smile, and brings back the memories and feelings I felt seeing my first VMotion.
David Davis says
Super-Cool post Duncan! This one needs to be entered into the history books of virtualization – some kind of virtualization almanac.
Jason Boche says
9/11/01, as well as the evening before, are memorable to me but I don’t have a VMotion story. I honestly don’t remember the first time I heard of or saw VMotion. My guess is spring or summer of 2005 because by September of that year I was a VCP on vCenter 1.0/ESX 2.0 which had VMotion but no DRS yet.
My thrill and amazement with VMware virtualization technology started much earlier, probably around the 2001 timeframe with VMware Workstation. It has been a fun ride.
Rick Blythe says
Any sufficiently advanced technology is indistinguishable from magic. –Arthur C. Clarke
Great stories to read how all these by now vmware gurus got addicted to a thing called virtualization
Stuart Miniman says
Wow, thinking back, I was the Program Manager in E-Lab for VMware when we first brought it in for testing with ESX 1.5. I remember how excited all of the engineers were when VMotion first was tested – it worked just like magic. We invited VMware to come to the big user conference (it must have been 2003 EMC Technology Summit – now called EMC World). I moderated a session where Ed Bugnion explained (in language that even I could understand) to everyone how VMotion worked. We had such large turnout that we kicked all of the EMC people out of the room so that customers could hear the session.
Jane Rimmer says
I was proud to be part of the team that was responsible for marketing VMotion in EMEA. Great read Duncan thanks!!
John Gannon says
Duncan – thanks for bringing back the memories. I guess I’ll share mine!
I was working for VMware PSO-Consulting when I first saw VMotion in action at a technical sales offsite (I think it was 2003 or 2004, when all the SE’s and PSO people at VMware could fit into one meeting room at Dinah’s!)
It hadn’t been released yet, so product management was showing us the beta version.
I believe the demo in our case was a streaming video, and just like your pinball example, it went off without a hitch.
And, not surprisingly, everyone in the room went wild. 🙂
Massimo Re Ferre' says
I don’t remember either when I see it first time. However I remember when I talked about it to a big bank. There was a veritas pre-sales that started laughing when I explained it. He is no longer laughing.
Oh… And IBM even had to introduce it about 5 years later on the unix platform since their (our at that time) customers kept asking “do you have vmotion?”. Not that this helped a lot IBM with a #deadplatformwalking … But yet fun to share.
Kristopher Lalletti says
I remember pitching the concept of vmotion at my peers many-oh-many years back; explaining that you didn’t really have to worry about where were your virtual machines, except they were operating somewhere within the cluster, that was ‘good enough’.
Obviously, they were heavily skeptical; one even insisted that we we maintain specific physical segregations for certain workloads.
So I politely nodded, smiled, agreed on their irrational perspective, and went-on with the initial plan of dynamically managing the workload with resource classes and setting DRS to ‘fully automatic’ with some affinity rules with much of a smile 🙂
VMotion is the only way to maintain 24/7 while resolving physical equipment/failures/problems daytime. It makes life so much more simpler.
Kelly Olivier says
I remember when I first saw VMotion, and I remember how I felt. For me the more memorable time was initiating a svMotion from VIMA of the VIMA VM I was commanding from. When that worked, I knew there were great things to come.
Bill Petro says
It was half a decade or so ago when I first saw VMotion. I was at EMC at the time and was already familiar with VMware. It was the explanation that you could “move a running Virtual Machine from one physical server to another, without disruption — using shared storage across the network” that caught my attention. Even before I saw the demo (a running video that did not stop in migration) I recognized the potential: disaster recovery, load balancing, hardware maintenance.
Indeed, VMware went on to develop all kinds of amazing capabilities, using VMotion as the underlying technology. I would explain to customers and train sales reps that this was the “secret sauce” that would enable a huge sea change in the industry. Years later, cloud computing, virtualized data centers, and private clouds “bursting” into public clouds are the result.
Tony Dunn says
One of top 2 game changers in my 10 years at VMware. The other…a new event called VMworld 🙂 Thanks for the great blog post Duncan!
Matt Liebowitz says
I remember pretty clearly the first time I saw it. I had been using ESX 1.5 and 2.0 and then in either late 2002 or early 2003 I went to a small event VMware was having in NYC to promote this new thing called VirtualCenter and vMotion. They showed a demo of a server running streaming video getting vMotioned between hosts and I remember the room practically gasped when they saw it. After seeing the demo, the question flying around the room was “When is VMware going public and how can I get in on the IPO?”
I was similarly wowed when I first saw DRS and HA work as well..
It was, and still is, an amazing technology that still makes people say wow. Good write up Duncan, it was fun to stroll down memory lane.
Juan Manuel Rey (@jreypo) says
My first VMotion experience was may be four years ago, it was not my first experience with VMware but at the time and since I was a HP-UX/Unix sysadmin, have to admit that I was very skeptical about it. Virtualization was very nice but a Linux virtual machine moving between the ESX servers without service lost? I was pretty sure that it was a fraud.
I never have been so pleased to misjudge something 😀
A fantastic article Duncan, as always 🙂
Bill Griffith says
Thanks for the perspective, Duncan. This, like many technologies from VMware, is truly amazing.
Rod Stokes says
As a VMware SE back in 2003, I remember a couple of us talking to the engineering team about the need for a management product and explaining that going to each ESX Server’s web interface didn’t cut it in an Enterprise environment. Up until then no one had really expected a customer to buy more than a handful of ESX servers.
I did the first vMotion demonstration in Europe to a customer. After applying the patches to ESX (a very scary thing in those days) I ran through the demo and it worked perfectly. This was just as well as there were senior execs from both the customer and VMware present!
Afterwards when going through it all with the technical team, I noticed that despite the ESX server saying it was connected at 1Gb, the switch said 100Mb. The switch was right (anyone remember the nightmare of auto-negotiation & mis-reporting?) and I had run the demo on a horribly unsupported 100Mb network. With some testing, we found it would work at 100Mb provided the VM was idle. Put any load on the VM and 1Gb was essential. At the time Gb networks were new to most customers and expensive and there was a lot of push back on this requirement in the coming months.
Ahhh when I were a lad 🙂
Travis Wood says
I can remember when I first heard about vMotion, in a presentation done by VMware for my company. Thats what got my excited about virtualisation and I put my hand up for the first VMware project my company got. It always amuses me when people see it for the first time and get so excited or nervous. They expect some lightning bolt to strike and fireworks to go off when it completes… but its just a progress bar.
Thanks Duncan for another great article.
I remember my first full install in ESX 2.5 with Virtual Center 1! I spent they day migrating my Exchange 2003, yes Exchange, around the place and blowing the minds of anyone I could! I was able to repeat it with the SQL Server as well.
I had AD boxes alongside print, web, and every major system we ran. all before the release of ESX 3!
This is the tech that got me excited and still today I feel is the key to VMware’s success.
For me, it wasn’t the first one we accomplished that made me realize exactly how game changing this technology was, it was in 2005 at a technology expo when I stood in front of the CIOs of three of the largest organizations within the federal dept. I work for and told them I would do hardware maintenance whenever I pleased; thank you very much. That arrogance came of course with the follow up statement they could walk me out if I ever hurt production operations. We did our demo (Harry Potter movie trailer because it’s magic after all) and asked if they had questions. Never looked back from that point.
Not only am I still there, but instead of using VMotion to show the power of virtualized operations, we simply put a whole host into maintenance mode and “let the computer take care of it.”
First time I saw vmotion in practice was 2006, while in training in Netherlands. The trainer showed us esx 2.x I think although it was not related to our training he was so excited to demo it for us and me as an IT rookie was in awe. That is when I also learned that version 3 would have what is now called DRS. Great piece of tech.
Bouke Groenescheij says
Hahaha, awesome post! You know – when teaching courses, some people still haven’t witnessed VMotion… Can you imagine that?!? You should see the look on their faces – just priceless! Anyway, although working with it for several of years – it is still a fascinating piece of excellent technology.
I first saw VMotion when VI3 was released mid 2006, it along with DRS blew me away!
Marco Broeken says
I know that when i witnessed vMotion for the first time, i was so enthusiast, I knew for sure this was going to change things… and it Did!. It was also the moment that changed my career from working at one company as a system engineer into going to be a Virtualization Consultant.
Around me no one really jumps up and down anymore when you live motion a machine from one physical server to the other, but the technology is priceless, and as you said this is what build VMware, I do agree with you, this is when i started!
I am doing my best to remember when I saw it the first time. I think I said bad words in disbelief, but I won’t admit it :). Now we have fully automated DRS moving stuff around. Add in DPM for a sprinkle of fun and you have established IT guys fainting.
It is a shame when things like that change the game so many people dig in their heels and try and prevent the sea of change. It happens every time there is a revolution. Out with the old, in with the new; an unstoppable force. Hopefully I’m never the establishment that isn’t willing to ride the wave.
Francis Béland says
Great post. On my side, it’s not really Vmotion that threw me in virtualization. I was using VMware Workstation 5 to build some test environments on my pc. It’s more the fact that a complete machine can be encapsulated in files that impressed me the most. I was back then a junior sys admin and I told to the senior sys admin how that would be wonderful to have that technology for servers since it would facilitate DR, backup, and that will decrease the cost of hardware servers. The senior guy told me that a product already exists for servers and I strated to read about ESX. I then learned about VMotion and I was amazed to see the theorical potential of that technology.
After my discovery, I built a business case to implement a VMware infrastructure in the company. Unfortunately, I left 3 weeks before they gave the approbation to do the project but I’ve found my career path back then: Getting specialized in VMware.
Serge Meeuwsen says
Super post!! You’re absolutely right about how this has changed IT. Great to read everyone’s personal stories too, which really emphasizes the point.
Like most of you, when it was demo’ed to me by VMware at Dell’s offices in Belgium (at the time by Jeremy van Doorn who was an SE back then) it blew me away too, realizing the enormous potential of it.
Since then I had similar experiences (albeit with slightly lesser impact) with other VMware technologies such as svMotion and Fault Tolerance.
Dave Convery says
BAH! It’s all smoke and mirrors! I see it countless times every day and I still don’t believe it!
Just kidding. I don’t remember my first exposure to VMotion either. I VIVIDLY remember 9/11, then the call from my boss that I was working that night. It was my day off. I worked at SunGard doing DR work. Needless to say, I was busy for a few weeks.
I can say that, if it wasn’t for VMotion/vMotion, I would not be doing what I have been doing since 2005. With the (US) economy in the crapper, I still get calls from recruiters looking for VMware experts. It has been a nice, recession-proof gig!
Wade Holmes says
I remember the first time I played with VMotion. I was an IT Specialist at IBM, and would steal time in one of the IBM BCDR datacenters in Sterling Forest, NY after hours to play with VMware stuff. I was like a kid in a candy store. At the time VMotion required FC, so it was a major undertaking and great learning experience to set up all the infrastructure myself, in IBM’s massive DR megacenters.
I knew from day 1 that VMware was a game changer, and from that day forward I was an instant VMware evangelist. Everyone in IBM would get tired of me ranting and raving about VMware, and how we have to utilize the technology in our BCDR offerings. From the day I first touched VMware and witnessed VMotion myself, my career has done nothing but skyrocketed. It is great to be passionate about what you do, so for that I thank VMware and all the great technology like VMotion!
Chadwick King says
@mike laverick – Halirious Dr.Evil plug “Sharks with lasers on their heads!”
I remember when I first encountered virtualization. I was pretty young in experience and didn’t really know much. I especially didn’t know of VMware and this was around December of 2007 (I know, I know is that really possible?). I actually lost a medical client to a vmware consultant, the irony of it was is that it actually motivated me to learn about the product and since then still remain fascinated.
My first vMotion experience happened in about January of 2007. I was watching many you tube videos about it because I couldn’t even believe it. In fact, I even recall me calling the guy that picked up the medical client and asking him in unbelief. Shortly after I got a job at a library and began testing vmware there and I remember when I first did a vMotion I felt like a young boy with new toys and did it over and over. It was on some old Dell biscuit servers but man it was cool. I was showing all the folks at the library and they was as just as much in unbelief as I was. I still think it is an amazing product and each new year VMware matures I get excited and fascinated because of all the amazing stuff VMware can do. Keep it going!
This is a great post Duncan nothing like reminiscing.
Erick Moore says
In late 2006 I had just deployed my first production ESX cluster. We had P2Vd our server monitoring app over as our first production virtual machine. About a week later the network guys informed us they were assigning new IPs to the management VLAN. Of course that would normally mean we would have to power off the system to reconfigure the DRAC, but we decided to try VMotion instead. Wow! We couldn’t believe it when our monitoring system stayed online once we kicked off the move. It just worked, and THAT just happened. We didn’t even get any false positives from the app. Crazy! The network guys were blown away too! I knew I never wanted to deal with a physical system EVER again after that.
Chris Monfet says
I remember seeing VMotion for the first time when I first used vCenter. I was thinking why do we need vCenter, the MUI works great. Then we VMotioned a live VM (with my ping -t running) and thought, “oh….that’s why we will need vCenter.” I did it a few more times just to make sure it worked. I did a demo for some of the other tech guys and it was like I was having a magic show at my cubicle. I was hooked.
Enad Kauffman says
My first exposure to vMotion, DRS, & HA was by my VMware SE at the time, as he was addicted and trying to get me hooked on the same drugs he was. I can’t honestly say that I grasped ALL the possibilities back then, but I sure saw enough to know that it was a game changer. The biggest benefit I saw initially was the pain of maintaining the ESX hosts was going to be greatly reduced by vMotion, and HA was going to make my failed-hardware nightmares go away. Imagine being able to patch a host in the middle of the day without anyone knowing! Imagine a VM restarting itself elsewhere after a hardware failure! Eureka!
We quickly had blades up and running in test. That lasted about two weeks, because wouldn’t you know it, we had a major production performance problem with our inbound spam filters, but no hardware available to solve the problem.
Well, I said we could always stand up virtual spam filters on the test bed as a band-aid. By the end of the day (thanks to templates), the performance problem was solved.
Well, almost everyone. A couple of months later, when I had to report progress of our VMware project to management and mentioned vMotion and DRS, my director said that we would absolutely not allow VMs to move willy-nilly from one host to another (he has an AIX heritage). He saw it as too big of a risk (this coming from the same guy that OK’d putting the spam filters on our test boxes).
A month later, I presented again. I showed them vMotion, and then I showed them that since last we’d spoken, there’d been well over 1000 DRS migrations of VMs on the cluster (I turned up the setting to 5 on purpose), and not once had it caused a production impact.
Now? Well, when you love something that much, you become obsessed with it. So now, I work for VMware. Guess you could say I’m hooked.
“At VMworld I asked around who was mainly responsible for VMotion back in the days but no one really had a clear answer …”
“Fast Transparent Migration for Virtual Machines”
Michael Nelson, Beng-Hong Lim, and Greg Hutchins, VMware, Inc.
Ed Bugnion says
Here’s my VMotion story… Not long after Mike’s first internal “pinball” demo to the team, I was scheduled to pay a visit to one of the world’s largest server vendors. At the time, VMware already had a number of OEM deals with large server vendors, and was very popular on expensive, scale-up systems with lots of sockets. But that particular vendor did not have that kind of gear, so we were hitting a wall with them on ESX.
So, in that last ditch attempt, we fly down there and present VMotion in powerpoint. They immediately get it and more importantly get the implications for their business. They ask for an install in their lab. So a few weeks later, Lance and Dino fly down there with some pre-alpha version of the code, lock themselves in their lab for a week or two, and get the “Men in black” demo to work reliably. We have an OEM deal.
And since then, virtualization is done on scale-out clusters.
Andrew Mitchell says
Reading Chad’s ‘Russian doll’ description reminds me of some fun I tried to have with vMotion a few years ago.
Step 1 – Install ESX 3.x on a physical server
Step 2 – Install another ESX instance as a VM running within the original ESX server
Step 3 – Attach both ESX instances to vCenter
Step 4 – Attempt (unsuccesfully) to vMotion the virtual ESX instance onto itself.
My brilliant theory was that if I was successful, as the virtual instance was no longer running on the physical server, I could power off the physical box and the VM would keep running. The ultimate version of DPM 🙂
It probably sounds crazy now, but at the time (and after quite a few drinks) it sounded like something fun to try…….
Steve Chambers says
My favourite customer quote about vMotion happened around 2008, after Vmotion was pretty well established: “I don’t want any of that Star Trek crap in my data center”. after a demo and explanation why it was (a) simple to do, and (b) helped his guys with maintenance and (c) look awesome to the app guys by eliminating a shed load of downtime, I offered to buy him some spock ears…
Andrew Lambeth says
Though it slightly predates vMotion I had a very similar “aha moment” thanks to Mike. While the vmm team were still working on getting SMP support fully working during development of (ESX 2 I think, maybe 1.5?), there were a few hoops you had to jump through to get the SMP guest up, one of which was something like switching the UP HAL to the SMP HAL after the guest was installed. Of course there is no valid reason to do this on real hardware, and it would very likely result in an unbootable system. The guest would pop up a message to that effect begging the user not to do this. Mike was helping me with something one day and we were going through this HAL swapping step and the guest popped up its scary dialog and he said “I’m not afraid of you.” and clicked it. It was just a light hearted comment, and I’d clicked through that same dialog plenty of times myself, but it was at that moment that I realized how revolutionary all this would become. Something that would be unthinkable like rendering your system unbootable, so you could spend the rest of the day reinstalling the OS, was now trivial and silly. The big bad OS had been reduced just another silly process running the machine. Thanks for the post, those were fun and exciting times.
Tom Kraus says
My aha moment was around 2003 or 2004 and I Was involved with a huge (30k+ mailboxes)deployment of Exchange 2003 (which had just been released). We were in the middle of defining the requirements and options for High Availability for Exchange. In the midst of that discussion the customer was looking at VMWare and I sat in a room of Exchange architects and support personnel while ESX and vMotion was demonstrated of a Mailbox server with the Outlook client connected. When the Mailbox server was migrated to another ESX host and there was only a split second non-disruptive impact the the Outlook client the room was silent and people were just looking at the Screen dumbfounded. One of the Exchange administrators finally spoke up and said “What was that ?”.
After reviewing all of the complicated and convoluted ways to provide HA through MSCS and other techniques to Exchange they could not believe how easy this Live Migration was. . .
Pang Chen says
I’m glad to see Mike get recognition after all these years. I first met Mike back in 2005 when we were the first tenants in our San Francisco office, and he was introduced as the “Father of VMotion” as humble as he appears. “VMotion” (used as a verb) now has a generic status much like Kleenex is to tissue. You wonder how we ever got by without it.
CHICKS DIG IT !!
Great post Duncan 🙂
I remember we still hadn’t license for VMotion so I wrote what I called “the poor man VMotion” script.
Basically after some verification the script
– suspend the VM
– unregister it
– register on the new host
– resume it
it was slow compared to the real VMotion and we never used for production VM
nevertheless it worked 🙂
Hello: Thanks for the history on this… i hope there is a wall of honor at vmware that has the names of the folks that gave us VMotion… As for my VMotion moment, truly, i almost fell off my chair. upon realizing the game changer it was (i.e. no more working on saturday nights, doing maintenance during business hours thanks to HA/DRS), i went from being a microsoft fan-boy to a VMWare devotee and have not looked back since then
RAMESH GEDDAM says
I heard about VMOTION in 2007, when working in VMware Inc. All I know and taught was, a running virtual machine gets migrated to another physical machine without any downtime.
It took, 5 odd months to learn in detail about it. It was really a big challenge for many of administrators to take a downtime for server to perform a minor OS/Hardware maintenance. Pre-copying of memory, quiescing file system etc is an awesome functionality of VMotion. It is a matured solution with scalable features like EVC as compared with other vendors like “MS Quick Migration which completely relies on clustering functionality”
VMware rules in Migration arena!
Denis Guyadeen says
The first time I heard about VMotion was at the EMC Sales Kickoff in 2004 (EMC acquired Legato, Documentum and VMware (in that specific order IIRC)). During the kickoff I think Tucci (or Donatelli) announced the CEO’s (Dave DeWalt, David Wright and Diane Greene) of each company on stage to present their technology portfolio. Diane Greene gave everyone a 101 on Virtualization and then presented a preview on Virtual Center and a technology that got a lot of us geeks blown away… VMotion… (Virtual Center and VMotion was not released yet). After that moment I spent as much time as I could playing around with it in the lab.
However my history with VMware goes back to 2000 when I worked in a law firm in the World Trade Center.. I was in charge of a Windows AD and Exchange lab (amongst other duties), but was also playing around with RHEL (it was supposed to be the next big thing). I found out about VMware and told my manager that it would be great for the lab environment (which it was). He looked at it but didn’t think it was anything special at the time (boy you should see the look on his face now when I remind him of it), as he was a MSFT fanboy.
My first was a few years back, and I just bounced a couple of virts between my small 4 host cluster for a day. Then came storage vmotion, that blew my mind.
The real proof, for me however, happened earlier this year: vmotioning a hospital’s live, production him servers, with a single ping drop, no client disconnects. Wow.
It’s amazing that, now, it’s just a part of the job.
I have been using vMotion for a while now and as amazing as it is – there are some dark moments I had experienced with it. I have probably performed thousands migrations between our hosts within last few month only, and once in awhile it screws things up. For instance, a domain controller went totally beserk once, spitting hundreds of errors into event logs right after it had been vMotioned. The other time it was another database server that stopped responding and had to be reindexed. I wonder if anyone else had similar kind of experience with vMotion…
I have just started learning vMotion feature.
I have some basic questions
1. Say, I have enabled vMotion on Management network and on dedicated other network (dedicated) as well. Which NIC will be used for vMotion if MultiNIC is not configured?
2. Every VM use to have one unique ID to identify VM across our virtual environment, when/how does vMotion keeps the track when VM starts it execution on destination host?
3. When VM starts processing all its application on destination host during vMotion?
I mean, which phase?. Phase 2: Precopy Phase or Phase 3: Switchover Phase?
please clarify . I read in your book that (quick resume section: It starts when vMotion creates shadow VM. ) I may come back to you on the same.