Gabe asked the question on one of my recent posts if it made sense to enable Hot-Add by default and if there was an impact/overhead?
Lets answer the impact/overhead portion first, yes there is an overhead. It is in the range of percents. You might ask yourself where this overhead is coming from and if that is vSphere overhead or… When CPU and Memory Hot-add is enabled the Guest OS, especially Windows, will accommodate for all possible memory and CPU changes. For CPU is will take the max amount of vCPUs into account, so with vSphere 5 that would be 32. For memory it will take 16 x power-on memory in to account, as that is the max you can provision . Does it have an impact? Again, a matter of percents. It could also lead to problems however when you don’t have sufficient memory provisioned as described in this KB by Microsoft: http://support.microsoft.com/kb/913568.
Another impact, mentioned by Valentin (VMware), is the fact that on ESXi 5.0 vNUMA would not be used if you had the HotAdd feature enabled for that VM.
What is our recommendation? Enable it only when you need it. Yes they impact might be small, but if you don’t need it why would you incur it?!
Bilal Hashmi says
I was thinking of a fake hot add way without enabling hot add.. of course this is not the best way to do it.. but just thinking out loud.. I haven’t tested any of this
So if you have a VM with FT enabled, one could power off the primary which would bring the secondary in service (this would also cause the secondary to spin up a new secondary because FT is enabled).. perhaps the trickiest part would be to bring the original primary back in service ones more mem is added etc without causing an outage…
I guess this is really not a good idea as I started seeing issues as soon as I started typing it.. but I figured I will throw the idea out there to see if anyone ever tried doing anything like this.. I know FT is not meant for these use cases which is why hot add exists… I know FT VMs didnt support hot (correct me if I am wrong)
I would also like to add that hot add at least in vSphere 4 from what I recall had licensing requirements and also depended on the type of guest involved. For instance I dont think win 2003 standard was supported.. Also, one must power off the VM to enable hot add.. you might as well add the resources .. unless your goal is to try and see what resources will be ideal … thoughts?
Gabrie van Zanten says
Thanks ! 🙂
Doug Davis says
Seeing as you can’t enable/disable HotAdd on the fly, doesn’t it defeat the point if you don’t have it enabled by default??? I personally always configure our templates to have whatever hotplug capability they can enabled, otherwise we end up with an outage for our customers when they ask us for a quick increase in resources.
Gabrie van Zanten says
@Doug: That is one of the reasons I asked it. When it is best to leave it disabled by default, that means that if you want to use this feature you should plan ahead. And if you’re able to plan ahead, why not add sufficient RAM and CPU to start with? I think there are only few cases in which you would really want the flexibility to hot-add.
On the other hand, as Duncan explained, I think this is an example where the vSphere features are way ahead of what the Guest OS is capable of.
Bilal Hashmi says
I think one of the reasons it may exists as well is to try and see the impact of resources on the VMs performance.. For instance if we know the VM needs more resources but not sure how much, perhaps enabling it might be a good idea to try adding more mem/cpu until we reach our desired performance.. and ones we reach that point perhaps disable it..
Or else the other option is, power off.. add.. test.. power off add.. test .. I think thats where one would benfit from hot add.. Why we wouldn’t know what resources we need? Good question 😀
Doug Davis says
Maybe my experience is unique, but our customers/end users never know what resources they really want upfront and tend not to take expert advice either, so having the ability for them to start small & easily increase resources allocated without impact is pretty much invaluable.
Bilal Hashmi says
I think when increasing resources mean a bigger invoice, then yes I will have to agree with you. I have seen that myself where customers would try and safe the moolah where they can and not go with whats recommended.
Besides, the overhead is minimal as Duncan pointed out and if that what keeps your customers happy then good as long as they understand that ability puts an overhead (very minimal though).. In any case still cheaper, robust and flexible than any other option out there IMO.
Starting small is not a bad idea… thats what I used to calm people down when they started noticing their underused webserver only had 1 CPU as opposed to 8 ones it got P2Ved. 🙂
James Hess says
“Enable it only when you need it. Yes they impact might be small, but if you don’t need it why would you incur it?”
Because you don’t know when or if you will need it or not. The purpose of Hot Add, as I see it, is to allow you to add resources on the fly to a running critical VM to help with a sudden unexpected capacity demand you could not have scheduled downtime for in advance, because you did not anticipate.
If you had known you were going to need more resources, then you would have just added the right amount of resources.
I would take the opposite view: Enable all hot add options by default for any new VMs that will be mission critical, that is, VMs you can’t simply shutdown at will.
Then (possibly later), when optimizing your environment, consider disabling Hot-Add based on the need for performance enhancement.
Evaluate how certain you can be that you won’t need to Hot-Add.
If you can be reasonably certain, then sacrifice hot add for the performance benefit.
Now after sacrificing hot add, fewer resources may be required, so you have an even larger buffer than before between amount of required resources and amount of available resources.
Duncan says
I think you misunderstood my recommendation. If a workload is critical and the OS supports hot-add and your customer expect this functionality, clearly it is a requirement and you should enable it. But if it is a lower-tier app, with no direct requirement and one which can easily incur a small amount of scheduled downtime, there’s no point really.
Dave says
Timely post, I was discussing this with a colleague last week. He was surprised when I told him we couldn’t hot-add CPU/memory to a VM by default.
After our conversation I considered updating our templates but decided against it due to similar concerns Duncan points out – inefficiency as well as stability concerns.
Mission critical VMs (IMO) shouldn’t be non-redundant/require downtime to perform maintenance. If they do, this is a nice feature to enable but I agree with Duncan’s recommendation about leaving it disabled unless it clearly adds value.
Dave
James Hess says
“After our conversation I considered updating our templates but decided against it due to similar concerns Duncan points out – inefficiency as well as stability concerns. ”
What inefficiency and stability concerns?
Can you quantify this? Did you have concrete data to declare Hot-Add inefficient or not stable?
If a VM is so important that you would have to schedule your downtime, then it seems like Hot-Add could be beneficial, to mitigate a load spike.
Otherwise, you have a tradeoff to make during a resource emergency — either [1] shutdown a VM with unscheduled downtime that is not supposed to have unscheduled downtime.
[2] Massively overallocate resources, so you never have much a chance of running into this — but then your resource bloat overhead can far exceed the Hot-Add overhead.
Overprovisioning and Hot-Add are both options that have overhead. Overprovisioning results in quantifiable reduction in efficiency of your hardware.
Hot-Add results in some miniscule reduction in efficiency that has not been adequately quantified.
If the assumption that Hot-Add’s overhead is negligible holds true, then why give a VM 30% more RAM than it will ever need (therefore introducing a multiple percentage point performance inefficiency in regards to memory consumption), versus enabling Hot-Add, so you can manually give the VM more, without incurring unscheduled downtime, should service degradation due to an unexpected RAM demand occur?
Dave says
@James Hess, the MS KB Duncan posted describes performance and stability issues with 2003. Not sure about 2008.
“When the Hot Add Memory feature is enabled, the operating system pre-allocates kernel resources to handle any future memory that may be added to the computer. Kernel resources are allocated based on the capabilities of the computer instead of on the RAM that is actually installed. The kernel may allocate significant resources to RAM that may never be installed.”
http://support.microsoft.com/kb/913568
Going back to my template comment, I have the ability to work on 80% of my systems “live” (through load shifting, clustering, VMotion, etc) and fortunately have a maintenance window every month so for my situation I think it makes sense to leave hot add off and perhaps enable it on the 20% remaining (if at all). Also, in my situation we would typically scale out (not up) to add resources (we’re a web service hosting business) so we would add web front-ends the majority of the time to accommodate load spikes which can be done live and with very little effort.
I couldn’t agree more about right-sizing your VMs. I find it easier to do in a hosting/technology environment rather than the corporate environment I was previously at. The business would rather throw money at hardware/ESX hosts than spend the time monitoring/tuning VMs. Just about everything was 2vCPU and 4GB of RAM – the horror! 🙂
Anyways, I’d encourage everybody to do their own testing and research to determine if hot add is beneficial for their environment.
Dave
andy says
Duncan
what happens to VM swap, if i have a windows 2008 with 4 GB of ram and turn on hot add, if i add more RAM does the swap file increase automatically or does it re-size at next power on – ta
Gabrie van Zanten says
Good question Andy! I was just playing in a test lab so I had some time to test this. As you add memory to the VM, the VM swap is immediately enlarged.
andy says
cheers
Jim Spring says
So here’s the thing to remember in your recommendations, vNuma only kicks in at 8 vCPU unless you add numa.vcpu.maxPerVirtualNode into the vmx. Therefore if someone doesn’t make config changes to their VMs it is still advantageous to them to enable hotadd on their smaller VMs to avoid downtime if they need be resized. While the KB says it’s bad on Windows, just remember that’s Windows 2003 x86. vNuma requires some underlying knowledge of the HW arch and really is there to help monster VMs where squeezing blood from a stone may be necessary to get highest TPS. If it were my recommendation to make, I would say help your customers out and enable hotadd until they get to the bigger VMs then dig in and start applying the hardcore performance tuning methods like optimizing vNuma to match the physical numa, storage processor paths, etc.
Jay Weinshenker says
I realize I’m way late to comment here, but it isn’t HotAdd (RAM) that disables vNUMA, it is HotPlug (CPU) – see KB 2040375
stoked says
We had this enabled by default and it bit us. We use Windows 2003 32bit with greater than 8 – 12GB of RAM. Using RAM greater than 4GB reduces the amount of Free PTE’s and Memory Hot-Add reduces it even more. We were running into all sorts of instability issues and tracked it down to the Memory Hot-Add option being enabled. With it enabled, our hosts were around 12k Free PTE’s right after a reboot, after disabling Hot-Add they were at 180k Free PTE’s.
Preston Gallwas says
So, what’s the word with Server 2008 R2 / Server 2012? A lot of the caution found online relates to 2003.
Steve Barry says
Is it time for an update to this discussion now that Windows Server 2012 and R2 have been out for a while? I personally haven’t enabled hot-add on my Server 2012/R2 templates yet, but I’m beginning to think it’s time to make a change.
Ben says
I would like to know as well. I am looking at updating my 2008R2 template to enable hot add CPU and RAM but a colleague is cautioning me to research this first. All I’ve seen is relating to 2003, so I’m curious about 2008, 2008R2, 20012 and 2012R2.