I’ve seen this floating around the communities a couple of times and someone also mentioned this during a VCDX Panel: setting limits on VMs when you are not allowed to decrease the memory. For example you want to P2V a server with 8GB of memory and an average utilization of 15%. According to normal guidelines it would make sense to resize the VM to 2GB, however due to political reasons (I paid for 8GB and I demand…) this is not an option. This is when people start looking into using limits. However I don’t recommend this approach and there’s a good reason for it.
Using limits can lead to serious performance issues when the VM starts swapping. As many of you know the first thing that happens when you reach the limit is that the balloon driver kicks in. The balloon driver will force the OS to swap out. Of course this will affect performance but at least when the OS gets to pick the pages it will do this in a smart way. When the OS reaches its limits the VMkernel will start swapping and this is where it gets nasty because the VMkernel does not take anything into account. It could easily swap out pages actively being used by your application or operating system which will affect the performance of your VM heavily. (That’s a short summary of the process, if you want a more in-depth explanation of this please read this excellent post by Scott “VMGuru” Herold.)
Swapping, either VMkernel or OS, is the reason I don’t recommend using limits. Just think about it for a minute. You probably convinced the application owner to virtualize their services with arguments like availability, flexibility and equal performance. Setting a limit will more than likely affect performance when the threshold is in reach and thus hurt their trust in virtualization and the IT organization. Another side effect is that there’s no way to recover from swapping without a reboot, which will mean availability will also be decreased. In other words; avoid setting limits.
I do however understand why admins take these drastic steps; but again I don’t agree. If you want to convince your application owner that their VM needs to be resized monitor it. Prove to them that the server is not utilizing the memory and claim it back. Claiming back is difficult, that’s why I personally recommend to invest more time and effort during the first phase of your P2V project, educate the application owner and convince them with the outcome of your capacity planning tools. Explain them how easy it is to increase memory and make them feel more comfortable by adding a week of aftercare which includes resource monitoring. If you really want to convince them, but that’s dependent on the level of maturity within the organization, change the cost-model and make it more attractive to downsize…
Matt says
I agree that limits can be dangerous but I think they have their place. In situations where the server is at a lower tier level than normal production servers, imposing limits can be a helpful way to stack more servers per host. As long as the end user understands that performance may not be optimal, that is. Generally speaking this is good for dev/test environments where there isn’t a large budget for hardware/software and you want to utilize as many VMs per host as possible.
If I have to do this, I use Resource Pools with limits imposed rather than imposing the limits directly at the VM level. This way if one of those VMs needs the memory while the others do not it will have a better chance of accessing it.
Good article as always.
nate says
Does the balloon driver kick in when the memory limit is reached on a particular VM? I was under the impression at least that it only kicked in when there was memory pressure on the host itself. Also of course swapping has other downsides, if VMs are aggressively swapping they can degrade your storage performance by quite a bit impacting everyone else at the same time regardless of whether or not they are swapping too.
Bouke Groenescheij says
I totally agree! When I was reading the title of this post I was thinking: ‘Oh no, do not use limits, please scale down!’, but after reading the full post I was totally relieved. Great article!
Paul Nothard says
Limits can be a Machiavellian way for admins to prove a point where they have been unable to prove their point by giving the insistent developer team less memory. Instead of monitoring the obviously reduced memory over a period of time, they monitor the memory, which has been transparently limited, and present their findings to the developers at the end of the process. If the admin is correct, the limit will have had no impact on matters. It’s more politics than technology that drives this approach though.
Strongly agree that limits should be an exception rather than the norm. As you say, the OS paging algorithm is cognisant of what it’s pushing to disk, whereas ESX is not as aware.
Aubrey Williams says
Limits are dangerous when one lacks the understanding of how and when to apply them. If you have a VM that, by analysis of historical data of a month or more, never uses more than 2GB, it’s wise to set a memory limit to 3GB. Limits, used wisely makes more resources available for not only future VM deployments, but for current VMs with variable performance characteristics.
Keep in mind that components like DRS and admission control are impacted by limites as well. When used wisely, this impact is positive.
Duncan Epping says
Euuuh Admission Control is not affected by limits, it’s affected by reservations.
Nil Paul says
What would happen if you have set the limit same as the memory being allocated to the VM, eg. 4gb VM has a limit of also 4GB? Would that affect performance. Going via what Scott at VMguru says, I would think not?
Duncan says
It would not affect performance until for one reason or the other the memory gets increased and the limit stays at 4GB. That’s why I also never set limits to the same amount of memory as provisioned. There’s no point to do this.