I had a discussion with someone around limiting a VM to a specific amount of Mhz’s after I found out that limits where set on most VMs. This environment was a “cloud” environment and the limit was set to create an extra level of fairness.
My question of course was doesn’t this impact performance? The answer was simple: No as a limit on a vCPU is only applied when there’s a resource constraint. It took me a couple of minutes to figure out what he actually tried to tell me but basically it came down to the following:
When a single VM has a limit of 300MHz and is the only VM running on a host than it will run it full speed as it will be constantly rescheduled for 300MHz.
However, that’s not what happens in my opinion. It took me a while to get the wording right but after a discussion with @frankdenneman this is what we came up with:
Look at a vCPU limit as a restriction within a specific time frame. When a time frame consists of 2000 units and a limit has been applied of 300 units it will take a full pass, so 300 “active” + 1700 units of waiting before it is scheduled again.
In other words applying a limit on a vCPU will slow your VM down no matter what. Even if there are no other VMs running on that 4 socket quad core host.
Would I ever recommend setting a limit? Only in very few cases. For instance when you have an old MS DOS application which is polling 10000 times a second it might be useful to limit it. Personally witnessed they can consume 100% of your resources, unnecessary as it isn’t doing anything actually.
In most cases however I would recommend against it. It will degrade user experience / performance and there is no need in my opinion. The VMkernel has got a great scheduler which will take fairness into account.
While reading this article I was shocked! Shocked at the part: “My question of course was doesn’t this impact performance? The answer was simple: No as a limit on a vCPU is only applied when there’s a resource constraint.”
After reading it twice I figured you asked the question to the administrator/designer of those VMs. Oh man, the limit is always applied. That’s why it is called a LIMIT! The same goes for memory, that limit is the LIMIT of physical memory the VM is able to use. The rest goes to swap file – be aware of that!
To be short: I NEVER use limits, unless I have a VM which is using resources it doesn’t need. The keyboard polling is an excellent example, but to be honest, I’ve never seen this in production environments. Who still runs DOS applications anyway in Enterprise Server farms (VDI environments though is a different story).
Anyway, thanks for the post.
The only other time I would recommend limits is if the environment has workloads you want to seperate. For example, if you are sharing resources between a prod and dev environment I would recommend setting a limit on the dev VMs or pool. This would probably only be necessary in smaller infrastructures where a seperate dev cluster is not available but there may be some cases in larger environments as well.
Duncan, are you sure you’re not confusing limits with shares ? From my knowledge (and I can be dead wrong here), limits are always applied, no matter what.
Shares are only applied when the server is constrained, and are relative priority values.
@AJ Ciampa: in that case I would use different shares. Like 800/vcpu for dev/test and 1000/vcpu (I would be careful too with the ‘low-normal-high’ thing too btw, but that’s a different story). It would be a waste of CPU resources when dev/test VM need resources, which are available, but is not scheduled since there is a limit. I would go with Duncan: “The VMkernel has got a great scheduler which will take fairness into account.”
There is another example – we have an foxpro-database running. And microsoft foxpro only runs when the cpu is lesser than 2000 Mhz – so here is the ‘SpeedLimit’ an excelent alternative to an old pc…
Hi,
I used CPU limits all the time during our initial migration to ESX. I had started to P2v servers running on old (slow) hardware and then limited the CPUs on the new VM’s, to a bit more than the old servers. This gave the users a speed increase, but did not give them the full power of the ESX hosts. As a consequence, as I added more old servers, the speed of each P2V did not reduce.
I did not want to create a situation where the first few servers ran fast, then started to slow down as more servers were virtualised. Regardless of how fast the new P2V’d systems were in comparision to the old servers, a user will ALWAYS complain if it starts to slow down later.
Fairness isn’t always the goal. For some of my VMs I want to be brutally unfair, but for the record most of my VMs are uncapped. Shares on a VM gets cleared after a vmotion, so that isn’t the best way in our case as we want DRS to be able to work full-auto. A limit on a CPU isn’t a bad thing if you know why you’re doing it. We have some tools installed on our guests that can “run away” and consume a lot more horsepower than they should, so limiting a VM to what it needs isn’t a bad thing as long as it works and is getting the job done without impact to user experience. I think this can also help you with consolidation as you don’t have to worry that a run away process will use more than you accounted for.
The only gotchya here is that a CPU alarm will not get thrown for a VM with limits in place. The metric is based on actual CPU mhz, not your limit mhz. The only work around I’ve come up with is a limit on each VM where I have to account for the size of the CPUs in my environment — very difficult as newer CPUs have even more mhz to worry about, so all hosts aren’t equal.
err I meant that I set an alarm per VM where the metrics are based on my limit… so if I have it limited to roughly half of the CPUs total mhz, my alarm would fire at 40 or 45%.
In my view there is a role to play for limits in a cloud deployment.
End users like consistency.
Assuming you build your cloud with several service tiers (Tier 1 = “Platinum” with no overcommitment – through to Tier 4 = “Bronze” with 4:1 vCPU:core overcommitment), users on Tier 4 could experience vastly different performance characteristics depending on other workloads being used on the system. Using limits would help smooth that out – whilst I accept it would result in some “waste” of system resources.
As PD UK states, there is a danger in giving users a performance level that’s unsustainable. Using (a fairly conservative) limit on an under-utilised system avoids problems down the line with users feeling that the system has slowed down.
Enterprise LOBs in particular want to be able to benchmark and capacity plan their deployments. Having wildly varying performance levels is not helpful.
IIRC this is the (only?) reason for using limits in the VMware courseware. 🙂
It’s also interesting to hear that there are folks out there who misunderstand how and when limits apply.
The VMware Design exam will thoroughly test understanding of shares, limits, and reservations.
Hi Duncan,
We ran into a similar situation w/CPU limits. Turns out someone set a 500MHz limit on one of our templates. We deployed about 10 or so VMs from this template and we started to scratch our heads a bit when we saw performance issues. We didn’t consider vCPU limits because we had originally agreed that we weren’t going to apply them. I guess that wasn’t made clear.
The limits absolutely have performance implications – we noticed that first hand. During local and remote file transfers, we saw higher than expected completion times. We checked network throughput, firewalls, NIC settings, the array, HBAs and SAN fab, the weather…everything (except limits – see above!) Thinking that it could be related to something running on the VM, we disabled some of the services that were running and tested local file transfers first. We created a 1GB file and ran rsync from one partition to another. The avg throughput averaged about 20Mbps. Sometimes it would peak at 45Mbps, but not often. We attributed the fluctuation to the other running VMs consuming more resources. After we saw the vCPU limit was set to 500MHz, we removed it and performed the same tests. The avg throughput for the rsync went from an avg of 20Mbps, to 90Mbps. As we increased the VM’s memory, the throughput went up. It eventually max’d out at about 150Mbps with 4GB of RAM.
In my environment, I don’t find limits very useful…at least for now. As we continue to run VMs and learn more about the workloads, we may consider applying limits to resources like CPU and memory on a per VM basis. For now, we’ll leave those settings alone.
But you still wouldn’t tick the ‘Unlimited’ box, right? I’ve heard it can drag your host performance down if you have CPU-monopolizing applications. Is that right?
In response to Bouke:
I definitely agree that using limits can be a waste of resources, but using shares improperly can be even worse. Keep in mind the resource pool paradox that Duncan covers here:
http://www.yellow-bricks.com/2010/02/22/the-resource-pool-priority-pie-paradox/
http://www.yellow-bricks.com/2009/11/13/resource-pools-and-shares/
Also, what if you are trying to implement an SLA-based pool? Would you design this with limits or shares?
With all of these tools that we have at our disposal, shares, reservations, limits, we can do some pretty cool things. We just need to pick the right tool for the job. No matter what I do for my customer I try to keep it simple and flexible enough so that we can adjust on the fly with minimal impact.
Your options are ‘unlimited’ or limited really. What unlimited means is that if I assign 1 vCPU then I can use up to 100% of a single physical CPU/core. It won’t be able to usurp the entire system on an 8 core box.
If I assign 1 vCPUs then I can use up to 100% of 2 cores, etc.
The big problem with reservations is it impacts supportability and repeatability. I’ve seen a lot of admins bang their heads against the wall troubleshooting an issue only to find that his counterpart applied a CPU reservation for example.
Even in a 1 man shop it becomes pretty easy to forget.
I tend to address these sorts of issues externally through my monitoring tools, etc. Most servers don’t spend a lot of time at 100% so that may be an event worth sending out a trap/page on so you can log in and figure out what is going on. Even if the box is supposed to use 100% of its allocated CPU resources you have to be careful since it may need to finish the task in a certain amount of time and limiting CPU will most definitely impact that.
I always find it helps to substitute “MHz” with “CPU cycles per second” when talking about limits and reservations. This may not be totally accurate , but IMO it helps to get a better understanding of what’s actually going on.
Of course this should be “Million cpu cycles per second”
@RobUpham : That’s an argument I heard a couple of times but just think about a job when there is resource contention. Instead of scheduling it once I will need to schedule it multiple times due to the limit which will eventually, at least from my knowledge, lead to scheduling issues as the scheduling options you have will decrease.
@RobUpham @duncan I also think the management of user perception is important – the last thing a CIO (or a service provider) wants is inconsistent performance. E.g. some days the “task” completes in under a minute, and some days it takes 5+ minutes because of resource contention -> leads to Incidents due to mismatched expections.
But, my question is how often this actually occurs in practice with vSphere deployments?
I know that some of the complaints leveled at AWS have been related to inconsistent performance related to resource contention.
I can’t see any reason to set a limit directly to a guest directly….mostly as it would be a maintenance nightmare tracking it down later.
I’m more a believer in setting limits on resource pools…such as for development or test guests (if they’re on the same cluster as some production guests).
Good article. ESX cpu scheduling algorithm working fine when all the vms are up and alive most of the time. Please look at this discussion, where the situation is totally different,
http://communities.vmware.com/message/1538410
Hi guys,
As a slight aside, since some of you will be way more knowledgable than myself, how does this “limit” actually get applied ?
After reading Scott Lowe’s short takes #41, he links back to this article after stating
Speaking of Duncan, his post on vCPU limits is a great read and helps dispel a common misconception about vCPU limits. Remember the definition of a hertz folks—300 million cycles per second (300MHz) means 300 million cycles per second. It doesn’t mean 500 million cycles per second, or 700 million cycles per second
For example, if you have a multicore / multisocket box, with CPU’s running at 3GHZ, when you limit a 1 vCPU to, say, 1GHZ, does it just get stopped when it has consumed 1/3 of it’s normal “slot” and then need to wait for it’s next slot ? What about when theres no contention ? Does it have the CPU for 1/3 second and then wait for 2/3 second before sarting again ? Obvsiouly the CPU runs at 3GHz, so when it is presented to this VM, then that VM runs at 3GHz so the only way to limit it to 1GHz, is to runs it for less time.
I am way of the mark here ?
Thanks
Phil
Actually, this is answered in the article. Forget that !
I work for a large outsourcer. As part of a new contract we will soon start charging our customer a monthly fee for hosted VMs based on their CPU power. In the contract CPU power is measured in RPE2 which we then convert to MHz for our particular ESX farm. Each available CPU “size” comes with a default amount of RAM which the customer can increase (to a capped max) for an additional monthly fee.
For our initial pricing we have made our lives easy by offering CPU power increments that map to an integer number of cores in the physical hosts (e.g. a “Small” VM has the CPU power of 1 core of an IBM x3850 M2 server). As time goes on and we change CPU architecture this convenience will no longer hold and we’ll likely need to use CPU MHz limits to enforce “get what you pay for” fairness.
Has anyone else been running an environment like this in production? Any recommendations or gotchas I should be aware of?
Thanks for any wisdom provided,
– Tom
Hey Tom,
That is insanely interesting.
Would you mind emailing me: andrewgaun…AT…..gmail I have been trying to measure cloud performance in some way. The difficulty is trying to come up with a common measure cross hardware platforms. I would be very interested to schmooze.
best
a
Hi Andrew,
Tom makes a reference to RPE2. This is a server relative performance estimate from Ideas International, that allows you to compare server performance irrespective of architecture, age. Its used extensively by large outsourcers for server resource charging. Our web site has further details, or contact sales@ideasinternational.com if you’d like more information