Author: Craig Risinger
Craig is a Consulting Architect working for VMware PSO in the US and a fellow member of the VCDX Panel. During the VCDX panels in Las Vegas a bunch of us had a discussion around Resource Pools and shares and this led to the following article. Thanks Craig! PS: I also wrote an article on this topic.
We run into this on a daily basis; Misunderstanding of the “shares” concept in combination with resource pools. To start with a bold statement: A few VMs in a Low-shares Resource Pool can outperform each of many VMs in a High-shares Resource Pool. How is this possible you might ask.
Resources are divided at the Resource Pool level first. Each Resource Pool is like a pie whose size determines amount of resources usable (during contention). Then that pie is subdivided among the VMs in the pool. A Resource Pool applies to all its VMs collectively. Thus a smaller pie divided among fewer VMs can yield more resources per VM than a larger pie divided by even more VMs.
An example will make it more clear:
Consider a “Test” Resource Pool with 1000 shares of CPU and 4 VMs, vs. a “Production” Resource Pool with 4000 shares of CPU and 50 VMs.
“Test” 1000 shares, 4 VMs => 250 units per VM (small pie, a few big slices):
“Production” 4000 shares, 50 VMs => 80 units per VM (bigger pie, many small slices):
I guess this makes it really obvious that shares might not always give you the results you expected it would.
Note that shares matter only when there is contention. If there are no Reservations or Limits defined, when there’s no resource shortage, every VM gets everything it requests.
vSphere 4.0 Quick Start Guide
I’m pretty sure this came out of my defense. Craig challenged me. I’m pretty sure I defended it correctly.
Not that I can speak for every situation, but I know for our shop using reservations at the resource pool level make the most sense. We do not want to affect slot sizing with reservations, yet want to make sure our production VMs receive a certain amount of dedicated resources if they need it. Shares on a per VM level make more sense or at a vApp level since (at least for us) a vApp consists of only a few servers it is easier to calculate share sizing. I still think per VM shares is best, with reservations at the resouce pool level then vApp level with expandable setting enabled. I think the accidental reversal of intentions happens a lot with shares at the resource pool level.
Keep in mind that outside of a RP itself, shares on a VM reset when it moves from host to host. Even if you hard code a VM from Normal 1000 shares to have 8000 shares, it will reset as soon as you vMotion or migrate it. Shares are a great tool and can make a huge difference in performance on higher density loads, but you really have to manage them. I miss Virtugo, whom back in the day could really make a big difference in performance just by managing shares.
I just finished up the beta design class and the instructor let us know about the implications of resource pools. It seems that a lot of people use them for vm organization which can lead to problems. He pointed out exactly what you mentioned above and I’m sure it affects a lot of people who use them. Thanks!
Ah interesting, I can see how shares would reset from host to host during a migration. I guess I will have to start setting shares at the vApp level, perhaps that is a better place for them in my environment or maybe I have no use for them at all. Contention isn’t really an issue anyway unless I had more than one host failure D:.
I’ve tried representing this requirement below a few different ways and the only way I’ve gotten it to work is a way in which I don’t think is best (yet it seems to work, ie, make people happy, so it’s what we go with). Here’s the requirement: If there is contention (either because utilization is bumping up against resource pool limits or physical host resource limits) the “Production” VM(s) will always win and the “Test” VM(s) will always lose. Perhaps a better, more precise, way to put it is: “If there is a “loser” in the “contention race” it will always be a “Test” VM.
Now I understand there may not be a perfect configuration to match that requirement EXACTLY but over the years I’ve tried to come up with the best representation of that as possible. So far the only layout I’ve found that works (ie, people aren’t calling and complaining) is this:
Finance buys a VM host with 8x3Ghz CPU so I create this:
Finance Resource Pool: “FinanceProduction” (shares: Normal, reservation: 24000, limit: 24000 * 1.1, 10% cushion because I’m such a nice guy)
VM: ProdVM1
VM: ProdVM2
VM: ProdVM3
Sub Resource Pool: “FinanceTest” (shares: LOW, reservation: 0, limit: )
VM: TestVM1
VM: TestVM2
VM: TestVM3
And for every other business unit in my company that purchases VM hosts, we add that host into our Cluster and basically give them back a “virtual host” (aka, their own Resource Pool) where we guarantee them they’ll get the resources they paid for plus 10% (using reservations and limits).
I think this work okay, but in my mind it punishes “Test” VMs too much (like, “Test” VMs won’t just be the losers, they’ll be strapped down and beaten to a pulp). In the past I thought moving the Sub Resource Pool “FinanceTest” out and making it a top-level Resource Pool would be a better implementation of the requirement I’m shooting for, but when I do that, the PROD people start calling my office, fast.
Any better suggestions?
Another point of clarification:
Those relative cpu/memory “share” values get utilized at the physical host layer, right? and not the Resource Pool layer. In other words, each physical host is running, say, 20 VMs and of those 20 there are different shares assigned to various VMs (some are “HIGH” others “NORMAL” and others “LOW”) due to the fact that those 20 are broken up into various Resource Pools. Regardless of the reservations guaranteed to any particular Resource Pool, if VM1 on VMHOST1 is “HIGH” shares and VM2 on that same VMHOST1 is “LOW” shares then VM2 will spend more time in “CPU Ready” land than VM1, right? While cpu/memory shares can get configured at the Resource Pool level, it seems to me that they actually MATTER at the physical host level (they are relative to the VMs on the particular ESX host, NOT the particular Resource Pool they are a part of).
what happens if you define the resource shares using “High”, “Medium” and “Low”? I thought those values dynamically adjusted based on the workload of the ESX host?
It’s certainly not dynamical! It’s in a 4:2:1 ratio.
@Benny:
Re “Regardless of the reservations guaranteed to any particular Resource Pool, if VM1 on VMHOST1 is “HIGH” shares and VM2 on that same VMHOST1 is “LOW” shares then VM2 will spend more time in “CPU Ready” land than VM1, right?”
No, that’s wrong.
First of all, forget about Reservations. That just complicates the discussion. Let’s just talk about RPs and shares.
Shares can be defined at the RP level.
Shares can be defined at the VM level, independently of what shares at the RP are. (RP-Prod can have 1/3 VMs at “High” shares, 1/3 at “Normal”, and 1/3 at “Low” shares at the VM level.)
When there’s contention, resources are divided at the RP level first.
You cannot compare “High” shares on VM-P1 in RP-Prod and “Low” shares on VM-T1 in RP “Test” and say VM-P1 always runs better. Shares can be directly compared only among siblings. If you want to compare shares among VMs that are in different resource pools, the shares _at the RP level_ provide a weighting.
Never forget about reservations, the whole problem comes up because you’re using shares when you should be using reservations.
Take a look at my post which shows where reservations address the requirement you have to guarantee resources to production VMs
http://demitasse.co.nz/wordpress2/2010/02/shares-are-abut-shortages-reservations-are-guarantees/
Please, I am researching since september 09 on a VMware ESX issue, because I moved a simple Apache-PHP-mysql server from a physical server to a VM. As soon as I have 50 to 100 HTML-page requests, the server times out or needs 20 seconds to serve a page that takes 0.5 secs on the physical server.
On our ESX cluster we have 5 servers with about 20 VMs on each. Our ESX admin says there is no need to build Resource Pools because every VM gets “dynamically” as much CPU-cicles/resources as needed, as Michael said in his post above.
So what’s about all those shares (high on my VM), Reservation (0 MHz on my VM), Limit (unlimited on my VM), Memory shares (normal and 8GB on my VM), Memory Reservation (0 MB on my VM), Memory Limit (unlimited on my VM), Advanced CPU Hyper Threading (hyperthreaded core sharing: Any on my VM), Processor Affinity (no affinity on my VM)? I seem to get not enough resources independently of those settings, even reducing the number of CPUs to 1 and back to 4 does not make any difference.
Here (http://www.vmware.com/products/esx/index.html) they say: “VMware ESX and ESXi set the record in virtual performance delivering up to 8,900 db transactions per second, 200,000 I/O operations per second, and up to 16,000 Exchange mailboxes per host.” So WHY can I not get to work my simple SUSE 11 server with apache-php-mysql on that VM??
I apologize because I am not an ESX-engineer, but I start to think that our ESX-admin is not one either or in any case not an experienced one. Any hints would be greately appreciated.
It’s really difficult to give an answer when I don’t know what the actual root cause is of the problem. It can be anything from basic over-provisioning to for instance a storage issue. I recommend you read my esxtop article and look at:
- %RDY
- QUED
- DAVG
- KAVG
and memory consumption in general, swapping etc.
Unfortunately I do not understand the technical abbreviations (%RDY, QUED, etc.) But I can give you more details if it helps:
- HP Blade-System with 5 Servers with 2 Intel Xeon Quad-Core each at 1.86GHz
- Lots of RAM, 60GB on each server
- Lots of SAN Disks/Luns, 9 LUNs at 1TB each
- standard installation, no resource pools, every VM unlimited CPU and 4-8GB RAM
We were sold this very expensive system for a professional use in our university and I think it is a scandal that our most productive VMs show very very poor performance.
I just read the article on RESERVATIONS (http://demitasse.co.nz/wordpress2/2010/02/shares-are-abut-shortages-reservations-are-guarantees/), so should I ask our ESX-Admin to create resource-pools with reservations on it?
Note: we are not talking about fine-tuning to get the last microsecond out of it, we are at a ridiculous level of 50-100 html-page requests causing our production servers to time-out or to return a page after 10-15 seconds!
OK, I had a look at the esxtop article and understand now the abbreviations.
Starting tomorrow I will do some tests with reservations and resource pools.