Yellow Bricks

by Duncan Epping

resource pools

Resource pool shares don’t make sense with vCloud Director?

· ·

I’ve had multiple discussions around Resource Pool level shares in vCloud Director over the last 2 years so I figured I would write an article about it. A lot easier to point people to the article instead, and it also allows me to gather feedback on this topic. If you feel I am completely off, please comment… I am going to quote a question which was raised recently.

One aspect of “noise neighbor” that seems to never be discussed within vCloud is the allocation of shares.  An organization with a single VM has better CPU resource access per VM than an organization that has 100 VMs.  The organization resource pools have equal number of shares, so each VM gets a smaller and smaller allocation of shares as the VM count in an organization virtual data center increases.

Before I explain the rationale behind the design decision around shares behavior in a vCloud environment it is important to understand some of the basics. An Org vDC is nothing more than a resource pool. The chosen “allocation model” for your Org vDC and the specified charateristics determine what your Resource Pool will look like. I wrote a fairly lengthy article about it a while back, if you don’t understand allocation models take a look at it.

When an Org vDC is created on a vSphere layer a resource pool is created and it will typically have the following characteristics. In this example I will use the “Allocation Pool” allocation model as it is the most commonly used:

Org vDC Characteristics –> Resource Pool Characteristics

  • Total amount of resources –> Limit set to Y
  • Percentage of resources guaranteed –> Reservation set to X

On top of that each resource pool has a fixed number of shares. The difference between the limit and the reservation is often referred to as the “bust space”. Typically each VM will also have a reservation set. If 80% of your memory resources are guaranteed this will result in a 80% reservation on memory on your VM as well. This means that when you start deploying new VMs in to that resource pool you will be able to create as many until the limit is reached. In other words:

10GHz/10GB allocation pool Org vDC with 80% guaranteed resources = Resource pool with a 10GHz/GB limit and an 8GHz/GB reservation. In this pool you can create as many VMs until you hit those limits. Resources are guaranteed up to 8GHz/8GB!

Now what about those shares? The statement is, will the Org vDC with 100 VMs have less resource access than the Org vDC with only 10 VMs? Lets use that previous example again:

10GHz/10GB allocation pool with 80% resource guaranteed. This results in a resource pool with a 10GHz/10GB limit and an 8GHz/GB reservation.

Two Org VDCs are deployed, and each have the exact same characteristics. In “Org VDC – 1” 10 VMs were provisioned, while in “Org VDC – 2” 100 VMs are provisioned. It should be pointed out that the provider charges these customers for their Org VDC. As both decided to have 8GHz/GB guaranteed that is what they will pay for and when they exceed that “guarantee” they will be charged for it on top of that. They are both capped at 10GHz/GB however.

If there is contention than shares come in to play. But when is that exactly? Well after the 8GHz/GB of resources has been used. So in that case Org VDCs  will be fighting over:

limit - reservation

In this scenario that is “10GHz/GB – 8GHz/GB = 2GHz/GB”. Is Org VDC 2 entitled to more resource access than Org VDC 1? No it is not. Let me repeat that, NO Org VDC 2 is not entitled to more resources.

Both Org VDC 1 and Org VDC 2 bought the exact same amount of resource. The only difference is that Org VDC 2 chose to deploy more VMs. Does that mean Org VDC 1’s VMs should receive less access to these resources just because they have less VMs? No they should not have less access! A provider cannot, in any shape or form, decide which Org VDC  is entitled to more resources in that burst space, especially not based on the amount of VMs deployed as this gives absolutely no indication of the importance of these workloads.Org VDC 2 should buy more resources to ensure their VMs get what they are demanding.

Org VDC 1 cannot suffer because Org VDC 2 decided to overcommit. Both are paying for an equal slice of the pie… and it is up to themselves to determine how to carve that slice up. If they notice their slice of the pie is not big enough, they should buy a bigger or an extra slice!

However, there is a a scenario where shares can cause a “problem”… If you use “Pay As You Go” and remove all “guarantees” (reservations) and have contention in that scenario each resource pool will get the same access to the resources. If you have resource pools (Org VDCs) with 500 VMs and resource pools with 10 VMs this could indeed lead to a problem for the larger resource pools. Keep in mind that there’s a reason these “guarantees” were introduced in the first place, and overcommitting to the point where resources are completely depleted is most definitely not a best practice.

Shares set on Resource Pools

· ·

During our session at the Dutch VMUG Frank was explaining Resource Pools and the impact of limits and reservations. As I had the feeling not everyone in the room was using resource pools I asked the following questions:

  1. How many people are using Resource Pools today?
    • Out of the roughly 300 people who attended our session 80 showed their hands. The follow-up question I asked was…
  2. How many people change the Shares setting from the default?
    • Out of those 80 hands roughly 20 people raised their hands and that lead me to the next question…
  3. How many people change the Shares value based on the amount of VMs running in that Resource Pool?
    • Now only a handful of people raised their hand.

That is what triggered this post as I believe it is an often made mistake. First of all when you create a Resource Pool there are a couple of things you can set a reservation, a limit and of course shares. For some reason shares are often overlooked. There are a couple of things I wanted to make sure everyone understands as judging by the numbers of hands that were raised I am certain there are a couple of common misunderstandings when it comes to Resource Pools:

  • If you create a Resource Pool a default Shares value is defined for the resource pool on both Memory and CPU
  • Shares specify the priority of the resource pool relative to other resource pools on the same level

This means that even if you don’t touch the shares values they will come into play whenever there is contention. This also means that the resource allocation on a VM level is dependent on the entitlement of the resource pool it belongs to.

Now what is the impact of that? I guess I should quote from the “The Resource Pool Priority-Pie Paradox” blog post my colleague Craig Risinger wrote as it clearly demonstrates the issues that can be encountered when Resource Pools are used and Shares values are not based on the relative priority AND the amount of VMs per pool.

“Test” 1000 shares, 4 VMs => 250 units per VM (small pie, a few big slices):

“Production” 4000 shares, 50 VMs => 80 units per VM (bigger pie, many small slices):

I guess this makes it really obvious that shares might not always give you the results you expected it would.

Another issue that could arise is when Virtual Machines are created on the same level as the Resource Pools…. Believe me it doesn’t take a lot for a single VM to have higher priority than a Resource Pool in times of contention.

Again, whenever you create a Resource Pool it will “inherit” the default shares value, which equals a 4vCPU/16GB Virtual Machine, and whenever there is contention these will come into play. Keep this in mind when designing your virtual infrastructure as it could potentially lead to unwanted results.

Custom shares on a Resource Pool, scripted

· ·

We’ve spoken about Resource Pools a couple of times over the last months and specifically about shares. (The Resource Pool Priority-Pie Paradox, Resource Pools and Shares.) The common question I received was how can we solve this. The solution is simple: Custom Shares.

However, the operational overhead associated with custom shares is something most people want to avoid. Luckily for those who have the requirement to use share based resource pools one my colleague Andrew Mitchell shared a powershell script. This powershell script defines custom shares based on a pre-defined weight and the amount of VMs / vCPUs in the resource pool. I would recommend to schedule the script to run on a weekly basis and ensure the correct amount of shares have been set to avoid running into one of the scenarios described in the articles above.

Please keep in mind that if you use nested resource pools you will need to run a separate script for each level in the hierarchy.

Eg. If the resource pools are setup like this the following you will need one script to set the shares for RP1, RP2 and RP3, and another script to set the shares for RP1-Child1 and RP1-Child2.

RP1
>>RP1-Child1
>>RP1-Child2
RP2
RP3

Download the script here. Again to emphasize it I am not the author, we would appreciate it though if you could share any modifications / enhancements to this script.