Again an interesting discussion we had amongst some colleagues (Thanks Frank, Andrew and Craig! Especially Craig as most text below comes from The Resource Master). The topic was CPU/Memory reservations and more specifically the difference in behavior of these two.

One would expect that both a CPU and Memory reservation would have the same behavior when it comes to claiming and releasing resources but unfortunately this is not the case. Or should we say fortunately?

The following is taken from the resource management guide:

CPU Reservation:
Consider a virtual machine with reservation=2GHz that is
totally idle. It has 2GHz reserved, but it is not using any of
its reservation. Other virtual machines cannot reserve these 2GHz. Other virtual machines can use these 2GHz, that is, idle
CPU reservations are not wasted.

Memory Reservation:
If a virtual machine has a memory reservation but has not yet accessed its full reservation, the unused memory can be reallocated to other virtual machines. After a virtual machine has accessed its full reservation, ESX Server allows the virtual machine to retain this much memory, and will not reclaim it, even if the virtual machine becomes idle and stops accessing memory.

The above paragraph is a bit misleading , as it seems to imply that a VM has to access its full reservation. What it should really say is “Memory which is protected by a reservation will not be reclaimed by ballooning or Host-level swapping even if it becomes idle,” and “Physical machine memory will not be allocated to the VM until the VM accesses virtual RAM needing physical RAM backing.” Then that pRAM is protected by the reservation and won’t be reclaimed by ballooning or .vswp-file swapping. If there is any .vswp memory at all as no .vswp is created when the reservation is equal to the provisioned memory.

Note, however, that even if pRAM is not allocated to the VM to back vRAM because the VM hasn’t accessed corresponding vRAM yet, the whole reservation is reserved, but the pRAM could still be used  This gets really confusing. But I think of it thus:

1. Reservations can be defined at the VM level or the Resource Pool level.
2. Reservations at the RP level are activated or reserved immediately.
3. Reservations at the VM level are activated or reserved when the VM is powered on.
4. An activated reservation is removed from the total physical Resource “Unreserved” accounting.
5. Reserving and using a resource are distinct: memory or CPU can be reserved but not used or used but not reserved.
6. CPU reservations are friendly.
7. Memory reservations are greedy and hoard memory.
8. Memory reservations are activated at startup, yet pRAM is only allocated as needed. Unallocated pRAM may be used by others.
9. Once pRAM is protected by a memory reservation, it will never be reclaimed by ballooning of .vswp-swapping even if the corresponding vRAM is idle.

Example: A VM has 4 GB of vRAM installed and a 3 GB memory reservation defined. When the VM starts, 3 GB of pRAM are reserved. If the host had 32 GB of RAM installed and no reservations active, it now has 29 GB “unreserved”.

However, if the VM accesses only 500 MB of vRAM, only 500 MB of pRAM are allocated (or granted) to it. Other VMs could use 2500 MB of RAM that you would think is part of the reservation. They cannot reserve that 2500 MB however. As soon as the VM accesses 3 GB of vRAM and so has 3 GB of pRAM backing it, no other VMs can use that 3 GB of pRAM even if the VM never touches it again, because that pRAM is now protected by the 3 GB Reservation.  If the VM uses 4 GB, it gets the 3 GB guaranteed never ballooned or swapped, but the remaining 1 GB is subject to ballooning or swapping.

Simple huh