Yellow Bricks

by Duncan Epping

vcenter

CPU/MEM Reservation Behavior

· ·

Again an interesting discussion we had amongst some colleagues (Thanks Frank, Andrew and Craig! Especially Craig as most text below comes from The Resource Guru). The topic was CPU/Memory reservations and more specifically the difference in behavior of these two.

One would expect that both a CPU and Memory reservation would have the same behavior when it comes to claiming and releasing resources but unfortunately this is not the case. Or should we say fortunately?

The following is taken from the resource management guide:

CPU Reservation:
Consider a virtual machine with reservation=2GHz that is
totally idle. It has 2GHz reserved, but it is not using any of
its reservation. Other virtual machines cannot reserve these 2GHz. Other virtual machines can use these 2GHz, that is, idle
CPU reservations are not wasted.

Memory Reservation:
If a virtual machine has a memory reservation but has not yet accessed its full reservation, the unused memory can be reallocated to other virtual machines. After a virtual machine has accessed its full reservation, ESX Server allows the virtual machine to retain this much memory, and will not reclaim it, even if the virtual machine becomes idle and stops accessing memory.

The above paragraph is a bit misleading , as it seems to imply that a VM has to access its full reservation. What it should really say is “Memory which is protected by a reservation will not be reclaimed by ballooning or Host-level swapping even if it becomes idle,” and “Physical machine memory will not be allocated to the VM until the VM accesses virtual RAM needing physical RAM backing.” Then that pRAM is protected by the reservation and won’t be reclaimed by ballooning or .vswp-file swapping. If there is any .vswp memory at all as no .vswp is created when the reservation is equal to the provisioned memory.

Note, however, that even if pRAM is not allocated to the VM to back vRAM because the VM hasn’t accessed corresponding vRAM yet, the whole reservation is reserved, but the pRAM could still be used  This gets really confusing. But I think of it thus:

  1. Reservations can be defined at the VM level or the Resource Pool level.
  2. Reservations at the RP level are activated or reserved immediately.
  3. Reservations at the VM level are activated or reserved when the VM is powered on.
  4. An activated reservation is removed from the total physical Resource “Unreserved” accounting.
  5. Reserving and using a resource are distinct: memory or CPU can be reserved but not used or used but not reserved.
  6. CPU reservations are friendly.
  7. Memory reservations are greedy and hoard memory.
  8. Memory reservations are activated at startup, yet pRAM is only allocated as needed. Unallocated pRAM may be used by others.
  9. Once pRAM is protected by a memory reservation, it will never be reclaimed by ballooning of .vswp-swapping even if the corresponding vRAM is idle.

Example: A VM has 4 GB of vRAM installed and a 3 GB memory reservation defined. When the VM starts, 3 GB of pRAM are reserved. If the host had 32 GB of RAM installed and no reservations active, it now has 29 GB “unreserved”.

However, if the VM accesses only 500 MB of vRAM, only 500 MB of pRAM are allocated (or granted) to it. Other VMs could use 2500 MB of RAM that you would think is part of the reservation. They cannot reserve that 2500 MB however. As soon as the VM accesses 3 GB of vRAM and so has 3 GB of pRAM backing it, no other VMs can use that 3 GB of pRAM even if the VM never touches it again, because that pRAM is now protected by the 3 GB Reservation.  If the VM uses 4 GB, it gets the 3 GB guaranteed never ballooned or swapped, but the remaining 1 GB is subject to ballooning or swapping.

Simple huh 😉

Re: Memory Compression

· ·

I was just reading Scott Drummonds article on Memory Compression. Scott explains where Memory Compression comes in to play. I guess the part I want to reply on is the following:

VMware’s long-term prioritization for managing the most aggressively over-committed memory looks like this:

  1. Do not swap if possible.  We will continue to leverage transparent page sharing and ballooning to make swapping a last resort.
  2. Use ODMC to a predefined cache to decrease memory utilization.*
  3. Swap to persistent memory (SSD) installed locally in the server.**
  4. Swap to the array, which may benefit from installed SSDs.

(*) Demonstrated in the lab and coming in a future product.
(**) Part of our vision and not yet demonstrated.

I just love it when we give insights in upcoming features but I am not sure I agree with the prioritization. I think there are several things that one needs to keep in mind. In other words there’s a cost associated to these decisions / features and your design needs to adjusted to these associated effects.

  1. TPS -> Although TPS is an amazing way of reducing the memory footprint you will need to figure out what the ratio of deduplication is. Especially when you are using Nehalem processors there’s a serious decrease. The reasons for the decrease of TPS effectiveness are the following:
    • NUMA – By default there is no inter node transparent page sharing (read Frank’s article for more info on this topic)
    • Large Pages – By default TPS does not share large(2MB) pages. TPS only shares small(4KB) pages. It will break large pages down in small pages when memory is scarce but it is definitely something you need to be aware off. (for more info read my article on this topic.
  2. Use ODMC -> I haven’t tested with ODMC yet and I don’t know what the associated cost is at the moment.
  3. Swap on local SSD -> Swap on local SSD will most definitely improve the speed when swapping occurs. However as Frank already described in his article there is an associated cost:
    • Disk space – You will need to make sure you will have enough disk space available to power on VMs or migrate VMs as these swap files will be created at power on or at migration.
    • Defaults – By default .vswp files are stored in the same folder as the .vmx. Changing this needs to be documented and taken into account during upgrades and design changes.
  4. Swap to array (SSD) -> This is the option that most customers use for the simple reason that it doesn’t require a local SSD disk. There are no changes needed to enable it and it’s easier to increase a SAN volume than it is to increase a local disk when needed. The associated costs however are:
    • Costs – Shared storage is relatively expensive compared to local disks
    • Defaults – If .vswp files need to be SSD based you will need to separate the .vswp from the rest of the VMs and created dedicated shared SSD volumes.

I fully agree with Scott that it’s an exciting feature and I can’t wait for it to be available. Keep in mind though that there is a trade off for every decision you make and that the result of a decision might not always end up as you expected it would. Even though Scott’s list makes totally sense there is more than  meets the eye.

VMware vCenter Update Manager 4.0 Update 1 Patch 1

· ·

VMware just released VMware vCenter Update Manager 4.0 Update 1 Patch 1.

This patch resolves the following issues :

  • After upgrading Cisco Nexus 1000V VSM to the latest version, you might not be able to patch the kernel of ESX hosts attached to the vDS (KB 1015717)Upgrading Cisco Nexus 1000V VSM to the latest version upgrades the Cisco Virtual Ethernet Module (VEM) on ESX hosts attached to the vDS. Subsequently, from the same vSphere Client instance, you might not be able to use a host patch baseline to apply patches to the ESX vmkernel64 or ESXi firmware of hosts attached to the vDS. Applying patches to ESX vmkernel64 or ESXi firmware requires that you include the compatible Cisco Nexus 1000V VEM patch in the baseline. However, this patch might not be available for selection in the Update Manager New Baseline wizard or in the Update Manager patch repository.
  • Upgrade of Cisco Nexus 1000V version 4.0(4)SV1(1) to version 4.0(4)SV1(2) with Update Manager might fail for hosts with certain patch levels (KB 1017069)If you are using Cisco Nexus 1000V version 4.0(4)SV1(1), and the ESX patch bulletins ESX400-200912401-BG or ESXi400-200912401-BG are installed on the host, you might not be able to upgrade to Cisco Nexus 1000V version 4.0(4)SV1(2).
  • Scanning of hosts in a cluster and staging of patches to hosts in a cluster might take a long time to finishThe scanning and staging operations of hosts in a cluster run sequentially. If a cluster contains a lot of hosts, scanning and staging patches might take a long time to be completed. Scanning and staging of hosts in a cluster run concurrently on all of the selected hosts in the cluster.

For details regarding these new fixes, please refer to the release notes.

VMware vCenter Update Manager 4.0 Update 1 Patch 1 is available for download.

VMware vCenter Update Manager 4.0 Update 1 is required for installation of this patch.

vSphere Security Hardening Guide script by @lamw

· ·

A couple of weeks ago I blogged about the vSphere Security Hardening Guide. Just a couple of days later William “the king of Perl” Lam already produced a script that checks the Hardening Guide best practices against your environment. It produces a great html based report.

Source

While going through the COS/HOST and VM documentation, I noticed there were quite a few checks that might benefit from having a script to validate the guidelines and that was the motivation for this script. Not all sections can be validated using the vSphere APIs and will require some manual validation and I’ve seperated the types of passes whether it’s a fail, pass or manual (which requires user intervention).

The script allows you to run the current existing guides as of (01/29/2010) against vCenter 4.0 hosting ESX(i) 4.0 hosts/virtual machines OR run it against an individual ESX(i) 4.0 host. The script allows you to run a subset of the checks and against different type of validation (ENTERPRISE,DMZ or SSLF). Upon completion, a report is generated including a grade for your environment.

A couple of details on the features:

  • Email report
  • Ability to execute subset of the checks (COS,HOST,VCENTER,VNETWORK,VM)
  • Ability execute specific test suite (ENTERPRISE,DMZ,SSLF)
  • Detail HTML summary report with letter grade

You can find an example report here. Great work again William, keep it up!

Setup cannot create vCenter Server Directory Services instance. Error 28038

· ·

While doing a new install of VMware vCenter Server I ran into the following error:

Setup cannot create vCenter Server Directory Services instance. Error 28038

This error is caused by the fact that the “Network Service” does not have enough permissions on the root of the drive you’re installing vCenter on. The solution is pretty straight forward and has been described in the KB article.

  1. Right-click the root drive and click Properties.
  2. Click the Security tab.
  3. Under Group and user names, click Add.
  4. Enter Network Service and click OK.
  5. Check Allow for the Read permission for the Network Service account in the Permissions for Administrators pane.
  6. Click Apply and OK.

If this does not resolve the issue look into the following KB articles: 1015887 , 1013822.