Yellow Bricks

by Duncan Epping

memory

Service Console Memory, a common misunderstanding (ESX 4.0+)

· ·

I was reading the Maximum vSphere book by Eric Siebert today and noticed something that I also spotted in Scott Lowe’s Mastering VMware Sphere book. Both Scott and Eric described the fact that the default amount of assigned Service Console memory for ESX has been increased from 272MB to X. I deliberately use “X” as both Eric and Scott mention a different value in their book.

The reason both Scott and Eric mention a different value in their book can be easily explained though, and I wrote about this a while ago, as of vSphere 4.0 there is no default value anymore. I gave this feedback to Scott a while back and of course he asked where this was documented. Back then it was nowhere to be found, well except for on my blog but that is not an official VMware publication. I asked our KB team to update the KB article that explains Service Console memory and I just noticed that they have:

src

ESX 4.x hosts – the default amount of RAM is dynamically configured to a value between 300MB and 800MB, depending on the amount of RAM that is installed in the host. For example, if the host has 32GB of memory the service console RAM will be set to 500MB, while a host which has 128GB of RAM will see the service console RAM set to 700MB. The maximum has not changed from 800MB, which would be seen on hosts with 256GB of RAM or higher, if it is being dynamically allocated.

This is exactly what I observed almost a year ago:

  • ESX Host – 8GB RAM -> Default allocated Service Console RAM = 300MB
  • ESX Host – 16GB RAM -> Default allocated Service Console RAM = 400MB
  • ESX Host – 32GB RAM -> Default allocated Service Console RAM = 500MB
  • ESX Host – 64GB RAM -> Default allocated Service Console RAM = 602MB
  • ESX Host – 96GB RAM -> Default allocated Service Console RAM = 661MB
  • ESX Host – 128GB RAM -> Default allocated Service Console RAM = 703MB
  • ESX Host – 256GB RAM -> Default allocated Service Console RAM = 800MB

Just wanted to point this out as I am certain that many people will not be aware of this.

Reclaiming idle memory

· ·

In the “CPU/MEM Reservation Behavior” article there was a lively discussion going on between Chris Huss(vmtrainers.com) and myself. I think the following comment by Chris more or less summarizes the discussion

I wasn’t aware that the balloon driver was involved with the Mem.IdleTax. I haven’t seen any documentation stating this…and assumed that the VMkernel just stopped mapping idle memory for the VM without letting it know. If the VM needed the memory again, the VMkernel would just re-map it.

I can be totally wrong about this, but I have not seen any documentation to debunk this theory. It is my belief that the Mem.IdleTax is a totally separate memory saving/shaving technique from the balloon driver or the .vswp file.

If VMware engineering has or would publish an official article on this…I think it would clear up alot of things.

To summarize; How does ESX reclaim idle memory or free memory from a virtual machine? The answer is simple. ESX has two idle memory reclamation mechanisms:

  1. Balloon driver
  2. vSwap

I would like to refer to page 29 of the Resource Management Guide where the above is stated. I do not think it is a coincidence that the paragraph above “memory reclamation” is “Memory Tax for Idle Virtual Machines”. (There is a third memory “reclamation” mechanism by the way, it is called “TPS”, but this is not used to specifically reclaim Idle Memory but rather to free up memory by sharing pages where possible.)

By default the balloon driver is used to reclaim idle memory. The balloon driver is in fact used as some operating systems only update there internal free memory map. Basically what I am saying is that the hypervisor is unaware of the fact that specific pages are unused as they might still contain data and the GOS(Guest Operating System) will not report to the hypervisor that the pages are not being used anymore. The balloon driver is used to notify the GOS that there is a lack of memory.

When the balloon inflates the GOS will first assign all “unused / free” pages to the balloon driver. If this is enough it will stop. If this isn’t enough the OS will decide which pages it will page out  until it reaches its threshold. The pages will need to be written to GOS swap as they might be needed later, they can’t just be reused without storing them somewhere.

I guess this section of the excellent white-paper “Memory Resource Management in VMware ESX Server” by Carl Waldspruger describes what I explained above.

The guest OS decides which particular pages to reclaim and, if necessary, pages them out to its own virtual disk. The balloon driver communicates the physical page number for each allocated page to ESX Server, which may then reclaim the corresponding machine page.

To be absolutely certain I reached out Carl Waldspruger to verify my statements/claims are correct. (Yes they were…)

By the way this concept is also described in the “VMware vSphere: Manage for Performance” course manual on page 151. Excellent course which I can recommend to everyone as it will not only explain this concept but also how to identify it and how to resolve it.