Yellow Bricks

by Duncan Epping

Server

Are you using DPM? We need you!

· ·

I just received a request from our Product Management team. Gil Adato is is working on the next generation of DPM and is seeking DPM current and past users and he asked me to post the following message on my blog. I hope you can help Gil and VMware taking DPM to the next level!

Dear VMware Customers,

Product management is starting planning efforts for the 2012 vSphere release, and we are considering numerous DPM improvements. In order to make the right decision, it’s critical that we get a better understanding of our customers’ experiences with the current product, and get customers’ feedback on what needs to be on the roadmap. We’re also interested in customers’ improvement suggestions and in sharing with them several improvement ideas that we have in mind. Customers’ input will be very helpful to us, and customers would see the benefit of communicating their comments, requirements and suggestions/wish-list directly to the product team.

If you’re a current or past DPM user and you’d like to be a part of the process and help shape the next generation of VMware’s products, please contact me directly.

My contact info is the following:

Gil Adato
gadato@vmware.com

When contacting me, please send me the following initial information:

  • Company Name
  • Customer contact name
  • Location (Country)
  • Customer’s email address

VMware’s product management team is looking forward to hearing from you and making you a part of the product development process.

Thank you for your cooperation,

Gil Adato

Memory Limits

· ·

We had a discussion internally around memory limits and what the use case would be for using them. I got some great feedback on my reply and comments so I decided to turn the whole thing into a blog article.

A comment made by one of our developers, which I highly respect, is what triggered my reply. Please note that this is not VMware’s view or usecase but what some of our customers feed back to our development team.

An admin may impose a limit on VMs executing on an unloaded host to better reflect the actual service a VM will likely get once the system is loaded; I’ve heard this use case from several admins)

From a memory performance perspective that is probably the worst thing an Admin can do in my humble opinion. If you are seriously overcommitting your hosts up to the point where swapping or ballooning will occur you need to think about the way you are provisioning. I can understand, well not really, people doing it on a CPU level as the impact is much smaller.

Andrew Mitchell commented on the same email and his reply is key to understanding the impact of memory limits.

“When modern OS’s boot, one of the first things they do is check to see how much RAM they have available then tune their caching algorithms and memory management accordingly. Applications such as SQL, Oracle and JVMs do much the same thing.”

I guess the best way to explain in one line is: The limit is not exposed to the OS itself and as such the App will suffer and so will the service provided to the user.

The funny thing about this is that although the App might request everything it can it, it might not even need it. In that case, more common than we think, it is better to decrease provisioned memory than to create an artificial boundary by applying a memory limit. The limit will more than likely impose an unneeded and unwanted performance impact. Simply lowering the amount of provisioned memory might impact performance but most likely will not as the OS will tune it’s caching algorithms and memory management accordingly.

Changes to Snapshot mechanism “Delete All”

· ·

Don’t know if anyone noticed it or not but with the latest set of patches VMware changed the “Delete All” mechanism that is part of the Snapshot feature. I wrote multiple articles about the “Delete All” functionality as it often led to completely filled up VMFS volumes when someone used without knowing the inner workings.

Source

When using the Delete All option in Snapshot Manager, the snapshot farthest from the base disk is committed to its parent, causing that parent snapshot to grow. When the commit is complete, that snapshot is removed and the process starts over on the newly updated snapshot to its parent. This continues until every snapshot has been committed.

This method can be relatively slow since data farthest from the base disk might be copied several times. More importantly, this method can aggressively use disk space if the snapshots are large, which is especially problematic if a limited amount of space is available on the datastore. The space issue is troublesome in that you might choose to delete snapshots explicitly to free up storage.

This issue is resolved in this release in that the order of snapshot consolidation has been modified to start with the snapshot closest to the base disk instead of farthest. The end result is that copying data repeatedly is avoided.

Just to give an example, 4 snapshots:

Old situation (pre vSphere 4 Update 2)

  • Base disk – 15GB
  • Snapshot 1 – 1GB –> possibly grows to 13GB
  • Snapshot 2 – 1GB –> possibly grows to 12GB
  • Snapshot 3 – 1GB –> possibly grows to 11GB
  • Snapshot 4 – 10GB

Snapshot 4 is copied in to Snapshot 3, Snapshot 3 in to Snapshot 2, Snapshot 2 in to Snapshot 1 and Snapshot 1 in to your Base disk. After the copy of Snapshot 1 in to the Base disk all Snapshots will be deleted. Please note that the total amount of diskspace consumed before the “Delete All” was 28GB. Right before the final merge the consumed diskspace is 61GB. This is just an example, just imagine what could happen with a 100GB data disk!

New situation

  • Base disk – 15GB
  • Snapshot 1 – 1GB
  • Snapshot 2 – 1GB
  • Snapshot 3 – 1GB
  • Snapshot 4 – 10GB

Snapshot 1 is copied in to Base disk, Snapshot 2 is copied in to Base disk, Snapshot 3 in to Base disk and Snapshot 4 in to your Base disk. After the copy of Snapshot 4 in to the Base disk all Snapshots will be deleted. Please note that the total amount of diskspace consumed before the “Delete All” was 28GB. Right before the final merge the consumed diskspace is still 28GB. Not only did VMware reduced the chances of running out of disk space, the time to commit the snapshot by using “delete all” has also been decreased using this new mechanism.

How does “das.maxvmrestartcount” work?

· ·

The amount of retries is configurable as of vCenter 2.5 U4 with the advanced option “das.maxvmrestartcount”. My colleague Hugo Strydom wrote about this a while ago and after a short discussion with one of our developers I realised Hugo’s article was not 100% correct. The default value is 5. Pre vCenter 2.5 U4 HA would keep retrying forever which could lead to serious problems as described in KB article 1009625 where multiple virtual machines would be registered on multiple hosts simultaneously leading to a confusing and inconsistent state. (http://kb.vmware.com/kb/1009625)

Important to note is that HA will try to start the virtual machine one of your hosts in the affected cluster; if this is unsuccessful on that host the restart count will be increased by 1. The first restart attempt will than occur after two minutes. If that one fails the next will occur after 4 minutes, and if that one fails the following will occur after 8 minutes until the “das.maxvmrestartcount” has been reached.

To make it more clear look at the following:

  • T+0 – Restart
  • T+2 – Restart retry 1
  • T+4 – Restart retry 2
  • T+8 – Restart retry 3
  • T+8 – Restart retry 4
  • T+8 – Restart retry 5

In other words, it could take up to 30 minutes before a successful restart has been initiated when using the default of “5” restarts max. If you increase that number, each following will also be “T+8” again.

CMDS/s vs IOPS?

· ·

Today I received a question around the difference between IOPS and CMDS/s. The reason for this was the high value of CMDS/s in “esxtop” which exceeded the expected amount of IOPS the disks could actually digest. I thought it would useful for everyone to know what the difference is:

  • IOPS = Input/Output Operations Per Second
    • Within esxtop this would be the outcome of “Number of Read commands(READS/s) + Number of Write commands(WRITES/s)”
  • CMDS/s = Total commands per second
    • Within esxtop this includes any command(for instance SCSI reservations) and not necessary only read/write IOs

One thing to stress though is that in any case the CMDS/s should be relatively close to IOPS, but when there are a lot of metadata changes due to snapshots for instance the difference can be significant. Where this significant difference came from is something we are still investigating and we are hoping to solve pretty soon. If we manage to solve it you can expect an update here.