Yellow Bricks

by Duncan Epping

VMware

Undocumented VCB config.js feature

· ·

One of my customers wanted to use the default VCB framework but did not want to quiesce the VM for several reasons. (Databases, Active Directory etc.) I could not find an option in the config.js file but noticed the following in the file glue.js:

// A fallback to be able to switch to non-quiesced snapshots
if (typeof(NO_QUIESCE) != "undefined") {
cmd +="-Q 0 ";
}

In other words, setting the option “NO_QUIESCE” with no value in config.js results in the VM not being quiesced, default it will quiesce the VM! I added the following line to the  config.js file to accomplish this:

NO_QUIESCE="";

VCB – Weird errors

· ·

Today I encountered a weird error during an implementation of VMware Consolidated Backup. When ever I ran vcbmounter I received the following error:”Run time error – the application the application has requested the Runtime to terminate in an unusual way.”

It seemed that for some reason the vcbuser did not have the role “VMware consolidated backup user” linked anymore. I would have expected an error in the line of “permission denied” or “not enough permissions to complete task” and definitely not a Runtime error…

Anyway I tested the script I blogged about yesterday and it works like a charm, check it out!

CDP, why?

· ·

That’s what I thought at first, why do I need CDP(Cisco Discovery Protocol)? Well yesterday CDP proved to be an excellent option for troubleshooting weird networking problems. One of my customers replaced a faulty NIC and accidentally switched two network cables. One of these cables was part of an etherchannel, this caused all kind of strange routing / connection errors to the virtual machines. CDP was very useful because I could setup a remote connection to the customer and check which vmnic was connected to which port on the switch. So use CDP when in doubt!
Click on the blue cloud next to the vmnic name on the right:

This is the result:

(Screenshots taken from the RTFM Upgrade Guide.)

VCB – Backup all running VMs

· ·

Today I was busy trying to find out a way to schedule VCB backups with Commvault with just 1 schedule without having to rewrite your script every time someone creates a VM. The standard procedure is to create a subclient for every VM which is very human error prone. I stumbled upon a blog on RTFM about backing up all running VM’s, the actual script was done by Andrew Neilson, thanks! [Read more…] about VCB – Backup all running VMs

CPU utilization increasing after VMotion in a DRS enabled cluster

· ·

VMwarewolf already posted this fix on his blog but had to remove it… Now VMware added it to their knowledge base. Check out the original article because it may change in time. For the lazy people I included how to diagnose the problem and more…

Diagnose the problem:

  1. Use the VI Client to log in to VirtualCenter as an administrator.
  2. Disable DRS in the cluster and wait for 1 minute.
  3. In the VI Client, note the virtual machine’s CPU usage from performance tab.
  4. In the VI Client, note the virtual machine’s memory overhead in the summary tab.
  5. Enable DRS in the cluster.
  6. Use VMotion to move the problematic virtual machine to another host.
  7. Note the virtual machine CPU usage and memory overhead on the new host.
  8. Disable DRS in the cluster and wait for 1 minute.
  9. Note the virtual machine CPU usage and memory overhead on the new host.

If the CPU usage of the virtual machine increases in step 7 in comparison to step 3, and decreases back to the original state (similar to the behavior in step 3) in step 9 with an observable increase in the overhead memory, this indicates the issue discussed in this article.

You do not need to disable DRS to work around this issue.

The workaround:

  1. Use the VI Client to log in to VirtualCenter as an administrator.
  2. Right-click your cluster from the inventory.
  3. Click Edit Settings.
  4. Ensure that VMware DRS is shown as enabled. If it is not enabled check the box to enable VMware DRS.
  5. Click OK.
  6. Click an ESX Server from the Inventory.
  7. Click the Configuration tab.
  8. Click Advanced Settings.
  9. Click the Mem option.
  10. Locate the Mem.VMOverheadGrowthLimit parameter.
  11. Change the value of this parameter to 5. (Note: By default this setting is set to -1.)
  12. Click OK.

To verify the setting has taken effect:

Log in to your ESX Server service console as root from either an SSH Session or directly from the console of the server.

  1. Type less /var/log/vmkernel.

A successfully changed setting displays a message similar to the following and no further action is required:
vmkernel: 1:16:23:57.956 cpu3:1036)Config: 414: VMOverheadGrowthLimit” = 5, Old Value: -1, (Status: 0x0)

If changing the setting was unsuccessful a message similar to the following is displayed:
vmkernel: 1:08:05:22.537 cpu2:1036)Config: 414: “VMOverheadGrowthLimit” = 0, Old Value: -1, (Status: 0x0)

Note: If you see a message changing the limit to 5 and then changing it back to -1, the fix is not successfully applied.

To fix multiple ESX Server hosts:

If this parameter needs to be changed on several hosts (or if the workaround fails for the individual host) use the following procedure to implement the workaround instead of changing every server individually:

  1. Log on to the VirtualCenter Server Console as an administrator.
  2. Make a backup copy of the vpxd.cfg file (typically it is located in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg).
  3. In the vpxd.cfg file, add the following configuration after the <vpxd> tag:
    <cluster>
    <VMOverheadGrowthLimit>5</VMOverheadGrowthLimit>
    </cluster>
    This configuration provides an initial growth margin in MB-to-virtual machine overhead memory. You can increase this amount to larger values if doing so further improves virtual machine performance.
  4. Restart the VMware VirtualCenter Server Service.Note: When you restart the VMware VirtualCenter Server Service, the new value for the overhead limit should be pushed down to all the clusters in VirtualCenter.