Yellow Bricks

by Duncan Epping

VMware

Multiple virtual CPU vm’s

· ·

I was always under the impression that ESX 3.x still used “strict co-scheduling” as 2.5.x did. In other words when you have a multi vcpu vm all vcpu’s need to be scheduled and started at the same time on seperate cores/cpu’s. You can imagine that this can cause the VM to have high “ready times”, which means waiting for physical cpu’s to be ready to serve the multiple cpu task.

About a week ago and a month ago two blog’s appeared around this subject which clarifies the way ESX does vcpu scheduling as of 3.x. Read them for more in depth information. (1, 2)

So in short. Since 3.x VMware ESX uses “relaxed coscheduling”. And is this as relaxed as the name implies? Yes it is. And for a simple reason:

Idle vCPUs, vCPUs on which the guest is executing the idle loop, are detected by ESX and descheduled so that they free up a processor that can be productively utilized by some other active vCPU. Descheduled idle vCPU’s are considered as making progress in the skew detection algorithm. As a result, for co-scheduling decisions, idle vCPUs do not accumulate skew and are treated as if they were running . This optimization ensures that idle guest vCPUs don’t waste physical processor resources, which can instead be allocated to other VMs.

In other words VM’s with multiple vCPU’s don’t take up cycles anymore when these vCPU’s aren’t used by the OS. ESX checks the CPU’s for the idle proces loop and when it’s idle the CPU will be released and available for other vCPU’s. This also means that when you are using an application that will use all vCPU’s the same problems will still exists as they did in ESX 2.5, my advice don’t over do it. Only 1 vCPU is more than enough most of the times!

And what appeared to be a sidenote in the blog that deserves special attention is the following statement:

The %CSTP column in the CPU statistics panel shows the fraction of time the VCPUs of a VM spent in the “co-stopped” state, waiting to be “co-started”. This gives an indication of the coscheduling overhead incurred by the VM. If this value is low, then any performance problems should be attributed to other issues, and not to the coscheduling of the VM’s virtual cpus.

In other words, check esxtop to determine if there are coscheduling problems.

Command line tips and tricks #1

· ·

Because I will be posting less in the upcoming weeks about problems I face at customer sites I will try to post some cool command-line tip or trick I discovered or picked up somewhere….

open ESX console ,via putty and type the following
vm-support -x
result: all the VMID’s also known as World ID’s,

And if you’re colleagues hardly ever clean up their snapshots:

find /vmfs/volumes -iname “*delta.vmdk”
result: every delta file gets listed, including the unregistered and/or orphaned snapshots ones!

VM Report

· ·

I had some spare time on my hands so I decided to add some useful stuff to the VM Reporting powershell script that was posted on this blog. This is what I ended up with, there’s still room for improvement like snapshot information and scsi controller info…


Get-VIServer -Server 192.168.1.1 -User admin -Password admin

$Report = @()

get-vm | % {
$vm = Get-View $_.ID
$ReportRow = "" | Select-Object VMName, Hostname, OS, IPAddress, VMState, TotalCPU, TotalMemory, MemoryUsage, TotalNics, ToolsStatus, ToolsVersion, MemoryLimit, MemoryReservation
$ReportRow.VMName = $vm.Name
$ReportRow.HostName = $vm.guest.hostname
$ReportRow.OS = $vm.guest.guestFullName
$ReportRow.IPAddress = $vm.guest.ipAddress
$ReportRow.VMState = $vm.summary.runtime.powerState
$ReportRow.TotalCPU = $vm.summary.config.numcpu
$ReportRow.TotalMemory = $vm.summary.config.memorysizemb
$ReportRow.MemoryUsage = $vm.summary.quickStats.guestMemoryUsage
$ReportRow.TotalNics = $vm.summary.config.numEthernetCards
$ReportRow.ToolsStatus = $vm.guest.toolsstatus
$ReportRow.ToolsVersion = $vm.config.tools.toolsversion
$ReportRow.MemoryLimit = $vm.resourceconfig.memoryallocation.limit
$ReportRow.MemoryReservation = $vm.resourceconfig.memoryallocation.reservation
$Report += $ReportRow
}
$Report | Export-CSV c:\export.csv

Scripted install

· ·

A while back I wrote a scripted install aka “cfg” file, and I just noticed I never published it. Check it out, it might be useful in one way or another. It also available for download here!

Especially changing the amount of active nics in a team can be useful(I’ve commented this out again, as of ESX 3.5 U3 this isn’t necessary anymore. Enabling vmotion via the vimsh command is still valid. [Read more…] about Scripted install

Deleting snapshots when everything else failse…

· ·

The common mis perception of the term “snapshot”, related to VMware, can cause huge problems. I’ve spend a lot of time the last years solving snapshot problems. For once and for all, a snapshot isn’t a static situation like a clone is. A snapshot can best be compared to a redo log, although technically it isn’t because it’s just a bitmap of disk sectors that changed. When you create a snapshot you only create a small “differences” file (*.delta.vmdk) which will contain all the differences until you delete or revert. Please remember reverting(go to) doesn’t delete the differences file! And this file can grow very fast depending on how many changes are made on the disk.

Another thing that people don’t know is the way “delete all” works, but I’ve already outlined that a while ago in a blog.

When you’ve got for instance a 10 levels deep nested snapshot tree with a very large last snapshot it would almost be impossible to press delete all because it will take up a lot of disk space. It would consume a lot of time doing a “delete” for every snapshot, and still it would always take up additional diskspace.

Another way to remove the snapshot is just by cloning the VM to another Datastore. This way you don’t need the extra disk space on the same datastore, and it might be a good moment to consider re-loadbalancing your lun’s again. [Read more…] about Deleting snapshots when everything else failse…