The article contains an extensive description of the sympoms but the most important one is this one:

For any virtual machine that VMware HA is wrongly trying to start, it also generates thousands or tens of thousands of vmware-x.log files (one created every few minutes). Contents of each log file shows the virtual machine starting up then failing to start. Despite these logs, the virtual machines are actually pingable and running, and vm-support -x shows them as available running virtual machines on the host.

The resolution for the problem is the following:

1. Stop and start HA on the cluster. (This stops the flooding)
2. # find . -name ‘-*.log’ | xargs rm
   This commands removes all log files. Make sure you run this from /vmfs/volumes!

• ESX350-200901401-SG – PATCH – Security – KB 1006651 – Updates VMkernel VMX hostd. But most important fix that this patch containts definitely is: VMware ESX and ESXi 3.5 U3 I/O failure on SAN LUNs, and LUN queue is blocked indefinitely. For a full description of this issue, see http://kb.vmware.com/kb/1008130.
• ESX350-200901402-SG – PATCH – Security – KB 1006652 – Security Update to ESX Scripts
• ESX350-200901404-BG – PATCH – Critical – KB 1006654 – Updates VMware Tools
• ESX350-200901405-BG – PATCH – General – KB 1006655 – Updates VMware-esx-lnxcfg
• ESX350-200901406-BG – PATCH – General – KB 1006656 – Updates Kernel Source and VMNIX
• ESX350-200901407-BG – PATCH – General – KB 1006657 – Updates Pegasus
• ESX350-200901408-BG – PATCH – Critical – KB 1006658 – Updates SATA Drivers
• ESX350-200901409-SG – PATCH – Security -KB 1006659 – SNMP Security Update
• ESX350-200901410-SG – PATCH – Security – KB 1006660 – Security Update for libxml2

There’s one “patch” released for ESXi (installable and embedded) 3.5:

• ESXe350-200901401-O-SG – PATCH – Security – KB 1006661 – KB 1006662 – KB 1007058 – Firmware Update
  If you are using SRM in combination with ESX besure to read the KB1006661 cause this patch contains several fixes related to SRM.

There’s also a whole bunch of 3.0.x patches released, so if you’re still running 3.0.x besure to look into these new patches.

Processors

Newly supported processors are:

• Quad-Core AMD Opteron 8300 series and 2300 series processor families (collectively code-named Barcelona)
• Dual-core Intel Xeon 5200 series processors (code-named Wolfdale)

Drivers

Upgraded drivers are:

• igb driver version 1.0.0
• Broadcom 5722 tg3 driver version upgraded from 3.43.to 3.58b
• Emulex lpfc driver version upgraded to 7.3.2_vmw10

Guest Operating Systems

Newly supported guest operating systems are:

• Microsoft Windows Server 2003 R2 Data Center Edition
• Red Hat Enterprise Linux 5.1

Enhanced Patch and Upgrade Management

Significant improvements to the following patch management utilities:

• esxupdate Utility – Support for super-bundle and rollup features in esxupdate.
• Support for VMware Update Manager.

Service Console

Update for service console from RHEL3 U8 to RHEL3 U9.

Management Agents

Support for new versions of Management Agents:

• HP Insight Manager 8.1
• Dell OpenManage Server Administrator 5.4
• IBM Director 5.20 Update 1

Backup Software

Support for additional backup software from third-party vendors:

• EMC Networker Version 7.4 SP1
• IBM Tivoli Storage Manager Version 5.5

Quality Improvements

A significant number of issues have been fixed. Some of these issues have been described in Resolved Issues in this Release.

Get it while it’s stil hot and read the complete release notes here!

vimsh -n -e /hostsvc/maintenance_mode_enter

Backup every running vm via vcb in just one command:

for /f “tokens=2 delims=:” %%i in (’vcbvmname -h <virtualcenterserver> -u <user> -p <password> -s Powerstate:on ^| find “name:”‘) do cscript pre-command.wsf “c:\program files\vmware\vmware consolidated backup framework\” %%i fullvm

Enable VMotion from the command line:

vimsh -n -e “hostsvc/vmotion/vnic_set vmk0″

About a week ago and a month ago two blog’s appeared around this subject which clarifies the way ESX does vcpu scheduling as of 3.x. Read them for more in depth information. (1, 2)

So in short. Since 3.x VMware ESX uses “relaxed coscheduling”. And is this as relaxed as the name implies? Yes it is. And for a simple reason:

Idle vCPUs, vCPUs on which the guest is executing the idle loop, are detected by ESX and descheduled so that they free up a processor that can be productively utilized by some other active vCPU. Descheduled idle vCPU’s are considered as making progress in the skew detection algorithm. As a result, for co-scheduling decisions, idle vCPUs do not accumulate skew and are treated as if they were running . This optimization ensures that idle guest vCPUs don’t waste physical processor resources, which can instead be allocated to other VMs.

In other words VM’s with multiple vCPU’s don’t take up cycles anymore when these vCPU’s aren’t used by the OS. ESX checks the CPU’s for the idle proces loop and when it’s idle the CPU will be released and available for other vCPU’s. This also means that when you are using an application that will use all vCPU’s the same problems will still exists as they did in ESX 2.5, my advice don’t over do it. Only 1 vCPU is more than enough most of the times!

And what appeared to be a sidenote in the blog that deserves special attention is the following statement:

The %CSTP column in the CPU statistics panel shows the fraction of time the VCPUs of a VM spent in the “co-stopped” state, waiting to be “co-started”. This gives an indication of the coscheduling overhead incurred by the VM. If this value is low, then any performance problems should be attributed to other issues, and not to the coscheduling of the VM’s virtual cpus.

In other words, check esxtop to determine if there are coscheduling problems.

open ESX console ,via putty and type the following
vm-support -x
result: all the VMID’s also known as World ID’s,

And if you’re colleagues hardly ever clean up their snapshots:

find /vmfs/volumes -iname “*delta.vmdk”
result: every delta file gets listed, including the unregistered and/or orphaned snapshots ones!

Another thing that people don’t know is the way “delete all” works, but I’ve already outlined that a while ago in a blog.

When you’ve got for instance a 10 levels deep nested snapshot tree with a very large last snapshot it would almost be impossible to press delete all because it will take up a lot of disk space. It would consume a lot of time doing a “delete” for every snapshot, and still it would always take up additional diskspace.

Another way to remove the snapshot is just by cloning the VM to another Datastore. This way you don’t need the extra disk space on the same datastore, and it might be a good moment to consider re-loadbalancing your lun’s again. But for some weird reason this doesn’t always commit your snapshot. VMware just posted a nice KB article on how do the same using the vmkfstools command.

If a clone through the VMware Infrastructure 3 client did not commit the snapshots; you can try to export the disk with vmkfstools to recreate the virtual machine:

1. Run the following command to create a directory for the new disk:

[root@bs-tse-d06 RHEL5]# mkdir /vmfs/volumes/openfiler-iscsi/new_RHEL5

2. Run the following command to point vmkfstools at the last snapshot file:

[root@bs-tse-d06 RHEL5]# vmkfstools -i RHEL5-000001.vmdk /vmfs/volumes/openfiler
iscsi/new_RHEL5/new_RHEL5.vmdk

Destination disk format: VMFS thick
Cloning disk ‘RHEL5-000001.vmdk’…
Clone: 3% done
[root@bs-tse-d06 RHEL5]#

3. Recreate the machine. Select Use an existing virtual disk.