http://kb.vmware.com/kb/4223326

To resolve this issue:

1. Disable VMware High Availability (HA). Otherwise, the virtual machines might be forcibly powered down by step 2.
2. At the service console, issue the command:

   service mgmt-vmware restart

3. At the service console, runthe command:

   service vmware-vpxa restart

4. Reconnect the virtual machines to VirtualCenter.Attempt to re-enable VMware High Availability (VMware HA) within VirtualCenter. If this doesn’t work, this means that vpxa did not install properly.
5. At the service console, run the command:

   rpm -qa | grep vpxa

6. At the service console, run the “rpm -e” command on the rpm file that displayed in the previous command.

   rpm -e <rpm from step 6>

7. This uninstalls the agent from the ESX host. Reconnecting the host to VirtualCenter re-installs the agent.
8. Reconnect the virtual machines in the usual manner within VirtualCenter.
9. Re-enable VMware HA.

Symptoms:

• After one of the following, a Virtual Machine appears as being registered on two ESX Servers:
  • A VMotion fails to complete correctly or times out in VirtualCenter
  • A DRS issue where virtual machines are VMotioned automatically in quick succession
  • When a machine is powered on during VMware HA failover.
  • The Service Console on an ESX host is low on memory starving the vpxa process
• In VirtualCenter, you see the virtual machine as appearing on one ESX Server for a few seconds, then it seems to be on the other.
• The virtual machine may appear to jump back and forth among different ESX hosts.

I’m not going to copy/paste the solution cause the KB article will probably change over time, but it’s most definitely worth looking into… it does sound like something that can happen to all of us.

Be aware of this when implementing or troubleshooting HA it can cost you a lot of time.

All you need to do is head over to Train Signal and enter the following coupon code “YELLOWBRICKS” and “FREESHIP”. By buying a CBT you also support Yellow-Bricks.com! Both coupon codes are valid till the end of the month, just two weeks left…

Adding a dependency to the VirtualCenter service so that it waits for SQL Express remedies this.

To create a service dependency:

1. Click Start > Run.
2. Type services.msc and press Enter.
3. Locate the SQL Express instance for VirtualCenter. For example, SQL Server (SQLEXP_VIM).
4. Open the SQL Express instance and note the Service Name. For example, MSSQL$SQLEXP_VIM .
5. In the Run dialog, type Regedit.exe and press Enter. Browse to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vpxd .
6. Double-click the DependOnService key and add the Service name using the name identified in step 4.
7. Close Regedit .
8. Go back to the Services Panel and open the SQL Server properties.
9. On the Dependencies tab, verify the VMware VirtualCenter service is listed as depending on the SQL service instance.

As you can see the solution is fairly easy. Keep in mind that you need to be running the SQL Server locally on the vCenter server for this to work, especially for larger environments I wouldn’t advise running both on the same box. For SMB environments this should work just fine.

• ESX disconnects randomly from VirtualCenter
• ESX disconnects when performing VI Client tasks from VirtualCenter.
• Tasks randomly timeout after a long idle time
• “An error occurred communicating to the remote host” pops up.

The article refers to an issue with vCenter Update 3 in combination with firewalls using state-ful inspection. The problem occurs because of SOAP timeouts, and this behavior did not exist in VC 2.0.x or 2.5 GA, as they used a different mechanism to communicate with ESX. The official KB article hasn’t been released yet but a temporary workaround has been published by Richard. If you run into any of the before mentioned issues head over to Richard’s website and try out the workaround until the fix or official KB article is released.

• inconsistent portgroup names
• inconsistent portgroup provisioning on hosts
• multiple VM’s with diskfiles on more than one datastore
• multiple VMr’s with more than one connected NIC

RDA can do a lot more of course, so I suggest you head over to their website and download the demo and see if your VI3 environment is healthy or not. For those that already tested the previous release, 1.2 offers the following new capabilities:

New IP Knowledge Module – including the ability to detect and resolve configuration issues across a broader range of network issues. RDA can now identify routing and subnet misconfiguration and can determine if a guest VM network stack is operating correctly, as well as check for duplicate IP address usage in a common subnet.

Expanded drill down diagnostics – providing data to explain issues and guide IT towards a quick resolution – going beyond the basic identification of errors to save IT time and money.

Advanced item level notification – offering email notifications which now include full details on the exact changes that RDA has detected. The detailed notifications provide IT administrators with the latest information, delivered directly to their inbox.

Broader platform support – including support for VMware ESXi.

Increased scalability – offering significant performance improvements, including enhanced support for large scale datacenters of 100+ hosts.

If you’re not familiar with CBT’s check out this example:

I managed to get a discount for my readers, buy one of the virtualization related(VMware ESX, Server, Workstation etc) CBT’s and get a 25% reduction! All you need to do is head over to Train Signal and enter the following coupon code “YELLOWBRICKS”. And by buying a CBT you also support Yellow-Bricks.com!

This customer wanted to set permissions on a “HA-DRS” Cluster level. Each cluster will be administered by a different group of admins. These admins should not be allowed to do any administrative tasks on one of the other clusters in vCenter. Half of the setup worked, as in the admins could do certain tasks on the ESX hosts, but there was no way they could create VM’s.

I’ve browsed through my documentation but couldn’t find anything useful but luckily VI:OPS contained an excellent document on this topic: VI3 Roles and Permissions.

I did a copy and paste of the information that clarified the problem we were facing:

VMs appear in the inventory in two places: under the “Virtual Machines and Templates” view and the “Hosts and Clusters” view. This is also reflected in their privilege inheritance: VMs inherit privileges from both the containing host/cluster object as well as the containing VM/Template folder. Under Hosts and Clusters, possible containing objects include: folders, clusters, hosts, and resource pools. The two views and hierarchies become unified at the top level datacenter (or any folder that contains the datacenter)

Certain tasks require privileges on both sides of the hierarchy. For example, to create a VM, you need to have the “VM > Inventory > Create” privilege on a VM folder (in the VM view) as well as “Resource > Assign VM to Resource Pool” somewhere on an object in the Host view (folder, cluster, host, or resource pool). If you have a role which contains both these privileges, and you assign it at the datacenter level, it will propogate down both sides of the hierarchy. If, however, you want to limit its scope, then you’d need to apply it separately to individual subsections on each side of the hierarchy.

In other words, creating VM’s requires permissions on both levels “Datacenter” and “Cluster”.

I personally use the following tools:

• Health Check script by A.Mikkelsen → for a quick overview of the current situation and setup, small files and easy to carry around, runs from the Service Console.
• VMware Health Analyzer Appliance → A linux appliance that can connect to your VC/ESX and analyze log files. At this point in time it’s only available for VMware Employees or Partners with access to Partner Central.
• Powershell: Report into MS Word → Alan Renouf created this great reporting powershell scripts. It dumps info into a word document. (And i’ve heard he’s also working on a Visio export)
• Powershell: Health Check Script → Create an html report with datastore, cpu, memory and snapshot info… and more.
• RVTools → Gives a quick overview of current VM setup like snapshots, memory, cpu etc.
• Common sense → I hardly encounter really huge problems, mainly decreased availability cause of choices made during implementation / design phase without following VMware’s guidelines. Use common sense is the best advise in this case and read the best practice documents and VMware’s collection of pdf’s!
• And when there are some disturbing errors in one of the various log files you have the option to run it through one of the many toolkits we internally have.

I’m not using the following tools actively during engagements because of licensing but they can be very usefull in your enviroment:

• Replicate Datacenter Analyzer → Analyze your VI3 environment, I wrote an article a few weeks ago on RDA, click here
• Veeam Monitor → Monitor your VI3 environment including performance graphs etc.
• Veeam Reporter → A reporting tool, which will come in handy when documenting environments and comparing the current config to an old config.
• Vizioncore vFoglight → Might come in handy when doing analyses of trends and pinpointing resource contention.
• Tripwire Configcheck → Analyze the security of your VMware ESX environment. Check my blog post on Configcheck here.