Yellow Bricks

by Duncan Epping

VirtualCenter

vCenter tasks time-out or ESX disconnects?

· ·

I just received an email from a fellow consultant about a customer which had vCenter tasks time-out every once in a while. At times also ESX hosts got disconnected for no apparent reason at all. He discovered the following article by Richard Blythe aka VMware Wolf: ESX disconnects randomly or when doing VI client tasks from VC, task randomly timeout after a long idle time. Richard created a list of issues/errors that might be related to this issue:

  • ESX disconnects randomly from VirtualCenter
  • ESX disconnects when performing VI Client tasks from VirtualCenter.
  • Tasks randomly timeout after a long idle time
  • “An error occurred communicating to the remote host” pops up.

The article refers to an issue with vCenter Update 3 in combination with firewalls using state-ful inspection. The problem occurs because of SOAP timeouts, and this behavior did not exist in VC 2.0.x or 2.5 GA, as they used a different mechanism to communicate with ESX. The official KB article hasn’t been released yet but a temporary workaround has been published by Richard. If you run into any of the before mentioned issues head over to Richard’s website and try out the workaround until the fix or official KB article is released.

Replicate Datacenter Analyzer 1.2

· ·

I was just testing Replicate Datacenter Analyzer(RDA) 1.2 at a customer site. Well “testing” might not be the correct word in this case. RDA 1.2 discovered several things which are impossible to discover manually when you’ve got 5+ hosts. In this case there were over 50 hosts and RDA exposed the following:

  • inconsistent portgroup names
  • inconsistent portgroup provisioning on hosts
  • multiple VM’s with diskfiles on more than one datastore
  • multiple VMr’s with more than one connected NIC

RDA can do a lot more of course, so I suggest you head over to their website and download the demo and see if your VI3 environment is healthy or not. For those that already tested the previous release, 1.2 offers the following new capabilities:

New IP Knowledge Module – including the ability to detect and resolve configuration issues across a broader range of network issues. RDA can now identify routing and subnet misconfiguration and can determine if a guest VM network stack is operating correctly, as well as check for duplicate IP address usage in a common subnet.

Expanded drill down diagnostics – providing data to explain issues and guide IT towards a quick resolution – going beyond the basic identification of errors to save IT time and money.

Advanced item level notification – offering email notifications which now include full details on the exact changes that RDA has detected. The detailed notifications provide IT administrators with the latest information, delivered directly to their inbox.

Broader platform support – including support for VMware ESXi.

Increased scalability – offering significant performance improvements, including enhanced support for large scale datacenters of 100+ hosts.

Train Signal

· ·

Almost a year ago I wrote an article on a CBT by Train Signal. I really enjoyed watching the CBT and that says a lot coming from someone that never used to be a CBT fan. In the end the ease of learning when and wherever you want is what convinced me. Another reason to use a CBT instead of a classroom training is costs, which is most definitely a valid reason in these times of financial crisis.

If you’re not familiar with CBT’s check out this example:

I managed to get a discount for my readers, buy one of the virtualization related(VMware ESX, Server, Workstation etc) CBT’s and get a 25% reduction! All you need to do is head over to Train Signal and enter the following coupon code “YELLOWBRICKS”. And by buying a CBT you also support Yellow-Bricks.com!

Permissions and roles

· ·

I was just troubleshooting a problem with permissions and roles at a customer site within vCenter. For some weird reason we could not create a VM. I hardly ever use this functionality and if I do it’s mostly on a “Hosts & Clusters” level.

This customer wanted to set permissions on a “HA-DRS” Cluster level. Each cluster will be administered by a different group of admins. These admins should not be allowed to do any administrative tasks on one of the other clusters in vCenter. Half of the setup worked, as in the admins could do certain tasks on the ESX hosts, but there was no way they could create VM’s.

I’ve browsed through my documentation but couldn’t find anything useful but luckily VI:OPS contained an excellent document on this topic: VI3 Roles and Permissions.

I did a copy and paste of the information that clarified the problem we were facing:

VMs appear in the inventory in two places: under the “Virtual Machines and Templates” view and the “Hosts and Clusters” view. This is also reflected in their privilege inheritance: VMs inherit privileges from both the containing host/cluster object as well as the containing VM/Template folder. Under Hosts and Clusters, possible containing objects include: folders, clusters, hosts, and resource pools. The two views and hierarchies become unified at the top level datacenter (or any folder that contains the datacenter)

Certain tasks require privileges on both sides of the hierarchy. For example, to create a VM, you need to have the “VM > Inventory > Create” privilege on a VM folder (in the VM view) as well as “Resource > Assign VM to Resource Pool” somewhere on an object in the Host view (folder, cluster, host, or resource pool). If you have a role which contains both these privileges, and you assign it at the datacenter level, it will propogate down both sides of the hierarchy. If, however, you want to limit its scope, then you’d need to apply it separately to individual subsections on each side of the hierarchy.

In other words, creating VM’s requires permissions on both levels “Datacenter” and “Cluster”.

Health Check tools I use

· ·

A few days ago Scott Lowe asked me which tools I use to deliver a health check engagement. A health check is a standard VMware PSO engagement, a VMware Consultant will be on site to check the status of your environment and will draw up a report.

I personally use the following tools:

  • Health Check script by A.Mikkelsen → for a quick overview of the current situation and setup, small files and easy to carry around, runs from the Service Console.
  • VMware Health Analyzer Appliance → A linux appliance that can connect to your VC/ESX and analyze log files. At this point in time it’s only available for VMware Employees or Partners with access to Partner Central.
  • Powershell: Report into MS Word → Alan Renouf created this great reporting powershell scripts. It dumps info into a word document. (And i’ve heard he’s also working on a Visio export)
  • Powershell: Health Check Script → Create an html report with datastore, cpu, memory and snapshot info… and more.
  • RVTools → Gives a quick overview of current VM setup like snapshots, memory, cpu etc.
  • Common sense → I hardly encounter really huge problems, mainly decreased availability cause of choices made during implementation / design phase without following VMware’s guidelines. Use common sense is the best advise in this case and read the best practice documents and VMware’s collection of pdf’s!
  • And when there are some disturbing errors in one of the various log files you have the option to run it through one of the many toolkits we internally have.

I’m not using the following tools actively during engagements because of licensing but they can be very usefull in your enviroment: