Yellow Bricks

by Duncan Epping

Permissions and roles

· ·

I was just troubleshooting a problem with permissions and roles at a customer site within vCenter. For some weird reason we could not create a VM. I hardly ever use this functionality and if I do it’s mostly on a “Hosts & Clusters” level.

This customer wanted to set permissions on a “HA-DRS” Cluster level. Each cluster will be administered by a different group of admins. These admins should not be allowed to do any administrative tasks on one of the other clusters in vCenter. Half of the setup worked, as in the admins could do certain tasks on the ESX hosts, but there was no way they could create VM’s.

I’ve browsed through my documentation but couldn’t find anything useful but luckily VI:OPS contained an excellent document on this topic: VI3 Roles and Permissions.

I did a copy and paste of the information that clarified the problem we were facing:

VMs appear in the inventory in two places: under the “Virtual Machines and Templates” view and the “Hosts and Clusters” view. This is also reflected in their privilege inheritance: VMs inherit privileges from both the containing host/cluster object as well as the containing VM/Template folder. Under Hosts and Clusters, possible containing objects include: folders, clusters, hosts, and resource pools. The two views and hierarchies become unified at the top level datacenter (or any folder that contains the datacenter)

Certain tasks require privileges on both sides of the hierarchy. For example, to create a VM, you need to have the “VM > Inventory > Create” privilege on a VM folder (in the VM view) as well as “Resource > Assign VM to Resource Pool” somewhere on an object in the Host view (folder, cluster, host, or resource pool). If you have a role which contains both these privileges, and you assign it at the datacenter level, it will propogate down both sides of the hierarchy. If, however, you want to limit its scope, then you’d need to apply it separately to individual subsections on each side of the hierarchy.

In other words, creating VM’s requires permissions on both levels “Datacenter” and “Cluster”.

Site Recovery Manager and MSCS

· ·

When reading several SRM docs I was wondering if Microsoft Clustering was supported or not. I knew that in version 1.0 it wasn’t supported. When reading the Release Notes I noticed the following:

Full Support for RDM devices
SRM now provides full support for virtual machines that use raw disk mapping (RDM) devices. This enables support of several new configurations, including Microsoft Cluster Server. (Virtual machine templates cannot use RDM devices.)

Microsoft Clustering Services is supported as of Update 1. But you will need to keep in mind when creating your Recovery Plan that all nodes of the cluster will belong to the same Protection Group and can possibly  be started up or shutdown at the same time….. I haven’t configured SRM in combination with MSCS so far, if any of you has any tips/tricks let me know.

Site Recovery Manager is not about installing… Part II

· ·

I’ve been playing around with Site Recovery Manager these last couple of days. Installing it was really easy and same goes for the basic configuration. I already wrote a blog about this topic a month ago or so but now I’ve experienced it myself. Most of the time during a Site Recover Manager project will be spent during the Plan & Design phase and writing documentation. I will just give you one example why. The following was taken from the SRM Course material:

Datastore Group
Replicated datastores containing the complete set of virtual machines that you want to protect with SRM

Protection Group
A group of virtual machines that are failed over together during test and recovery

For those who don’t know, there’s a one on one mapping between Datastore Groups and Protection Groups. So in other words, once you’ve mapped a Datastore Group to a Protection Group there’s no way of changing it without having to recreate the Protection Group.

I think a picture says more than a 1000 words so I stole this one from the Evaluator Guide to clarify the relationship between datastore, Datastore Groups and Protection Groups:

Notice that there are multiple datastores in Datastore Group 2 because VM4 has disks in both datastores. So these datastores are joined into one Datastore Group. This Datastore Group will have a one to one relationship with a Protection Group. Keep in mind, this is really important: a Protection Group contains VM’s that are failed over together during test and recovery.

If you’ve got VM’s with multiple disks on multiple datastores with no logic in which disk is placed on which datastore you could and probably will end up with all datastores being member of the same Datastore Group. Being member of the same Datastore Group means being part of the same Protection Group. Being part of the same Protection Group will result in a less granular fail-over. It’s all or nothing in this case and I can imagine most companies would like to have some sort of tiering model in place or even better fail over services one at a time. (This doesn’t mean by the way that if you create multiple Protection Group that you can’t fail over everything at the same time, they can all be joined in a Recovery Plan)

Some might think that you would be able to randomly add disks to datastores after you finished configuring. This clearly isn’t the case. If you add a disk to a protected(!) VM the Datastore Group will be recomputed. In our situation this meant that all VM’s in the “Medium Priority” Protection Group were moved over to the “High Priority” Protection Group. This was caused by the fact that we added a disk to a “Medium Priority” VM and placed it on a “High Priority” datastore. As you can imagine this also causes your Recovery Plans to end up with a “warning”, you will need to reconfigure the moved VM’s before you can fail them over as part of your “High Priority” datastore. (Which probably wasn’t the desired strategy…)

When I was searching the internet for information on SRM I stumbled upon this article on the VMware Uptime blog by Lee Dilworth. I’ve taken the following from the “What we’ve learnt” post, which confirms what we’ve seen the last couple of days:

Datastore Group computation is triggered by the following events:

  • Existing VM is deleted or unregistered
  • VM is storage vmotioned to a different datastore
  • New disk is attached to VM on a datastore previously not used by the VM
  • New datastore is created
  • Existing datastore is expanded

So in other words, moving VM’s from one Datastore to another or creating a new disk on a different Datastore can cause problems because the Datastore Group computation will be re-run. Not only do you need to take virtual disk placement in consideration when configuring SRM, you will also need to be really careful when moving virtual disks. Documentation, Design and Planning is key here.

I would suggest documenting current disk placement before you even start implementing SRM, and given the results you might need to move disks around before you start with SRM. Make sure to check your documentation and design before randomly adding virtual disks when SRM has been implemented. Documenting your current disk placement can be done easily with the script that Hugo created this week by the way, and I would suggest to regularly create reports and save them.

Expect some more SRM stuff coming up over the next couple of weeks.

Tripwire Configcheck

· ·

When I published my article on tools/scripts I use during a VMware Healthcheck I received a couple of emails on Tripwire’s Configcheck. I’ve been on a holiday for a couple of weeks so it took me a bit longer than usual to check out the product.

Configcheck can be downloaded for free. Configcheck is a Java Application so you will need to install JRE. Installing JRE can be a bit of a pain sometimes on a server so this is one of the reasons for me that will make it hard to actively use Configcheck at customer sites. (This depends on the customers policy.) Installing the product is fairly easy though:

  1. Download Java JRE.
  2. Download the file configcheck.zip to a Windows machine that has Java Runtime Environment (JRE) version 1.5, or higher.
  3. Unzip the configcheck.zip file

That’s it, fairly easy. Now you can run “configcheck.cmd” to check the specified ESX host on security issues. Once the check is complete you can click the test results to view remediation steps. The test results will look like this:

As you can see, 37 Passed and 40 Failed. Not really surprising considering the fact that I ran this against a newly build ESX 3.5 U3 host. No modifications whatsoever. Clicking the test results didn’t work on my test servers because of the lack of an internet connection. Unfortunately it’s also not possible to export the data in this version. It’s free and Tripwire’s Enterprise edition does give you this capability, if you need export and a whole lot more check it out. You can find a data-sheet with a comparison between Configcheck and enterprise here.

Luckily Tripwire also provides the remediation steps in pdf form. For instance the remediation steps for 1.2.2 “Verify the log files to keep is equal to 10”:

Description: 
This test determines if virtual machines are configured to keep 10 log files when the recommended log rotate size of 100KB is exceeded. Virtual machines log activity in their respective vmware.log files. If growth of these log files is not limited, it is possible for virtual machines to cause a denial of service on the ESX Server by filling up the VMFS volume. There are two options for preventing virtual machines from flooding the hard disk of the host: size-based log file rotation or disabling logging for the virtual machine. This policy checks for size-based log file rotation because disabling logging altogether limits troubleshooting options.

Remediation:
To remediate failure of this policy test, configure the virtual machine to keep 10 log files when the recommended log rotate size of 100KB is exceeded. Configuring the virtual machine to keep 10 log files when the recommended log rotate size of 100KB is exceeded:

Login to the VirtualCenter or use the VI Client to connect directly to the ESX Server hosting the improperly configured virtual machine.

  1. Power off the virtual machine if needed.
  2. Right click the virtual machine and click Edit Settings.
  3. Select the Options tab.
  4. Select Advanced > General, and click the Configuration Parameters button.
  5. Look for a row with log.keepOld and set the value to 10.
  6. If the row does not exist, then click the Add Row button.
  7. In the Name field type log.keepOld.
  8. In the Value field type the value to 10.
  9. Click OK to close the Configuration Parameters dialog.
  10. Click OK to close the Virtual Machine Properties dialog.
As you can see, the description and remediation explain why and what to do in a fairly extensive manner. Which is great cause not does this make solving the “problem” really easy, Tripwire’s Configcheck also educates the SysAdmin!

Cool Tool Update: RVTools 2.2

· ·

I’ve wrote about this tool several times so most of you must have tested it by now and are probably actively using it: RVTools. Rob just notified me that he uploaded a brand new version of his tool. The following have been added to version 2.2:

  • New vDatastore tab. The “vDatastore” tab displays for each datastore the name, connectivity status, file system type, number of virtual machines on the datastore, total capacity in MB’s, free capacity in MB’s, multiple host access indication and the url.
  • Your custom defined fields are now visible on most of the tab-pages
  • New menu option “export data to csv file”
  • New “upgrade policy” field on vTools tab-page
  • New “Sync time with host” field on vTools tab-page
  • The field “OS” which is displayed on most of the tab-pages now displays the name of the guest OS according to the VMware Tools. In previous versions we used the configuration value. The vTools tab displays both “OS” fields.

Here’s a screenshot of the new tab “vDatastore”:

If you’ve never used RVTools before besure to check it out, it’s worth it. And if you are already using it download it and upgrade!