Yellow Bricks

by Duncan Epping

VMware

Health status not showing..

· ·

I’ve seen this one on the VMTN forum a couple of times. When the Health Status isn’t showing you could do the following to fix it:

  1. Restart VirtualCenter service on the VC Server
  2. Restart mgmt-vmware service on the hosts that are affected (service mgmt-vmware restart)
  3. Restart vmware-vpxa on the hosts that are affected (service vmware-vpxa restart)

If the above did not fix the issue:

  1. Disconnect the affected Host from Inventory on VC
  2. Reconnect the affected Host from Inventory on VC

And if that doesn’t work this is also a possible solution:

  1. Restart the Pegasus service (service pegasus restart)

HA configuration and incompatible networks

· ·

There seems to be a lot of fuss about HA not being reconfigured when Update 2 is installed.

The error message that appears:

“HA Agent on <hostname> in cluster <clustername> in <datacenter> has an error Incompatible HA Networks: Host has network(s) that don’t exist on cluster members: <ip address>: Cluster has network(s) missing on host: <ip address>: Consider using Advanced Cluster Settings das.allowNetwork to control network usage”

Pre-Update 2 environments would except incompatible networks between hosts in a cluster and just install/reconfigure. As of Update 2 this clearly isn’t the case any more, there are a couple of misunderstandings that I want to clear up:

If you have redundant service consoles set up they don’t need to be on the same subnet. (Better said, they should not be on the same subnet because of a bug described in this blog!) But they do need to be the same on every host. In other words you can’t mix up subnets, this will not work:

Host A – Service Console – 192.168.1.10
Host B – Service Console – 10.0.0.10

In this case you will need to change the IP-Address of Host B. Or add an additional Service Console named “Service Console HA” to both and filter out the first. You can filter out the first by setting the Service Console used for HA to a specific portgroup:

das.allowNetwork0 “Service Console HA”

For more info read this topic and especially the reply that msevigny posted. The knowledge base article Marc points out to in his post is an internal one, as soon as it’s officially released I will let you guys know.

Update: HA Advanced Options

· ·

A while back I wrote down all the HA advanced options. With ESX 3.5 Update 2 VMware added a couple extra advanced options, this is the complete list:

  • das.failuredetectiontime – Amount of milliseconds, timeout time for isolation response action(with a default of 15000 milliseconds).
  • das.isolationaddress[x] – IP adres the ESX hosts uses for it’s heartbeat, where [x] = 0‐9. It will use the default gateway by default.
  • das.usedefaultisolationaddress – Value can be true or false and needs to be set in case the default gateway, which is the default isolation address shouldn’t be used for this purpose.
  • das.poweroffonisolation – Values are False or True, this is for setting the isolation response. Default a VM will be powered off.
  • das.vmMemoryMinMB – Higher values will reserve more space for failovers.
  • das.vmCpuMinMHz – Higher values will reserve more space for failovers.
  • das.defaultfailoverhost – Value is a hostname, this host will be the primary failover host.

The new ones:

  • das.failuredetectioninterval – Changes the heartbeat interval among HA hosts. By default, this occurs every second (1000 milliseconds).
  • das.allowVmotionNetworks – Allows a NIC that is used for VMotion networks to be
    considered for VMware HA usage. This permits a host to have only one NIC configured for management and VMotion combined.
  • das.allowNetwork[x] – Enables the use of port group names to control the networks used for VMware HA, where [x] = 0 – ?. You can set the value to be ʺService Console 2ʺ or ʺManagement Networkʺ to use (only) the networks associated with those port group names in the networking configuration.
  • das.isolationShutdownTimeout – Shutdown time out for the isolation response “Shutdown VM”, default is 300 seconds. In other words, if a VM isn’t shutdown clean when isolation response occured it’s being powered off after 300 seconds.

ESX 3.5 Update 2 available now!

· ·

Am I the first one to notice this? VMware just released Update 2 for ESX(i) 3.5 and a whole bunch of new patches!

So what’s new?

  • Windows Server 2008 support – Windows Server 2008 (Standard, Enterprise, and Datacenter editions) is supported as a guest operating system. With VMware’s memory overcommit technology and the reliability of ESX, virtual machine density can be maximized with this new guest operating system to achieve the highest degree of ROI. Guest operating system customizations and Microsoft Cluster Server (MSCS) are not supported with Windows Server 2008.
  • Enhanced VMotion Compatibility – Enhanced VMotion compatibility (EVC) simplifies VMotion compatibility issues across CPU generations by automatically configuring server CPUs with Intel FlexMigration or AMD-V Extended Migration technologies to be compatible with older servers. Once EVC is enabled for a cluster in the VirtualCenter inventory, all hosts in that cluster are configured to ensure CPU compatibility for VMotion. VirtualCenter will not permit the addition of hosts which cannot be automatically configured to be compatible with those already in the EVC cluster.
  • Storage VMotion – Storage VMotion from a FC/iSCSI datastore to another FC/iSCSI datastore is supported. This support is extended on ESX/ESXi 3.5 Update 1 as well.
  • VSS quiescing support – When creating quiesced snapshot of Windows Server 2003 guests, both filesystem and application quiescing are supported. With Windows Server 2008 guests, only filesystem quiescing is supported. For more information, see the Virtual Machine Backup Guide and the VMware Consolidated Backup 1.5 Release Notes.
  • Hot Virtual Extend Support – The ability to extend a virtual disk while virtual machines are running is provided. Hot extend is supported for vmfs flat virtual disks without snapshots opened in persistent mode.
  • 192 vCPUs per host – VMware now supports increasing the maximum number of vCPUs per host 192 given that the maximum number of Virtual Machines per host is 170 and that no more than 3 virtual floppy devices or virtual CDROM devices are configured on the host at any given time. This support is extended on ESX 3.5 Update 1 as well.

I really really like the VSS support for Snapshots, especially for VCB this is a great feature! And what about hot extending your harddisk, this makes a VMFS datastore as flexible as a RDM datastore!

For Hardware there are also a couple of really great additions:

  • 8Gb Fiber Channel HBAs – Support is available for 8Gb fiber channel HBAs. See the I/O Compatibility Guide for ESX Server 3.5 and ESX Server 3i for details.
  • SAS arrays – more configurations are supported.  See the Storage/SAN Compatibility Guide for ESX Server 3.5 and ESX Server 3i for details.
  • 10 GbE iSCSI initiator – iSCSI over a 10GbE interface is supported. This support is extended on ESX Server 3.5 Update 1, ESX Server version 3.5 Update 1 Embedded and ESX Server version 3.5 Update 1 Installable as well.
  • 10 GbE NFS support – NFS over a 10GbE interface is supported.
  • IBM System x3950 M2 – x3950 M2 in a 4-chassis configuration is supported, complete with hardware management capabilities through multi-node Intelligent Platform Management Interface (IPMI) driver and provider. Systems with up to 32 cores are fully supported.  Systems with more than 32 cores are supported experimentally.
  • IPMI OEM extension support – Execution of IPMI OEM extension commands is supported.
  • System health monitoring through CIM providers – More Common Information Model (CIM) providers are added for enhanced hardware monitoring, including storage management providers provided by QLogic and Emulex.  LSI MegaRAID providers are also included and are supported experimentally.
  • CIM SMASH/Server Management API – The VMware CIM SMASH/Server Management API provides an interface for developers building CIM-compliant applications to monitor and manage the health of systems.  CIM SMASH is now a fully supported interface on ESX Server 3.5 and VMware ESX Server 3i.
  • Display of system health information – More system health information is displayed in VI Client for both ESX Server 3.5 and VMware ESX Server 3i.
  • Remote CLI – Remote Command Line Interface (CLI) is now supported on ESX Server 3.5 as well as ESX Server 3i. See the Remote Command-Line Interface Installation and Reference Guide for more information.

One of the important thing in my opinion is the full support for the CIM Smash API! And iSCSI over a 10GBe interface, same goes for NFS! 8GB fibre and SAS arrays is a great extension.

  • VMware High Availability – VirtualCenter 2.5 update 2 adds full support for monitoring individual virtual machine failures based on VMware tools heartbeats. This release also extends support for clusters containing mixed combinations of ESX and ESXi hosts, and minimizes previous configuration dependencies on DNS.
  • VirtualCenter Alarms – VirtualCenter 2.5 Update 2 extends support for alarms on the overall health of the server by considering the health of each of the individual system components such as memory and power supplies. Alarms can now be configured to trigger when host health degrades.
  • Guided Consolidation – now provides administrators with the ability to filter the list of discovered systems by computer name, IP address, domain name or analyzing status. Administrators can also choose to explicitly add physical hosts for analysis, without waiting for systems to be auto-discovered by the Consolidation wizard. Systems can be manually added for analysis by specifying either a hostname or IP address. Multiple hostnames or IP addresses, separated by comma or semi-colon delimiters, may also be specified for analysis. Systems can also be manually added for analysis by specifying an IP address range or by importing a file containing a list of hostnames or IP addresses that need to be analyzed for consolidation. Guided Consolidation also allows administrators to override the provided recommendations and manually invoke the conversion wizard.
  • Live Cloning – VirtualCenter 2.5 Update 2 provides the ability of creating a clone of a powered-on virtual machine without any downtime to the running virtual machine. Therefore, administrators are no longer required to power off a virtual machine in order to create a clone of it.
  • Single Sign-On – You can now automatically authenticate to VirtualCenter using your current Windows domain login credentials on the local workstation, as long as the credentials are valid on the VirtualCenter server. This capability also supports logging in to Windows using Certificates and Smartcards. It can be used with the VI Client or the VI Remote CLI to ensure that scripts written using the VI Toolkits can take advantage of the Windows credentials of your current session to automatically connect to VirtualCenter.

One of the best new features described above in my opinion is the extension of Alarms! It’s awesome that VirtualCenter will report on hardware health! But what about that live cloning, that will definitely come in handy when troubleshooting a live production environment. Just copy the server, start it without the network attached and try to solve the problem!

DOWNLOAD it now:

ESX 3.5 Update 2
ESXi 3.5 installable Update 2
VirtualCenter 2.5 Update 2
VMware Consolidated Backup 1.5

File Level Recovery from within a VMDK backup

· ·

Today there was a cool article released on the Storage Nuts & Bolts blog. It’s about file level recovery from within a VMDK based on an NFS datastore. NFS is just mentioned cause this is the preferred way to go for the NetApp family, but it doesn’t matter NFS or VMFS the solution they came up with works for the both of them.

This option was recently discovered and further developed (Registry entries and Batch script) by one of the NetApp SE’s (Mike Arndt) and it’s very effective and free for those customers that already have a CIFS license with their NetApp arrays, which is a very large percentage. The other important factor is that Mike has made it a point-and-click process. Great job Mike!!!

As part of their VMware Disk Developer’s Kit, VMware provides a vmware-mount.exe utility that allows for mounting an existing VMDK on a Windows Driver letter. We’ll be using this utility as well to mount the VMDK as well as some Registry Entries and a Batch Script to further simplify the file recovery process.

I don’t to republish the complete article, so visit the blog and use the script and registry settings these guys created. Especially the first option with registry settings, which give you the possibility to mount a VMDK with just a “right click – mount” action is great!