Yellow Bricks

by Duncan Epping

service console

Storage VMotion fails after Service Console IP change

· ·

As some of you know I spend a lot of time on the VMTN forums, helping people out and learning from other experts. Today someone posted about Storage VMotion not working after a Service Console IP change. It reminded me of a problem I faced a while ago. The solution wasn’t obvious but fairly easy, because the problem was solved with a slightly different approach I wrote it down:

Disconnect the ESX host from VirtualCenter
Stop the VMware VirtualCenter Server service
Remove the /etc/opt/vmware/vpxa/vpxa.cfg file from the ESX host that’s affected
Run this script on the database:
———–
UPDATE [VCDB].[dbo].[VPX_HOST]
SET [IP_ADDRESS] = ‘w.x.y.z’
WHERE [DNS_NAME] = ‘name of esx host as it is listed in the table’
———–
“w.x.y.z” above is the new ip address
Start the VMware VirtualCenter Server service
Add the host to the cluster again

Thanks goes out to BigRolTide for pointing me out to this solution. (In my case updating the database wasn’t necessarry)

Update: http://kb.vmware.com/kb/1006768

Partitioning ESX during installations

· ·

I see this a lot, and I also made the same mistake for a long time, because of the defaults of the ESX installer a lot of people tend to create a partition for /var/log. But keep in mind this means that everything else inside /var will belong to the root file system.

So what’s the problem? Well there’s a possibility that your / fills up with core dumps! Some of you might know this and some of you might not know this, but /var/core is the place were core dumps are stored for the Service Console. So when for some reason it starts generating dumps it can and probably will fill up your root filesystem very quick and thus cause the host to halt. Last time I witnessed this it was caused by a hardware vendor agent.

Virtual Infrastructure Management Assistant aka VIMA

· ·

I was just catching up and read this blog from Scott about managing ESXin a COS-less environment. The most interesting part of this blog is the following:

The subject of deployment is a key issue when we think about losing the Service Console. One approach to handling these issues is deploying physical machines; another would be to deploy virtual machines to handle these tasks. Partners could wrap up the agents that would typically be deployed in the Service Console as a virtual appliance, but then users could end up with numerous virtual appliances. What if VMware were to provide a virtual infrastructure management appliance? That’s what VIMA (Virtual Infrastructure Management Assistant) is.

VIMA is a virtual appliance packaged as OVF and is distributed, maintained, and supported by VMware. This is downloaded and installed by the customer according to their management procedures. This will be a well-known deployment environment that partners can rely upon being present. This will be a 64-bit Linux distribution with VMware Tools, VI Perl Toolkit, the Remote CLI (now known as the VI CLI), and a JRE already present. VIMA can be patched for updates, and it allows you to manage one or more VMware ESX hosts directly or through VirtualCenter. VIMA can enable agents to authenticate themselves, and VIMA will rotate its passwords on the hosts. Additionally, sample code and documentation will be available for programming applications to work in VIMA.

Anyone interested in VIMA can e-mail vima_request@vmware.com and request access to pre-GA versions of VIMA. VIMA is expected for general release in the fourth quarter of this year. All VIMA releases will work with both ESX and ESXi

Read the full article for more info, but this is again an exciting addition to the portfolio. Having an virtual appliance which can contain agents and communicate with your vCenter Server and your ESX(i) host is a real benefit. You can keep the number of agents down to a minimum with the same flexibility and usability.

Why I dislike agents in my Service Console

· ·

I’ve never been a huge fan of agents in the Service Console. Too many times I’ve seen hosts fail because of an agent that had a memory leak etc. Now it seems that running the HP IM agents causes your ESX 3.5 U2 to become unavailable after a certain amount of time.

The errors that appear:

0 Z root 8536 3673 0 79 0 – 0 nct> Aug05 ? 00:00:00 cimservera
0 Z root 8537 3673 0 79 0 – 0 nct> Aug05 ? 00:00:00 cimservera
0 Z root 8543 3673 0 78 0 – 0 nct> Aug05 ? 00:00:00 cimservera
0 Z root 32350 3673 0 79 0 – 0 nct> Aug06 ? 00:00:00 cimservera
0 Z root 32351 3673 0 79 0 – 0 nct> Aug06 ? 00:00:00 cimservera
0 Z root 32352 3673 0 79 0 – 0 nct> Aug06 ? 00:00:00 cimservera
0 Z root 32353 3673 0 78 0 – 0 nct> Aug06 ? 00:00:00 cimservera

HStrydom on the VMTN forum posted the following:

I am having the same issue. What happens after 17 days is that there are about 32000 of these processes. ESX has a max value of +- 32000 PID’s. Thus when all have been used up, one cannot SSH into the server, log in from the console or the ESX server disconnects from VC.

Also we have HP servers with the HP agents loaded. Our Dell servers does not have this problem.

Have a look at your cron log, /var/log/cron & cron.1. you might see that some of the job have not run. Also look in your /var/log/messages. There is a lot of login failures.

In other words, if you see the same thing happening call HP and let’s hope they release a fix soon! And in the meanwhile start thinking about ESXi, it’s problems like these that makes you think about why you even need a Service Console in the first place.

das.allowNetwork where and when

· ·

I’ve seen a lot misinterpretations of the new advanced HA option “das.allowNetwork”. What is this option for and when should I use it or shouldn’t is the most asked question.

So let’s start with a little history, back in the days “pre 3.5 U2” one could configure a cluster on multiple IP subnets without facing any problems, but this could lead into start up or execution problems. HA would configure without checking if the network on every host was compatible with the other, but in ESX 3.5 U2 this procedure changed. As of this version during the configuration of HA the host checks it’s network for compatability with the first configured HA host. So if the first host has the following configuration:
esx01 – 192.168.1.11 – 255.255.255.0.

And the second host has the following configuration:
esx02- 192.168.2.11 – 255.255.255.0

HA will fail with the following error:
“HA Agent on <hostname> in cluster <clustername> in <datacenter> has an error Incompatible HA Networks: Host has network(s) that don’t exist on cluster members: <ip address>: Cluster has network(s) missing on host: <ip address>: Consider using Advanced Cluster Settings das.allowNetwork to control network usage”

With the advance option “das.allowNetwork” you can rule out certain Service Consoles for HA usage. In other words, if you have several SC’s on each ESX host you can specify which SC should be used for HA. HA, as most of you probably know, uses the SC network for it’s heartbeat.

A couple of examples when to use das.allowNetwork and when not to use it:

  1. So what should you do to get things running when you’ve used different subnets for each host?
    Well that’s an easy one… get all your Service Consoles on the same subnet. Even if’s a routable vlan, HA will just calculate the network and if the values mismatch than configuring HA will fail!
  2. So what if I’ve got multiple SC’s and some are on different subnets but there’s at least one on each host on the same subnet with the same portgroup name?
    Here’s where das.allowNetwork comes in to place, set this option in the “Advanced Option” section of the HA tab and specify which SC should be used for HA. The ones that aren’t specified will not be used.

There have been numerous people asking if this “network compatibility” check could be disabled. At this moment the answer is no and there’s now workaround at this moment. When using das.allowNetwork specify the first as “das.allowNetwork0” the second as “das.allowNetwork1” etc.

More info can be found here:

KB1006541

KB1006606

Resource Management PDF