Source

• Symptom: NIC Teaming load balancing properties do not take effect with global vSwitch configuration settings.
• Resolution: Override the load balancing order at the port group level, under Policy Exceptions, select the checkbox and choose the proper load balancing from the dropdown menu. Please note that this workaround only works until the next reboot of ESX.

If we refer to the current version which is ESX 3.5 u3, the maximum number of Vcpu per ESX server is only 128 per ESX Servers. Personally, I think the number of Vcpu per ESX servers is too minimal. Imagine if we do run a servers with 4 or 8 physical CPU sockets and we consolidate 40 : 1 Physical server in our virtualization environment, we will hit to the bottleneck on maximum numbers of Vcpu per ESX servers but not due to the CPU consumption

Reading this short section one might think, why reply it makes sense? No, it doesn’t make sense at all:

• The current limit isn’t 128, it’s 192 vCPU’s.
  So even with a 40:1 ratio and all VMs provisioned with 4 vCPU’s you wouldn’t hit this limit. Read the max config guide, it’s the bible for virtualization consultants.
• But even more important: co-scheduling and over provisioning will impact performance. With most VM’s running 2 or even 4 vCPU’s scheduling will be almost impossible even with the relaxed co-scheduling techniques ESX is using these days. In other words, please don’t use multi vCPU VMs as a standard, you can read more on c0-scheduling here.

The author asked VMware to bump up the max number of vCPU’s. Now for a VDI environment this can and will be useful I think. Again if you are hitting the number with a 16 core machine, you might need to reconsider your provisioning strategy.

I expect the number to go up… especially after watching Stephen Herrod’s keynote at VMworld Europe 2009.

Both configurations are supported by VMware and each has their own pros and cons:

1. Requirements: 2 physical nics, vlan’s and vlan trunking:
   • vSwitch0: 2 Physical nics(vmnic0 & vmnic2)
   • 2 Portgroups(Service Console & VMkernel)
   • Service Console active on vmnic0 and standby on vmnic2
   • VMkernel active on vmnic2 and standby on vmnic0

   Each portgroup has a vlan ID assigned and runs dedicated on its own physical NIC, only in the case of a fault it’s switched over to the standby NIC. It will return to the original NIC when the physical NIC is up and running again. This is achieved by setting “Failback” to Yes!

   Pros: Only need 2 NICs in total for the Service Console and VMkernel, especially useful in Blade environments. Simple setup.

   Cons: If the connection is dropped several times it will cause the NIC to fail-over each time which can cause HA to kick in eventually. You will need to set the Failure Detection Time (das.failuredetectiontime) to 60 seconds opposed to the default of 15 seconds. (According to VMware best practices.)

2. Requirements: 3 physical NICs, vlan’s and vlan trunking:
   • vSwitch0 – 1 Physical NIC(vmnic0) – 1 Portgroup(Service Console)
   • vSwitch1 – 2 Physical NICs(vmnic1 & vmnic3)
   • 2 Portgroups (VMkernel & Secondary Service Console)

   The primary Service Console runs dedicated on a physical NIC, vmnic0, with a vlan assigned on either the physical switch port or the portgroup. (I would prefer portgroup for consistency).

   The second vSwitch, vSwitch1, will run the VMkernel active on vmnic1 and standby on vmnic3. The secondary Service Console will be active on vmnic3. For the secondary Service Console I would prefer to set the vmnic1 to “unused”, this way you are 100% sure that the Service Console will run only on vmnic0 and vmnic3.

   Pros: a lower “failure detection time”(das.failuredetectiontime) can be defined this is because of the fact that the service console is already active. Here the Failure Detection Time can be set to 20 seconds, further no Spanning Tree problems will occur as the setup contains two service consoles, and subsequently 2 MAC addresses. (This is also the reason why I would set vmnic1 to unused for this portgroup)

   Cons: Need to set extra isolation addresses(das.isolationaddress), and secondary Service Console preferably in a different subnet.

Thanks goes out to mr Tom Howarth for a sanity/grammar check.

• ESX disconnects randomly from VirtualCenter
• ESX disconnects when performing VI Client tasks from VirtualCenter.
• Tasks randomly timeout after a long idle time
• “An error occurred communicating to the remote host” pops up.

The article refers to an issue with vCenter Update 3 in combination with firewalls using state-ful inspection. The problem occurs because of SOAP timeouts, and this behavior did not exist in VC 2.0.x or 2.5 GA, as they used a different mechanism to communicate with ESX. The official KB article hasn’t been released yet but a temporary workaround has been published by Richard. If you run into any of the before mentioned issues head over to Richard’s website and try out the workaround until the fix or official KB article is released.

• ESX350-200901401-SG – PATCH – Security – KB 1006651 – Updates VMkernel VMX hostd. But most important fix that this patch containts definitely is: VMware ESX and ESXi 3.5 U3 I/O failure on SAN LUNs, and LUN queue is blocked indefinitely. For a full description of this issue, see http://kb.vmware.com/kb/1008130.
• ESX350-200901402-SG – PATCH – Security – KB 1006652 – Security Update to ESX Scripts
• ESX350-200901404-BG – PATCH – Critical – KB 1006654 – Updates VMware Tools
• ESX350-200901405-BG – PATCH – General – KB 1006655 – Updates VMware-esx-lnxcfg
• ESX350-200901406-BG – PATCH – General – KB 1006656 – Updates Kernel Source and VMNIX
• ESX350-200901407-BG – PATCH – General – KB 1006657 – Updates Pegasus
• ESX350-200901408-BG – PATCH – Critical – KB 1006658 – Updates SATA Drivers
• ESX350-200901409-SG – PATCH – Security -KB 1006659 – SNMP Security Update
• ESX350-200901410-SG – PATCH – Security – KB 1006660 – Security Update for libxml2

There’s one “patch” released for ESXi (installable and embedded) 3.5:

• ESXe350-200901401-O-SG – PATCH – Security – KB 1006661 – KB 1006662 – KB 1007058 – Firmware Update
  If you are using SRM in combination with ESX besure to read the KB1006661 cause this patch contains several fixes related to SRM.

There’s also a whole bunch of 3.0.x patches released, so if you’re still running 3.0.x besure to look into these new patches.

vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0x8fdbd0
vmkernel: 8:18:59:58.640 cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)

During the conversation I knew I’d seen this problem before. But the problem that I witnessed was related to a high threshold value in Vizioncore vFoglight. I knew it was possible to change the setting:

1. Open vCenter, and click a specific host
2. Click on the “Configurations” tab
3. Click on Advanced Settings, VMFS3
4. Change the value of “VMFS3.MaxHeapSizeMB”

The default value is 16MB, this allows for a maximum of 4TB of open vmdk’s on a single host. The max setting is 128MB which allows for a maximum of 32TB of open vmdk’s on a single host. Keep this in mind when designing your environment.

Keep in mind that this is ESX 3.5 only, you can’t change the heap size in ESX 3.0.x.

As most of you know I work for VMware which means I receive a huge amount of email and documents with excellent technical info. Some I can’t blog about but some I can. Last week I received this VMware HA implementations notes document by Seva Semouchin. Seva is a Technical Account Manager based in Germany!

This document deals about VMware HA clusters and all the advanced options you have with the use cases for these options. And that’s what makes this document very useful, the use cases! So everyone dealing with HA clusters should definitely read this document. Thanks Seva for letting me distribute it, and thanks for a great document!

This document deals about VMware HA clusters and all the advanced options you have with the use cases for these options. And that’s what makes this document very useful, the use cases! So everyone dealing with HA clusters should definitely read this document. Thanks Seva for letting me distribute it, and thanks for a great document!

1. Configuration of host IP address is inconsistent on host : address resolved to Host misconfigured. IP address of not found on local interfaces
2. cmd addnode failed for primary node: Internal AAM Error – agent could not start

So the first error(1.) was reported by esxhost01 and the second(2.) by esxhost02.

Let’s start with esxhost01.

So this customer had a VMotion and Management portgroup on two seperate vSwitches. This error seems to indicate that during the configuration HA is using the VMotion portgroup. These hosts have been added to VC with the management portgroup IP(IP+Name also in dns). So how do I make sure that HA isn’t using the VMotion network for HA, it’s easy go to your cluster and open up the advanced options for HA and add the following key with the value false:

• das.allowVmotionNetworks=false

In other words, don’t use the VMotion network for the HA heartbeat. The weird thing in this case is that it shouldn’t use the VMotion network by default so there seems to be a glitch here…

So now for the second problem.

The HA(AAM) agent could not start. So just to make sure that the USB key wasn’t corrupt the key was recreated. But still this error occurred. As some of you might now, that if you want to use HA with a disk less server you will need to create a userworld swap on the SAN. (Read this KB for more info on that one…) So just to make sure that the swap wasn’t causing this problem the directory was cleaned out and and HA was reconfigured. When the directory was emptied the HA agent installed without any problem at all…

• When reinstalling ESXi or when strange HA errors occur clean up the userworld swap!

Thanks goes out to Remco for providing me with the additional details!

How does a primary and / or secondary get selected?

• The first 5 hosts that join the VMware HA cluster are automatically selected as “primary nodes”
• All the others are automatically selected as “secondary nodes”
• When you do a reconfigure for HA the primary nodes and secondary nodes are selected again, this is random

What’s up with these primaries and secondaries?

• Primary nodes hold cluster settings and all node states which are synced between primaries
• Secondary nodes send their state info(resource occupation) to the primary nodes
• Nodes send heartbeats to each other, primary nodes send heartbeats to primary nodes only and secondary also only to primary. And they do this every second. (Which is a changeable value: das.failuredetectioninterval)

So what if a primary node fails, will a secondary be promoted?

• No, there will only be a new primary appointed when the failed one is removed from the cluster. A secondary will be promoted to primary at random.

But what if all my primary nodes fail?

• This is an unaddressed issue, that’s the reason why you can only account for 4 host failures within a cluster! There needs to be at least one primary!

So when does the gateway come in play?

• Actually the gateway, which is the default “isolation address”, will only be used when an isolation has occurred. So when the AAM client thinks it’s isolated it will check the isolation addresses.

So if anyone has a question just drop it here and I’ll try to answer it and update the above list…