Yellow Bricks

by Duncan Epping

cluster

Why does HA not power-on VMs after a full cluster shutdown?

· ·

I received this question and figured I would write a quick post about it, as it comes up occasionally. Why does vSphere HA no power-on VMs after a full cluster is brought back online after a full cluster shutdown? In this case, the customer had a power outage, so their hosts and all VMs were powered off, by an administrator cleanly, as a result of the backup power unit running out of power. Unfortunately, this happens more frequently than you would think.

When VMs are powered off by an administrator, or anyone/anything (PowerCLI etc) else which has permissions to power off VMs, then vCenter Server will mark these VMs as “cleanly powered off”. Next, also the state of the VMs is tracked by vSphere HA. So if a host is powered off, HA will know if the VM was powered on, or powered off at the time the host goes missing.

Now, when the host (or hosts) returns for duty, vSphere HA will of course verify what the last known state was of the cluster. It will read the list of all the VMs that were powered on, and it will restart those that were powered on and are configured for HA. It will also look at a VM property called “runtime.cleanPowerOff”, this property indicates if the VM was cleanly powered off by an Admin or a script, or if the VM was for instance powered off by vSphere HA itself. (PDL response etc.) Depending on the value of the property, the VM will, or will not be restarted.

Having said all of that, when you power off a VM manually through the UI, or via a script, then the VM will be marked as being “cleanly powered off”. This means that HA has no reason to restart it, as the powered-off VM is not the result of a host, network, or storage failure.

vCLS VMs not powering on, insufficient resources error

· ·

This week I had someone internally me asking about a situation where vCLS VMs (learn more about vSphere Cluster Service here.) were not powering on and an error was thrown stating “insufficient resources”. I had seen this issue before at some point and I knew it had something to do with the VM version and EVC. The details of the error messages seem to support that. The UI showed the following on the “power on virtual machine” task:

Insufficient resources

And then when you would look at the details of the error you could see the following:

The target host does not support the virtual machine's current hardware requirements.

Or you could see:

Feature 'MWAIT' was absent, but must be present.

So how do you solve this problem? First of all, this could be two different problems. We solved it the following way, please note that the second option was just us fiddling around to get the VMs provisioned and powered-on, and this is not the official VMware procedure to get it working. I have reported this to the engineers to figure out why this happens, and to get it fixed. There are two options, please use Option 1, as this is a requirement for EVC and the recommended method when you see the “MWAIT” error:

Option 1:

Verify if “Monitor/MWAIT” is set to Enabled in the BIOS. If it is set to Disabled, then this is why the power-on fails. vCLS has per-VM EVC enabled on the VM.

If you can’t enable Monitor/MWAIT, then below is the procedure for disabling “per VM EVC” for the provisioned vCLS VMs.

Option 2:

  1. Upgrade the VM’s “Compatibility” version to at least “VM version 14” (right-click the VM)
  2. Click on the VM, click on the Configure tab and click on “VMware EVC”
  3. Click on “Edit” and click on “Yes” when you are informed to not make changes to the VM
  4. Disable “EVC”
  5. Repeat for the other vCLS VMs

I want to mention cosmin.gq, as it seems the issue (and resolution with regards to disabling EVC) was also reported on that blog, and considering they reported it in October already it only seems fair to mention them here also.

VMworld Reveals: VMware Cluster Memory (OCTO2746BU)

· ·

At VMworld, various cool new technologies were previewed. In this series of articles, I will write about some of those previewed technologies. Unfortunately, I can’t cover them all as there are simply too many. This article is about VMware Cluster Memory, which was session OCTO2746BU. For those who want to see the session, you can find it here. I first learned about VMware Cluster Memory at our VMware internal R&D conference in May this year, and immediately got excited about it. Please note that this is a summary of a session which is discussing a Technical Preview, this feature/product may never be released, and this preview does not represent a commitment of any kind, and this feature (or it’s functionality) is subject to change. Now let’s dive into it, what is Cluster Memory?

Well, it is exactly what you would expect it to be. Providing the ability to create a pool of cross-host memory resources. Now in order to do this, the first problem that needs to be looked at is the network. As mentioned in the session, the ratio of network to memory latency has lowered significantly. In 1997 the ratio was 1000 roughly, right now it is below 10. Meaning that network latency has lowered from milliseconds to low microseconds. Today to reach these low microseconds latencies technologies like RDMA will need to be considered. This change is very important for the Cluster Memory feature being discussed. Also very important, is the fact that RDMA is affordable, and this means it will be coming to a data center near you soon. A huge difference compared to years ago.

[Read more…] about VMworld Reveals: VMware Cluster Memory (OCTO2746BU)

Books linked, buy paper Clustering Deep Dive get ebook for 2.95!

· ·

We just managed to link the paper and electronic version of the Clustering Deep Dive. This means that if you buy the paper book today, you can get the e-book at a discount. This was something a lot of you have asked for, so we pushed it through. Unfortunately, it did mean we had to re-upload the book to a different back-end system and “history” is lost, so those who already bought the paper version of the book, unfortunately, can’t get the same deal. If you are interested in getting both versions, go here. Or click below on the book, or one of the other books I recommend reading 🙂


You asked for it: vSphere 6.7 Clustering Deep Dive ebook, now available!

· ·

We knew when we released the paper version of the book that many would yell: What about an e-book? Although sales numbers of the Host Deep Dive and previous Clustering Deep Dive books have shown that by far most people prefer a printed copy, we decided to go ahead and create an ebook as well. It is not as simple unfortunately as simply uploading a PDF or an MS Word file. We had to spend evenings reformatting the book in an e-book authoring tool, compile it, review it, fix issues, compile again etc. Nevertheless, it is done!

So what we did is we just uploaded it to Amazon, and we made it available for 14,95 USD, or whatever that roughly converts to in your local currency in your local store. We also noticed there was a bundling option, so as soon as the ebook and the paper copy are linked you can buy the ebook alongside the paper copy for only 2,99 USD. (Linking the book may still take a couple of days, we’ve initiated the process with Amazon and are waiting for them to complete it.)

You wanted it, so go out and pick it up, right before the weekend! Also, note that we have both the ebook and the paper version available right now, we are working on linking the books, so you can get a nice deal for both versions. Also, I would highly recommend picking up the Host Deep Dive books as well, and while you are at it pick up the VDI guide, it is an excellent read! Amazon links are on the right side for your convenience.