Yellow Bricks

by Duncan Epping

vsphere cluster service

vCLS VMs not powering on, insufficient resources error

· ·

This week I had someone internally me asking about a situation where vCLS VMs (learn more about vSphere Cluster Service here.) were not powering on and an error was thrown stating “insufficient resources”. I had seen this issue before at some point and I knew it had something to do with the VM version and EVC. The details of the error messages seem to support that. The UI showed the following on the “power on virtual machine” task:

Insufficient resources

And then when you would look at the details of the error you could see the following:

The target host does not support the virtual machine's current hardware requirements.

Or you could see:

Feature 'MWAIT' was absent, but must be present.

So how do you solve this problem? First of all, this could be two different problems. We solved it the following way, please note that the second option was just us fiddling around to get the VMs provisioned and powered-on, and this is not the official VMware procedure to get it working. I have reported this to the engineers to figure out why this happens, and to get it fixed. There are two options, please use Option 1, as this is a requirement for EVC and the recommended method when you see the “MWAIT” error:

Option 1:

Verify if “Monitor/MWAIT” is set to Enabled in the BIOS. If it is set to Disabled, then this is why the power-on fails. vCLS has per-VM EVC enabled on the VM.

If you can’t enable Monitor/MWAIT, then below is the procedure for disabling “per VM EVC” for the provisioned vCLS VMs.

Option 2:

  1. Upgrade the VM’s “Compatibility” version to at least “VM version 14” (right-click the VM)
  2. Click on the VM, click on the Configure tab and click on “VMware EVC”
  3. Click on “Edit” and click on “Yes” when you are informed to not make changes to the VM
  4. Disable “EVC”
  5. Repeat for the other vCLS VMs

I want to mention cosmin.gq, as it seems the issue (and resolution with regards to disabling EVC) was also reported on that blog, and considering they reported it in October already it only seems fair to mention them here also.

How to login to the vCLS VMs!?

· ·

I was asked this question this week, how you can login to the vCLS VMs. Now before I share the video, I want to mention that I do not encourage people doing this, but as it is documented and supported I do want to provide a simple “how to” for how this works. If you want to login to the vCLS VM, maybe for troubleshooting if needed or for auditing, you can do so by SSH’ing first into your vCenter Server. When logged in to the vCenter Server you run the following command, which then returns the password, this will then allow you to login to the console of the vCLS VM. Again, I do not want to encourage you to do this. Either way, below you find the command for retrieving the password, and a short demo of me retrieving the password and logging in.

/usr/lib/vmware-wcp/decrypt_clustervm_pw.py

 

Did you know vSphere 7.0 Update 1 also has a Skyline Health Check for vSphere Clustering Services?

· ·

I did not know this, but yesterday the PM for vCLS reached out to me and informed me that we now have a Skyline Health Check as well for vSphere Clustering Services. The funny thing is that I actually requested this health check to be added after having a discussion on the topic of vCLS with the PM. Very impressive how fast the engineering team managed to include an additional health check for a brand new feature, this close to the release. I created a short demo, which shows you where you can find the vSphere Skyline Health option in the vSphere Client, and of course, it shows the vCLS Health Check being triggered. If you see the health check triggered, you can as mentioned enable retread mode and disable it again, this will provision a fresh set of vCLS VMs. How you do this you can find in this “considerations blog“, or simply watch the demo I shared here.

Demo Time: How to delete the vCLS VMs

· ·

As I have a bunch of questions about how you can delete the vSphere Cluster Service VMs (vCLS VMs) I figured I would create a quick demo. It is pretty straightforward, and it should only be used when people are doing some kind of full cluster maintenance. This demo shows you how to get the VMs deleted by leveraging a vCenter Server Level Advanced setting (config.vcls.clusters.domain-c<identifier>.enabled). I have also written a post that has a bunch of requirements, Q&A, and considerations for the vCLS VMs, if you are interested in that read it here.

Here’s the summary of how to delete the VMs: Go to your vCenter Server object, go to the configure tab, then go to “Advanced Settings”, add the key “config.vcls.clusters.domain-c<identifier>.enabled” and set it to false. The domain “c-number” for your cluster can be found in the URL when you click on the cluster. It should look something like the following, where the bold part is the important bit: https://vcsa-06.rainpole.com/ui/app/cluster;nav=h/urn:vmomi:ClusterComputeResource:domain-c22:4df0badc-1655-40de-9181-3422d6c36a3e/summary. If you want to recreate the VMs, simply set the value to “true” when the deletion task has completed.

Note, if you have a resource pool configuration, enabling “retreat mode” (disabling vCLS)) doesn’t impact resource pools in any shape or form, it just impacts DRS load balancing. Anyway, I hope you find the demo useful.