Unmounting datastore fails due to vSphere HA?

On the VMware Community Forums someone reported he was having issues unmounting datastores when vSphere HA was enabled. Internally I contacted various folks to see what was going on. The error that this customer was hitting was the following:

The vSphere HA agent on host '<hostname>' failed to quiesce file activity on datastore '/vmfs/volumes/<volume id>'

After some emails back and forth with Support and Engineering (awesome to work with such a team by the way!) the issue was discovered and it seems that in two separate instances issues were resolved that had to do with unmounting of datastores. Keith Farkas explained on the forums how you can figure out if you are hitting those exact problems or not and in which release they are fixed, but at I realize those kind of threads are difficult to find I figured I would post it here for future reference:

You can determine if you are encountering this issue by searching the VC log files. Find the task corresponding to the unmount request, and see if the follow error message is logged during the task’s execution (Fixed in 5.1 U1a) :

2012-09-28T11:24:08.707Z [7F7728EC5700 error 'DAS'] [VpxdDas::SetDatastoreDisabledForHACallback] Failed to disable datastore /vmfs/volumes/505dc9ea-2f199983-764a-001b7858bddc on host [vim.HostSystem:host-30,10.112.28.11]: N3Csi5Fault16NotAuthenticated9ExceptionE(csi.fault.NotAuthenticated)

While we are on the subject, I’ll also mention that there is another know issue in VC 5.0 that was fixed in VC5.0U1 (the fix is in VC 5.1 too). This issue related to unmounting a force mounted VMFS datastore. You can determine whether you are hitting this error by again checking the VC log files. If you see an error message such as the following with VC 5.0, then you may be hitting this problem. A work around, like above, is to disable HA while you unmount the datastore.

2011-11-29T07:20:17.108-08:00 [04528 info 'Default' opID=19B77743-00000A40] [VpxLRO] -- ERROR task-396 -- host-384 -- vim.host.StorageSystem.unmountForceMountedVmfsVolume: vim.fault.PlatformConfigFault:

CPU Affinity and vSphere HA

On the VMware Community Forums someone asked today if CPU Affinity and vSphere HA worked in conjunction and if it was supported. To be fair I never tested this scenario, but I was certain it was supported and would work… Never hurts to  validate though before you answer a question like that. I connected to my lab and disabled a VM for DRS so I could enable CPU affinity. I pinned the CPUs down to core 0 and 1 as shown in the screenshot below:

cpu affinity

After pinning the vCPUs to a set of logical CPUs I powered on the VM. The result was, as expected, a “Protected” virtual machine as shown in the screenshot below.

HA protection

But would it get restarted if anything happened to the host? Yes it would, and I tested this of course. I switched the server off which was running this virtual machine and within a minute vSphere HA restarted the virtual machine on one of the other hosts in the cluster. So there you have it, CPU Affinity and vSphere HA work fine.

PS: Would I ever recommend using CPU Affinity? No I would not!

vCenter Single Sign On aka SSO, what do I recommend?

I have had various people asking me over the last 9 months what I would recommend when it comes to SSO. Would I use a multi-site configuration, maybe even an HA configuration or would I go for the Basic configuration? What about when I have multiple vCenter Server instances, would I share the SSO instance between these or deploy multiple SSO instances? All very valid questions I would say. I have kept my head low intentionally the last year to be honest, but after reading this excellent blog post by Josh Odgers where he posted an awesome  architectural decision flow chart I figured it was time voice my opinion. Just look at this impression of the flow chart (for full resolution visit Josh’s website):

Complex? Yes I agree, probably too complex for most people. Difficult to digest, and that is not due to Josh’s diagramming skills. SSO has various deployment models (multi site, HA, basic), and then there is the option to deploy it centralized or localized as well. On top of that there is also the option to protect it using Heartbeat. Now you can probably understand why the flow diagram ended up looking complex. Many different options but what makes sense?

Justin King already mentioned this in his blog series on SSO (part 1, 2, 3, 4) as a suggestion, but lets drive it home! Although it might seem like it defeats the purpose I would recommend the following in almost every single scenario one can imagine: Basic SSO deployment, local to vCenter Server instance. Really, the KISS principle applies here. (Keep It Simple SSO!) Why do I recommend this? Well for the following simple reasons:

  • SSO in HA mode does not make sense as clustering the SSO database is not supported, so although you just deployed an HA solution you still end up with a single point of failure!
  • You could separate SSO from vCenter, but why would you create a dependency on network connection between the vCenter instance and the SSO instance? It is asking for trouble.
  • A centralized SSO instance sounds like it make sense, but the problem here is that it requires all connecting vCenter instances to be on the same version. Yes indeed, this complicates your operational model. So go localized for now.

So is there a valid reason to deviate from this? Yes there is and it is called Linked Mode. Linked Mode “requires” SSO to be deployed in a “multi-site” configuration, this is probably one of the few reasons I would not follow the KISS principle when there is a requirement for linked-mode… personally I never use Linked Mode though, I find it confusing.

So there you have it, KISS!

(ab)using vSphere advanced settings?

Almost on a daily basis I get questions from colleagues and customers about specific advanced settings. Somehow they spotted a vSphere advanced setting and wonder if they should set it. They go on a hunt to figure out what it is this specific vSphere advanced setting does and typically find a description on a random website that makes it sound like it is a good idea to configure it. I even had someone asking if I could give a list of all optimized values for the advanced kernel parameters recently. My answer was short and maybe a bit blunt, but I think my it was clear:

I hope the sign above makes it clear you should not randomly set advanced settings. Some of you will laugh and say “well that is obvious” while others probably will scratch their head and open their vSphere client and check the advanced settings section. I know I discuss advanced settings every once in a while, but you should only apply these settings when:

  1. You have a requirement to implement this advanced setting, do not tweak them “just because you can”. An example would be in a stretched cluster you set “disk.terminateVMOnPDLDefault” because of the infrastructure implemented.
  2. The advanced setting solves a problem in your environment (and preferably in that case see 3)
  3. When recommend by VMware Global Support Services

If you have implemented an advanced setting, document it and with every update or upgrade validate it is still applicable to that specific version or not. (If you are aspiring to be a VCDX, this is key.) If it no longer applies, remove revert to default!

How to register a Storage Provider using the vSphere Web Client

I needed to register a Storage Provider for vSphere Storage APIs for Storage Awareness (VASA) today. I force myself to use the vSphere Web Client and it had me looking for this option for a couple of minutes. It actually was the second time this week I had to do this, so I figured if I need to search for it there will probably be more people hitting the same issue. So where can you register those VASA Storage Provider’s in the Web Client?

  • In your vSphere Web Client “home screen” click “vCenter”
  • Now in the “Inventory Lists” click “vCenter Servers”
  • Select your “vCenter Server” in the left pane
  • Click the “Manage” tab in the right pane
  • Click “Storage Provider” in the right pane
  • Click on the “green plus”
  • Fill out your details and hit “OK” just like the example below (VNX, block storage)
    registering a Storage Provider

I personally find this not very intuitive and would prefer to have it in the Rules and Profiles section of the Web Client, and when I do configure it… I should be able to configure it for all vCenter Server instances just by select all or individual vCenter Servers. Do you agree? I am going to push for this within VMware, so if you don’t agree, please speak up and let me know why :-).