Yellow Bricks

by Duncan Epping

kb

Startup update: Runecast 2.0

· ·

Last week I was briefed by Runecast (together with Cormac) on the new version, Runecast 2.0, which was released/announced today. I always enjoy talking to Stan as every time we talk they have something new which surprises me, or he tells me about something cool on the roadmap. For those who did not read my previous articles, Runecast is a company which focusses on analyzing VMware environments and assess the environment on potential issues. These issues could be anything ranging from configuration issues, driver/firmware issue, to security issues. It reminds me very much of what we have with vSAN which is the health check. The big difference though is that this solution includes many more checks and doesn’t just focus on vSAN but on many different parts of the stack. Just to give you an idea, today Runecast can analyze your vSphere environment up to vSphere 6.7 and can also analyze vSAN and NSX-V. The cool thing is that it also does this “offline”, they have an appliance and regular updates (rules and features) and this means that even in a dark site this would work.

A lot of Runecast’s customers are either in the financial space or government space. I guess this is also why their focus for the 2.0 version was primarily on PCI-DSS. With over 200 technical checks, which map against PCI-DSS requirements, they (as Runecast told me) have by far the largest collection of requirements in an automated analyzer (for VMware) in the industry. Definitely, a smart enhancement, if you are not interested in PCI-DSS, you can simply disable the whole check and it will never show up in your interface. You can also, if only a limited number of clusters should be validated, filter out certain results.

The 20 version of Runecast also comes with a lot of updates around the appliance, now I consider these “internals” as for most customers it is not relevant in terms of the value it offers, but it is important to know from a security perspective I guess.

This version also introduces a historical perspective. Meaning that starting with Runecast 2.0 the historical information of checks is stored. This will allow you to see some form of trending when it comes to the different checks/validations. You could for instance now track if you do updates and maintenance if the number of potential issues is going down. You could also task someone with validating the reported issues and fixing those when or where possible. This should over time improve the availability, reliability, and security of your environment.

Last but not least the UI has been fully overhauled. They redesigned it just to make it easier to read and understand. Also, a couple of dashboards were added, which makes sense… a new release means new dashboards!

If you happen to go to VMworld, make sure to stop by their booth and have a look, I think you will find it interesting. Or simply read the Runecast blog, and download the appliance and try it out.

Startup update: Runecast

· ·

A while ago I introduced Runecast on my blog. I have known these guys for a while and this week I had to pleasure to be briefed on their new release: Runecast 1.7. The big ticket item in this release for sure it the vSAN Support. You may ask yourself why you would need Runecast when you have things like the health check and the “online” health check, well it seems that Runecast’s implementation covers more detail. Anyway, what is Runecast? As a company they refer to themselves as the knowledge automation experts, and I think that is a fair statement.

Runecast has developed an appliance which can be connected to one or multiple vCenter Server instances. After linking these you can “scan” the environment and Runecast will tell you about the risks. Not just from a security perspective, but it will also assess logs, configuration and even best practices. Your whole environment will be assessed in a report will be provided in a simple HTML-5 interface, or in the Web Client or the vSphere H5 client even. I said “simple”, but the information provided and the detail is far from simple… When I say simple I refer to their user interface. It is slick, and very easy to use.

Since I discussed Runecast last they added some additional features, like for instance a VRO plugin, full rest API, improved log search, Web Client and H5 client plugins but more importantly for many government agencies: DISA STIG compliancy checks. Yes, Runecast can check your environment against DISA STIG and report on any potential issues. Nice right?

This new release, version 1.7, now brings vSAN support. It also includes a new dashboard widget, which provides faster insights in how your environment is behaving. For vSAN in particular they didn’t only include KB article checks, but also implemented all best practices from the Design and Sizing guide, Network Design guide and the Stretched Cluster white paper. And they even hinted about adding best practices which are listed in the Essential vSAN book Cormac and I wrote, how cool is that? What is also nice is that their appliance is supported with vSAN 5.x and 6.x, and requires no direct access to the internet. You can simply download the appliance and install, and then update with the latest dataset by downloading an ISO.

Oh and before I forget, of course they also provide all the guidance and info needed around Spectre/Meltdown. Where normally their trial is limited, they actually do provide ALL info needed for Spectre/Meltdown as they realized that this is very valuable to customers and felt they could not hold this back.

For the Runecast blog on the 1.7 release go here.

Alert: vSphere 5.5 U1 and NFS issue!

· ·

Some had already reported on this on twitter and the various blog posts but I had to wait until I received the green light from our KB/GSS team. An issue has been discovered with vSphere 5.5 Update 1 that is related to loss of connection of NFS based datastores. (NFS volumes include VSA datastores.)

*** Patch released, read more about it here ***

This is a serious issue, as it results in an APD of the datastore meaning that the virtual machines will not be able to do any IO to the datastore at the time of the APD. This by itself can result in BSOD’s for Windows guests and filesystems becoming read only for Linux guests.

Witnessed log entries can include:

2014-04-01T14:35:08.074Z: [APDCorrelator] 9413898746us: [vob.storage.apd.start] Device or filesystem with identifier [12345678-abcdefg0] has entered the All Paths Down state.
2014-04-01T14:35:08.075Z: [APDCorrelator] 9414268686us: [esx.problem.storage.apd.start] Device or filesystem with identifier [12345678-abcdefg0] has entered the All Paths Down state.
2014-04-01T14:36:55.274Z: No correlator for vob.vmfs.nfs.server.disconnect
2014-04-01T14:36:55.274Z: [vmfsCorrelator] 9521467867us: [esx.problem.vmfs.nfs.server.disconnect] 192.168.1.1/NFS-DS1 12345678-abcdefg0-0000-000000000000 NFS-DS1
2014-04-01T14:37:28.081Z: [APDCorrelator] 9553899639us: [vob.storage.apd.timeout] Device or filesystem with identifier [12345678-abcdefg0] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.
2014-04-01T14:37:28.081Z: [APDCorrelator] 9554275221us: [esx.problem.storage.apd.timeout] Device or filesystem with identifier [12345678-abcdefg0] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.

If you are hitting these issues than VMware recommends reverting back to vSphere 5.5. Please monitor the following KB closely for more details and hopefully a fix in the near future: http://kb.vmware.com/kb/2076392

 

CloudPhysics KB Advisor, how cool is that?

· ·

Just imagine, you have 3-8 hosts – an EMC array – Dell hardware – some FibreChannel cards – Specific versions of firmware – Specific versions of ESXi and vCenter… How do you know what works and what does not? Well, you go to kb.vmware.com and you do a search and try to figure out what applies to you and what does not. In this depicted environment of only 3-8 hosts that should be simple? Well with thousands of KB articles I can ensure you that it is not… Just imagine now that you have 2 arrays and 2 clusters of 8 hosts… Or you add iSCSI to the mix? Yes it gets extremely overly complicated really really quick, in fact I would say it is impossible to figure out what does and does not apply to your environment. How do you solve that?

Well you don’t solve that yourself, it requires a big database and an analytics engine behind it… Big data platform even. Luckily though, the smart folks of CloudPhysics have solved it for you. Sign up, download the appliance and let them do the work for you… It doesn’t get any easier than that if you ask me. Some more details can be found in the press release.

I knew the CPhy guys were working on this, surprises me that no one else has done this so far to be honest. What an elegant / simple / awesome solution! Thanks CloudPhysics for making my life once again a whole lot easier.

Setup cannot create vCenter Server Directory Services instance. Error 28038

· ·

While doing a new install of VMware vCenter Server I ran into the following error:

Setup cannot create vCenter Server Directory Services instance. Error 28038

This error is caused by the fact that the “Network Service” does not have enough permissions on the root of the drive you’re installing vCenter on. The solution is pretty straight forward and has been described in the KB article.

  1. Right-click the root drive and click Properties.
  2. Click the Security tab.
  3. Under Group and user names, click Add.
  4. Enter Network Service and click OK.
  5. Check Allow for the Read permission for the Network Service account in the Permissions for Administrators pane.
  6. Click Apply and OK.

If this does not resolve the issue look into the following KB articles: 1015887 , 1013822.