Yellow Bricks

by Duncan Epping

Want to play around with Kubernetes? Try MicroK8s!

Leave a Comment

Two weeks ago I wanted to play around with Kubernetes for a day or two. I found this training course internally at VMware that allowed me to go through some labs. I asked around if anyone had some tips on getting Kubernetes up and running fast. I couldn’t be bothered with creating a multi node kubernetes cluster. I wanted to play around with some of the commands and YAML files. I tried Atomic as suggested by the lab manual, but there were way too many steps involved to install/configure Kubernetes if you ask me. Next option would be some version hosted in a cloud of choice, but I didn’t want to incur the cost. After digging around I stumbled on MicroK8s. It sounded easy, so I figured I would give it a try. When it comes to Linux my preference is Ubuntu/Debian, it is just what I am most familiar with, and as MicroK8s comes from Canonical I figured I would give it a try. As Kelsey Hightower suggested on twitter yesterday (which triggered this article), it is just one command away:

I downloaded the latest Ubuntu Server ISO and I created a VMware Fusion VM. I stepped through the installation wizard of Ubuntu Server and then noticed it already provided the option even to install “microk8s”, I selected the package, and some additional packages I figured I would need, and clicked done. Literally within minutes, I had a fresh single node Kubernetes configuration, which for me worked straight out of the box!

After it is done configuring, click reboot and login. I created an alias for kubectl, as I didn’t want to type “microk8s.kubectl” every time or install a different version:

sudo snap alias microk8s.kubectl kubectl

I also enabled the Kubernetes dashboard from the get-go, which can be done by running the command “microk8s.enable dashboard“. There are a whole bunch of articles out there that can take you through the steps of deploying your first container, making it highly available by specifying the number of instances so I am not going to do that. I don’t want to pretend to be an expert, as I am far from that. Also, check the MicroK8s documentation, it is pretty decent. My colleague Myles Gray has a very good tutorial on why containers, very good read which I also recommend for people who just want to know a bit more about it like myself.

Mixing versions of ESXi in the same vSphere / vSAN cluster?

2 Comments

I have seen this question being asked a couple of times in the past months, and to be honest I was a bit surprised people asked about this. Various customers were wondering if it is supported to mix versions of ESXi in the same vSphere or vSAN Cluster? I can be short whether this is support or not, yes it is. Would I recommend it? No, I would not!

Why not? Well mainly for operational reasons, it just makes life more complex. Just think about a troubleshooting scenario, you now need to remember which version you are running on which host and understand the “known issues” for each version. Also, for vSAN things are even more complex as you could have “components” running on a different version of ESXi. On top of that, it could even be the case that a certain command or esxcli namespace is not available on a particular version of ESXi.

Another concern is when doing upgrades or updates, you need to take the current version into account when updating, or more importantly when upgrading! Also, remember that firmware/driver combination may be different for a particular version of vSphere/vSAN as well, this could also make life more complex and definitely increases the chances of making mistakes!

Is this documented anywhere? Well, there are two KBs which hint at this, for vSAN specifically I have requested our team to document this in an upcoming document. The KBs that hint at this are below.

Free Spanish VMware technology ebook available now!

3 Comments

A couple of months ago I was asked if I wanted to write a foreword for an upcoming ebook. I have done this various times, but this one was particularly interesting. Why? Well, there are 3 good reasons:

  1. This book is written by 14 community experts, many of which I have met over the past years.
  2. It is a free, but sponsored, ebook!
  3. All sponsor proceeds will go to charity.

The first book I wrote was also a book with multiple authors, it was only a handful of people and that was painful enough as it is. An insane amount of coordination is involved usually and I have a lot of respect for these guys, 14 people writing a single book is not easy.

On top of that, these guys decided to cover multiple VMware technologies, ranging from NSX to VDI to vSphere etc. Very cool if you ask me. Oh, and before I forget… They have already managed to collect over 25.000 Euro for charity. Great job guys, what an achievement. I am not going to say much more, just download the book (if you read/speak Spanish)! Thanks for letting me part of this.

https://www.vmwareporvexperts.org/

vSphere HA virtual machine failed to failover error on VMs in a partitioned cluster

Leave a Comment

I received two questions this week around partition scenarios where after the failure has been lifted some VMs display the error message “vSphere HA virtual machine failed to failover”. The question that then arises is: why did HA try to restart it, and why did it fail? Well, first of all, this is an error that in most cases you can safely ignore. There’s a KB on the topic which gives a bit of detail to be found here, but let me explain to also in a bit more depth.

In a partition scenario, each partition will have its own master node. If there is no form of communication (datastore / network) possible, what the HA master will do is it will list all the VMs that are currently not running within that partition. It will also want to try to restart those VMs. A partition is extremely uncommon in normal environments but may happen in a stretched cluster. In a stretched cluster when a partition happens a datastore only belongs to 1 location. The VMs which appear to be missing typically are running in the other location, as typically the other location will have access to that particular datastore. Although the master has listed these VMs as “missing and need to restart” it will not be able to do this. Why? It doesn’t have access to the datastore itself, or when it has access to the datastore the files are locked as the VMs are still running. As a result, this will, unfortunately, be reported as a failed failover. Even though the VM was still running and there was no need for a failover. So if you hit this during certain failure scenarios, and the VMs were running as you expected, you can safely ignore this error.

vSAN Stretched Cluster failure scenarios and component votes

1 Comment

I was at a customer last week and had an interesting question about the vSAN voting mechanism. This customer had a stretched cluster and used RAID-5 within each location to protect the data on top of replicating across locations. During certain failure scenarios unexpectedly the data remained available, of course, it is great that you have higher availability than expected, but why did this happen? What this customer tested was powering off the Witness (which is deemed as a site failure) and next powered of 2 hosts in 1 location, which exceeds the “failures to tolerate” in a single location. You would expect, based on all documentation so far, that the data would be unavailable. Well for some VMs this was the case, but for others, that was not the case. Why is this? Well, it is all about the vote count in this case. Look at the below diagram and the number of votes for each component first.

In the above scenario if the Witness (W) fails we have 4 votes less. Out of a total of 13 that is not a problem. If two additional hosts fail, this is most likely still not a problem, even though you are exceeding the provided “failures to tolerate”. However, if by any chance Host1 is one of those failed hosts then you would lose quorum. Host1 has a component with 2 votes. So if host1 has failed and the witness has failed and host2 for instance, you have now lost 7 out of 13 votes. This means quorum is lost. Please note that that single component with 2 votes is random. For a different VM/Object it could be that the component which is placed on host6 or host7 has 2 votes.

Another thing to point out, if host5-8 all would fail the data is still available. However, if then host3 and host4 would fail the object would become unavailable. Even though you still would have quorum across locations, you have now also exceeded the specified “failures to tolerate” within the location. This is also something that will be taken in to account.

I hope that helps.