5.5

4 is the minimum number of hosts for VSAN if you ask me

Duncan Epping · Oct 24, 2013 ·

<Update 1-oct-15>Make sure to read this article also as it is based on Virtual SAN 6.1, which is the current latest version </update>

What is the minimum number of hosts for VSAN? This is one of those discussions which is difficult… I mean, what is the minimum number of hosts for vSphere HA for instance. If you ask anyone that question then most people will say: the minimum number for HA is 2. However, when you think about why you are using vSphere HA then you will realize pretty quick that the actual minimum number is 3.

Why is that? Well you can imagine that when you need to upgrade your hosts you also want some form of resiliency for your virtual machines. Guess what, if you have only 2 hosts and you are upgrading 1 of them and the other fails… Where would your virtual machines be restarted? I can give you the answer: nowhere. The only host you had left is in maintenance mode and undergoing an upgrade. So in that case you are … euhm screwed.

Now lets looks at VSAN, in order to comply to a “number of failures to tolerate = 1” policy you will need 3 hosts at a minimum at all times. Even if 1 host fails miserably then you can still access your data because with 3 hosts and 2 mirror copies and a witness you will still have > 50% of your copies available. But what happens when you place one of those hosts in maintenance mode?

Well I guess when both remaining hosts keep on functioning as expected then all VMs will just keep on running, however if one fails… then… then you have a challenge. So think about the number of hosts you want to have supporting your VSAN datastore!

I guess the question then arises, with this “number of failures to tolerate” policy, how many hosts do I need at a minimum? How many mirror copies will be created and how many witnesses? Also, how many hosts will I need when I want to take “maintenance mode” in to consideration?

Number of Failures	Mirror copies	Witnesses	Min. Hosts	Hosts + Maintenance
0	1	0	1 host	n/a
1	2	1	3 hosts	4 hosts
2	3	2	5 hosts	6 hosts
3	4	3	7 hosts	8 hosts

I hope that helps making the right decision…

How to configure the Virtual SAN observer for monitoring/troubleshooting

Duncan Epping · Oct 21, 2013 ·

There have been various blog posts on the topic of configuring the Virtual SAN observer on both Windows and Linux by Rawlinson Rivera and Erik Bussink. I like to keep things in a single location and document them for my own use so I figured I would do a write-up for yellow-bricks.com. First of all, what is the Virtual SAN / VSAN observer? One of our engineers (Christian Dickmann) published an internal blog on this topic and I believe it explains what it is / what it does best:

You will also find VSAN datastore as well as VM level performance statistics in the vSphere Web Client. If however you are the kind of guy who wants to really drill down on your VSAN performance in-depth, down to the physical disk layers, understand cache hit rates, reasons for observed latencies, etc. then the vSphere Web Client won’t satisfy your thirst in vSphere 5.5. That’s where the VSAN observer comes in.

So how do I enable it? Well I am a big fan of the vCenter Server Appliance so that will be my focus. Just a couple of short steps to get this up and running luckily:

Open an ssh session to your vCenter Server Appliance:
- ssh root@<name or ip of your vcva>
Open rvc using your root account and the vCenter name, in my case:
- rvc root@localhost
Now do a “cd” in to your vCenter object (you can do an “ls” so see what the names are of your objects on any level), and if you do tab it will be completed with your datacenter object:
- cd localhost/Datacenter/
Now do a “cd” again, the first object is “computers” and the second is your “cluster”, in my case that looks as follows:
- cd computers/VSANCluster/
Now you can start the VSAN observer using the following command:
- vsan.observer . –run-webserver –force
Now you can see the observer querying stats every 60 seconds, and as mentioned you can stop this by doing a <Ctrl>+<C>

Fairly straight forward right? You can now go to the observer console using:

http://<vcenter name or ip>:8010
The below is what it should look like (Thanks Rawlinson for the nice screenshot)

Now one thing that is important to realize is that everything is kept in memory until you stop the VSAN observer… So it will take up GBs after a couple of hours. This tool is intended for short term monitoring and troubleshooting. Now there are some other commands in RVC that might be useful. One of the commands I found useful was “vsan.resync_dashboard”. Basically it shows you what is happening in terms of mirror sync’ing. If you fail a host, you should see the sync happening here…

I also found “vsan.vm_object_info” very useful and interesting as it allows you to see the state of your objects. And for the geeks who do not prefer to see the pretty graphs the observer shows, take a look at “vsan.vm_perf_stats”.

Pretty pictures Friday, the VSAN edition…

Duncan Epping · Oct 11, 2013 ·

I’ve been working on a whole bunch of VSAN diagrams… I’ve shared a couple already via twitter and in the various blog articles, but I liked the following two very much and figured I would share it with you as well. Hoping they make sense to everyone. Also, if there are any other VSAN concepts you like to see visualized let me know and I will see what I can do.

First one shows how VSAN mirrors writes to two active mirror copies. Writes need to be acknowledged by all active copies, but note they are acknowledged as soon as they his the flash buffer! De-staging from buffer to HDD is a completely independent process, even between the two hosts this happens independently from each other.

The second diagram is all about striping. When the stripe width is defined using the VM Storage Policies then objects will grow in increments of 1MB. In other words: stripe segment 1 will go to esxi-02 and stripe segment 2 will go to esxi-03 and so on.

Just a little something I figured was nice to share on a Friday, some nice light pre-weekend/VMworld content 🙂

Virtual SAN news flash pt 1

Duncan Epping · Oct 3, 2013 ·

I had a couple of things I wanted to write about with regards to Virtual SAN which I felt weren’t beefy enough to dedicate a full article to so I figured I would combine a couple of news worthy items and create a Virtual SAN news flash article / series.

I was playing with Virtual SAN last week and I noticed something I hadn’t noticed yet… I was running vSphere with an Enterprise license and I added the Virtual SAN license for my cluster. After adding the Virtual SAN license all of a sudden I had the Distributed Switch capability on the cluster I had VSAN licensed. Now I am not sure what this will look like when VSAN will go GA, but for now those who want to test with VSAN and want to use the Distributed Switch you can. Use the Distributed Switch to guarantee bandwidth (leveraging Network IO Control) to Virtual SAN when combining different types of traffic like vMotion / Management / VM traffic on a 10GbE pair. I would highly recommend to start playing around with this and get experienced. Especially because vSphere HA traffic and VSAN traffic are combined on a single NIC pair and you do not want HA traffic to be impacted by replication traffic.
The Samsung SM1625 SSD series (eMLC) has been certified for Virtual SAN. It comes in sizes ranging between 100Gb and 800GB and can do up to 120k IOps random read… Nice to see the list of supported SSDs expanding, will try to get my hands on one of these at some point to see if I can do some testing.
Most people by now are aware of the challenges there were with the AHCI controller. I was just talking with one of the VSAN engineers who mentioned that they have managed to do a full root cause analysis and pinpoint the root of this problem. Currently there is a team working on solving it and things are looking good and hopefully soon a new driver will be released, when we do I will let you guys know as I realize that many use these controllers in their home-lab.

I created a folder on my VSAN datastore, but how do I delete it?

Duncan Epping · Sep 27, 2013 ·

I created a folder on my VSAN datastore using the vSphere Web Client, but when I wanted to deleted it I received this error message that that wasn’t possible. So how do I delete a VSAN folder when I don’t need it any longer? It is fairly straight forward, you open up an SSH session to your host and do the following:

change directory to /vmfs/volumes/vsanDatastore
run “ls -l” in /vmfs/volumes/vsanDatastore to identify the folder you want to delete
run “/usr/lib/vmware/osfs/bin/osfs-rmdir <name-of-the-folder>” to delete the folder

This is what it would look like:

/vmfs/volumes/vsan:5261f0c54e0c785a-81e199f6c9a23d73 # ls -lah total 6144 drwxr-xr-x 1 root root 512 Sep 27 03:17 . drwxr-xr-x 1 root root 512 Sep 27 03:17 .. drwxr-xr-t 1 root root 1.4K Sep 24 05:38 16254152-1469-2c18-3319-002590c0c254 drwxr-xr-t 1 root root 1.2K Sep 26 01:21 85803a52-6858-ded5-b40b-00259088447a lrwxr-xr-x 1 root root 36 Sep 27 03:17 ISO -> e64d1b52-1828-04ca-95a8-00259088447e lrwxr-xr-x 1 root root 36 Sep 27 03:17 TestVM -> ed31d351-a222-83bf-bb70-002590884480 drwxr-xr-t 1 root root 1.4K Sep 27 01:40 cc8ebe51-6881-7dc8-37f8-00259088447e drwxr-xr-t 1 root root 1.2K Sep 27 01:52 e64d1b52-1828-04ca-95a8-00259088447e drwxr-xr-t 1 root root 1.2K Jul 3 07:52 ed31d351-a222-83bf-bb70-002590884480 lrwxr-xr-x 1 root root 36 Sep 27 03:17 iso -> 16254152-1469-2c18-3319-002590c0c254 lrwxr-xr-x 1 root root 36 Sep 27 03:17 las-fg01-vc01.vmwcs.com -> cc8ebe51-6881-7dc8-37f8-00259088447e lrwxr-xr-x 1 root root 36 Sep 27 03:17 vmw-iol-01 -> 85803a52-6858-ded5-b40b-00259088447a

/vmfs/volumes/vsan:5261f0c54e0c785a-81e199f6c9a23d73 # /usr/lib/vmware/osfs/bin/osfs-rmdir vmw-iol-01

Deleting directory 85803a52-6858-ded5-b40b-00259088447a in container id 5261f0c54e0c785a81e199f6c9a23d73 backed by vsan

Be careful though, cause when you delete it guess what… it is gone! Yes not being able to delete it using the Web Client is a known issue, and on the roadmap to be fixed.