For those who want to start testing the beta of vSphere Virtual SAN in their lab with vSphere 5.5 I figured it would make sense to describe how I created my nested lab. (Do note that performance will be far from optimal) I am not going to describe how to install ESXi nested as there are a billion articles out there that describe how to do that.I suggest creating ESXi hosts with 3 disks each and a minimum of 5GB of memory per host:
- Disk 1 – 5GB
- Disk 2 – 20GB
- Disk 3 – 200GB
After you have installed ESXi and imported a vCenter Server Appliance (my preference for lab usage, so easy and fast to set up!) you add your ESXi hosts to your vCenter Server. Note to the vCenter Server NOT to a Cluster yet.
Login via SSH to each of your ESXi hosts and run the following commands:
- esxcli storage nmp satp rule add –satp VMW_SATP_LOCAL –device mpx.vmhba2:C0:T0:L0 –option “enable_local enable_ssd”
- esxcli storage nmp satp rule add –satp VMW_SATP_LOCAL –device mpx.vmhba3:C0:T0:L0 –option “enable_local”
- esxcli storage core claiming reclaim -d mpx.vmhba2:C0:T0:L0
- esxcli storage core claiming reclaim -d mpx.vmhba3:C0:T0:L0
These two commands ensure that the disks are seen as “local” disks by Virtual SAN and that the “20GB” disk is seen as an “SSD”, although it isn’t using an SSD. There is another option which might even be better, you can simply add a VMX setting to specify the disks are SSDs. Check William’s awesome blog post for the how to.
After running these two commands we will need to make sure the hosts are configured properly for Virtual SAN. First we will add them to our vCenter Server, but without adding them to a cluster! So just add them on a Datacenter level.
Now we will properly configure the host. We will need to create an additional VMkernel adapter, do this for each of the three hosts:
- Click on your host within the web client
- Click “Manage” -> “Networking” -> “VMkernel Adapters”
- Click the “Add host networking” icon
- Select “VMkernel Network Adapter”
- Select the correct vSwitch
- Provide an IP-Address and tick the “Virtual SAN” traffic tickbox!
- Next -> Next -> Finish
When this is configured for all three hosts, configure a cluster:
- Click your “Datacenter” object
- On the “Getting started” tab click “Create a cluster”
- Give the cluster a name and tick the “Turn On” tickbox for Virtual SAN
- Also enable HA and DRS if required
Now you should be able to move your hosts in to the cluster. With the Web Client for vSphere 5.5 you can simply drag and drop the hosts one by one in to the cluster. VSAN will now be automatically configured for these hosts… Nice right. When all configuration tasks are completed just click on your Cluster object and then “Manage” -> “Settings” -> “Virtual SAN”. Now you should see the amount of hosts part of the VSAN cluster, number of SSDs and number of data disks.
Now before you get started there is one thing you will need to do, and that is enable “VM Storage Policies” on your cluster / hosts. You can do this via the Web Client as follows:
- Click the “home” icon
- Click “VM Storage Policies”
- Click the little policy icon with the green checkmark, second from the left
- Select your cluster and click “Enable” and then close
Now note that you have enabled VM Storage Policies, there are no pre-defined policies. Yes there is a “default policy”, but you can only see that on the command line. For those interested just open up an SSH session and run the following command:
~ # esxcli vsan policy getdefault
Policy Class Policy Value
------------ --------------------------------------------------------
cluster (("hostFailuresToTolerate" i1) )
vdisk (("hostFailuresToTolerate" i1) )
vmnamespace (("hostFailuresToTolerate" i1) )
vmswap (("hostFailuresToTolerate" i1) ("forceProvisioning" i1))
~ #
Now this means that in the case of “hostFailuresToTolerate”, Virtual SAN can tolerate a 1 host failure before you potentially lose data. In other words, in a 3 node cluster you will have 2 copies of your data and a witness. Now if you would like to have N+2 resilience instead of N+1 it is fairly straight forward. You do the following:
- Click the “home” icon
- Click “VM Storage Policies”
- Click the “New VM Storage Policy” icon
- Give it a name, I used “N+2 resiliency” and click “Next”
- Click “Next” on Rule-Sets and select a vendor, which will be “vSan”
- Now click <add capability> and select “Number of failures to tolerate” and set it to 2 and click “Next”
- Click “Next” -> “Finish”
That is it for creating a new profile. Of course you can make these as complex as you want, their are various other options like “Number of disk stripes” and “Flash read cache reservation %”. For now I wouldn’t recommend tweaking these too much unless you absolutely understand the impact of changing these.
In order to use the profile you will go to an existing virtual machine and you right click it and do the following:
- Click “All vCenter Actions”
- Click “VM Storage Service Policies”
- Click “Manage VM Storage Policies”
- Select the appropriate policy on “Home VM Storage Policy” and do not forget to hit the “Apply to disks” button
- Click OK
Now the new policy will be applied to your virtual machine and its disk objects! Also while deploying a new virtual machine you can in the provisioning workflow immediately select the correct policy so that it is deployed in a correct fashion.
These are some of the basics for testing VSAN in a virtual environment… now register and get ready to play!
Excellent guide! however I am getting stuck at one spot. I’m doing this on Fusion, and have enabled the emulated SSD per Williams post. My 3 hosts are showing up with an SSD and NON-SSD disks. After adding the Vkernel and enabling VSAN for all 3 hosts, I created a cluster, and enabled VSAN. Added hosts to cluster fine, except when my VSAN datastore is created, there is no space as it’s not automatically adding the disks. If I manually try to add, it shows complete, but I still see no disks. I see the disks as not in use, and 0/3 eligible, but they should be eligible right?
Did you enter a license key!?
It’s always the simple things… 🙂 Thanks! that did the trick.
I am trying to set this up in a lab running off of Vmware Workstation 9. The ESX hosts see the storage as SSD and I can add them as cache but if I try to uses the disks for VSAN they show up as ineligible. Is there a good way to determine why an SSD disk is ineligible?
Just an update, it seems the minimum size had to be 5G for the SSD disk to be eligible.
Listing commands some mistake.
Instead of long names -device,-option to use short: – d and-o respectively.
Like: esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -d mpx.vmhba1:C0:T0:L0 -o “enable_local enable_ssd”
– -option works as well, somehow my theme doesn’t display a double –
fixed it by marking it as “code”
Victor, thank you for the short option. Much easier.
When doing this I keep on getting “Host cannot communicate with all other nodes in the VSAN enabled cluster” however all hosts can communicate and have a dedicated VSAN enabled VMK, they can all also see the VSAN datastore
¿Do you have a solution to the problem? I have this same problem with the U1
My problem was I was using nested ESXi nodes and they had been cloned, this had caused a unique ID to be the same that needed to be changed
Can you share the steps to resolve this issue ?
Have you found solution how to change this unique ID?
Just found this one
http://www.virtuallyghetto.com/2013/12/how-to-properly-clone-nested-esxi-vm.html
Hey Duncan,
Great post as always.
Are you able to license VSAN using the host evaluation license or can it only be run with a full license? I can’t get my lab licensed, I have three hosts on eval and I’m unable to enable VSAN.
Hi David,
I have the same error with a ”host not licensed for vsan” . did you find the solution???
Thanks
Thanks for the great guide. Got to the esxcli part, fixed the — after pasting the command, but getting this error:
# esxcli storage nmp satp rule add –satp VMW_SATP_LOCAL –device mpx.vmhba2:C
0:T0:L0 –option “enable_local enable_ssd”
Error: Invalid option enable_ssd”
Will keep trying.
Hi Bro,
Can you share me the how to install and configration of the Virtual SAN. help me any document step by step.. please give me any replay…
Hey Duncan,
I have already setup ESX 5 nested lab on single dell Poweredge 2900. I am now trying it with ESX 5.5. It won’t let me create nested esx as it doesn’t com up as an option. I am hearing this is because we need version 10 vm for esx vm on esx 5.5 host. Do I have any other options except putting esx 5 or 5.1 on standstandalone single dell poweredge server and then hosting esx 5.5 vms as needed for test lab?
hi Duncan,
happy new year 2014.
I try to used vsan into Esx5.5 nested lab with workstation 10.
I have a cluster with two hosts licenced with vcloud suite std.
Each host had 10Go SSD and 200Go non-ssd.
I used the vsan beta licence however my disk are still ineligible.
any idea ?
Regards
Eric
And you actually made sure those disks are seen as local disks? Please open a thread on the VMware VSAN Beta Community with screenshots of what you see. Makes troubleshooting a bit easier.
Here i have also created nested ESXi 5.5 and emulated SSD then tried to configure the vSAN by turning on the virtual SAN in cluster. I am not able to see the disk, but i can see them in ineligible mode.
What will be the reason, i want to do it in evaluation version.
Hi duncan! ,
Great Post. Im trying VSAN functionality but i have an error. I have a problem when creating ”disk groups” in VSAN. I have Vcenter 5.5 which is managing three ESX 5.5 host (1 SSD and 2 HDD), all newly installed with version 60 days. The error it gives is while trying to create Disk Groups saying: ” VSAN is not licensed in this Host” . Please your comments
Thanks for your answer!!.
When I follow all these instructions and set up the hosts, networks and invoke the commands required to emulate SSD on a drive in my host, I do see the SSD drive available in STORAGE view, but the vSAN “Create Disk Group” page doesn’t see the SSD.
It sees the other non-ssd.
Why does everything else show the SSD, and the vSAN “Disk Group” screen not see it?
great post !
On ESX 5.5 you can use the following commands for SSD:
esxcli storage nmp satp rule remove -s VMW_SATP_LOCAL -d naa.xxx
esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -d naa.xxx -o ‘enable_local enable_ssd’
esxcli storage core claiming reclaim -d naa.xxx
For the non SSD disks you can use:
esxcli storage nmp satp rule remove -s VMW_SATP_LOCAL -d naa.yyy
esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -d naa.yyy -o ‘enable_local’
esxcli storage core claiming reclaim -d naa.yyy
benefit is that you can configure SSD and NONSSD per disk ID instead
Hi Duncan,
I try to install Vsan in Vsphere 6.0,but Vsan is not successful,i have config 3 disk,1 disk to install esxi,1disk is ssd to cache and 1 disk HDD ( for 3 hosts). Vsan configuration can not see any disk in storage (Flash disks in use (0 of 0 eligible) and Data disks in use (0 of 0 eligible))
Thanks for you help!