It seems Virtual SAN is on fire… I’ve been getting emails about VSAN configurations on a daily basis from partners, VMware field and customers. More and more customers seem to be looking at deploying Virtual SAN with a high number of disks and this week I ran in to something that I wanted to share with you. I thought it was common knowledge, but I guess I was wrong. If you are planning to have more than 8 disks / devices in your Virtual SAN host then it is good to know that you will either need a second disk controller (Check the HCL!) or use a SAS Expander! This design / sizing principle is listed in the VSAN Ready Node PDF, and should also make its way to the Design / Sizing Guide soon. The following is mentioned in the VSAN Ready Node PDF:
No more than 8 disks (SSD + HDD) supported behind a single controller. In case you need to add more than 8 disks, please add an additional controller.
I met up with the product manager for the compatibility guide and asked him which SAS Expanders are supported, he mentioned that support is done per platform and that the following vendors and platforms have just been update. Note that the HCL will still need to be updated for these:
- Dell SAS Expanders 5.5 and 6.0 hybrid. Supported on a specific configuration: H730 on 730d platform with 24 drives.
- Lenovo SAS Expanders 5.5. Supported on a specific configuration: Lenovo 720ix, RD650 platform with 12 drives.
We also are working on support for HP and Cisco (and others over time), and you can also expect an update on those soon. For now, keep in mind that when designing / planning for Virtual SAN the number of virtual disks may mean you need to change the configuration of your host.
Good thing I used an R730xd then. Haven’t run into an issue yet, I’m even using the H330 instead of an H730, this way I don’t have to do the RAID0 thing.
All dell 13th generation percs support disk passthru natively so the h730 does not require a raid0 per disk. It was the previous generation that had the limitation.
8disk max is common, also sas expander. you cannot use standard sas expander on hp server and also no 2nd controller to get devices into vsan? really?
Euuh, as the post an the quote of the VSAN Ready Node states:
“In case you need to add more than 8 disks, please add an additional controller.”
and
“We also are working on support for HP and Cisco ”
More details for HP soon, but in the meanwhile a second controller should be no problem at all.
I’ve been running Dell 730xd Virtual SAN clusters with 18 drives per server (3 disk groups) successfully for a while now. There was a struggle at first where I had to run in RAID 0 before they released new drivers for the PERC H730P controllers. But have been running in pass-through mode for over 6 months now. I may not be pushing them as hard as some people might but for our usage they have been working well.
So Drew, you’re running 3 H730p controllers in your setup?
No, I am running 1 h730p in each server.
As of last week supported up to 24 drives with a SAS Expander!
Bad timing that I just saw this now. Our R730xds just came in. One H730, Two SSDs and 8 SAS drives each.
Supported with the R730XD as of last week, so you are good.
Is this a recent development? I’m confused. I literally just racked 4 Dell R730’s for a VDI cluster and had them spec’d exactly like the “Value Configuration” on page 34 of their (Dell’s) VSAN whitepaper at:
http://en.community.dell.com/cfs-file/__key/telligent-evolution-components-attachments/13-4491-00-00-20-43-81-92/VMware-Virtual-SAN-for-ESXi-5-5-_2D00_-v.6.5.5.pdf
2 disk groups. 1 SSD and 6 300GB 15k drives per group for a total of 14 drives. Just 1 H730P mini controller just like in that document.
Figured I couldn’t go wrong following a tested design. I can’t add another disk controller due to the 2 nVidia GRID cards, a Teradici card, 10Gig network card and FC card connecting to our legacy storage SAN.
Hmm… Sounds like something worth knowing. If you are running 14 total SAS disks and two of them being your SSD performance tier, their could be bandwidth contention… maybe. This may not be the case when utilizing PCIe as your performance tier. Just a thought
Which controller did you use? If you used the H730 then you are good, supported as of last week with up to 24 drives
Duncan, We specced our VSAN cluster off the Dell 12G (R720XD) generation, and looking at that document it states that with 8 HDD + 2 SSD we only need 1x H710p.
However I believe the R720XD backplane has a built in SAS Expander. (2x SFF-8087 cables for 12+2 drives).
We have populated the box with 12+2 as of course this makes most commercial sense.
Can you clarify on whether the Dell backplanes are supported SAS Expanders?
http://partnerweb.vmware.com/programs/vsan/Virtual%20SAN%20Ready%20Nodes.pdf
According to the note in this paper you should be good.
uuuurgh I have just installed 10 x Cisco UCS C240 M3S hosts, which have one LSI 6G MegaRAID 9271-8i cards… each host has 2 x 400GB SAS SSD & 12 X 900GB 6G SAS spindles.. This kit was purchased before my arrival at the company.. so we need additional controllers right Duncan? Cheers @pshearduk
Good question Paul, not sure about this one. I will ask one of our PMs.
Much appreciated Duncan.
These hosts are “VSAN ready nodes” too btw
A quick update.. If I see the 14 disks (12 x SAS, 2 x SSD) via the LSI MegaRAID BIOS, does this mean that I already have a SAS expander installed? or do I still need the 2nd LSI card?
Thanks
Paul.
Your scenario is fine, you are supported. It is with new 12G controllers that these restrictions apply.
Yes, just to confirm Duncan’s above comments. We have our VSAN up and in the testing phase housed in 4 R730xds, single H730 in each, Two disk groups of 4 SAS10k and 1 SSD per group.
No problems yet.
Duncan – so you are saying the 12G controllers, like the Cisco 12G SAS Modular Raid Controller, max out at 8 drives per controller?
Thanks,
Bob
Yes that is what I am saying.