It seems Virtual SAN is on fire… I’ve been getting emails about VSAN configurations on a daily basis from partners, VMware field and customers. More and more customers seem to be looking at deploying Virtual SAN with a high number of disks and this week I ran in to something that I wanted to share with you. I thought it was common knowledge, but I guess I was wrong. If you are planning to have more than 8 disks / devices in your Virtual SAN host then it is good to know that you will either need a second disk controller (Check the HCL!) or use a SAS Expander! This design / sizing principle is listed in the VSAN Ready Node PDF, and should also make its way to the Design / Sizing Guide soon. The following is mentioned in the VSAN Ready Node PDF:
No more than 8 disks (SSD + HDD) supported behind a single controller. In case you need to add more than 8 disks, please add an additional controller.
I met up with the product manager for the compatibility guide and asked him which SAS Expanders are supported, he mentioned that support is done per platform and that the following vendors and platforms have just been update. Note that the HCL will still need to be updated for these:
- Dell SAS Expanders 5.5 and 6.0 hybrid. Supported on a specific configuration: H730 on 730d platform with 24 drives.
- Lenovo SAS Expanders 5.5. Supported on a specific configuration: Lenovo 720ix, RD650 platform with 12 drives.
We also are working on support for HP and Cisco (and others over time), and you can also expect an update on those soon. For now, keep in mind that when designing / planning for Virtual SAN the number of virtual disks may mean you need to change the configuration of your host.
Bryan Pizzuti says
Good thing I used an R730xd then. Haven’t run into an issue yet, I’m even using the H330 instead of an H730, this way I don’t have to do the RAID0 thing.
robbie smith (@virtualrob) says
All dell 13th generation percs support disk passthru natively so the h730 does not require a raid0 per disk. It was the previous generation that had the limitation.
Mark says
8disk max is common, also sas expander. you cannot use standard sas expander on hp server and also no 2nd controller to get devices into vsan? really?
Duncan Epping says
Euuh, as the post an the quote of the VSAN Ready Node states:
“In case you need to add more than 8 disks, please add an additional controller.”
and
“We also are working on support for HP and Cisco ”
More details for HP soon, but in the meanwhile a second controller should be no problem at all.
Drew says
I’ve been running Dell 730xd Virtual SAN clusters with 18 drives per server (3 disk groups) successfully for a while now. There was a struggle at first where I had to run in RAID 0 before they released new drivers for the PERC H730P controllers. But have been running in pass-through mode for over 6 months now. I may not be pushing them as hard as some people might but for our usage they have been working well.
Braf says
So Drew, you’re running 3 H730p controllers in your setup?
Drew says
No, I am running 1 h730p in each server.
Duncan Epping says
As of last week supported up to 24 drives with a SAS Expander!
Brad says
Bad timing that I just saw this now. Our R730xds just came in. One H730, Two SSDs and 8 SAS drives each.
Duncan Epping says
Supported with the R730XD as of last week, so you are good.
Kenneth Slish says
Is this a recent development? I’m confused. I literally just racked 4 Dell R730’s for a VDI cluster and had them spec’d exactly like the “Value Configuration” on page 34 of their (Dell’s) VSAN whitepaper at:
http://en.community.dell.com/cfs-file/__key/telligent-evolution-components-attachments/13-4491-00-00-20-43-81-92/VMware-Virtual-SAN-for-ESXi-5-5-_2D00_-v.6.5.5.pdf
2 disk groups. 1 SSD and 6 300GB 15k drives per group for a total of 14 drives. Just 1 H730P mini controller just like in that document.
Figured I couldn’t go wrong following a tested design. I can’t add another disk controller due to the 2 nVidia GRID cards, a Teradici card, 10Gig network card and FC card connecting to our legacy storage SAN.
Jon says
Hmm… Sounds like something worth knowing. If you are running 14 total SAS disks and two of them being your SSD performance tier, their could be bandwidth contention… maybe. This may not be the case when utilizing PCIe as your performance tier. Just a thought
Duncan Epping says
Which controller did you use? If you used the H730 then you are good, supported as of last week with up to 24 drives
Myles Gray says
Duncan, We specced our VSAN cluster off the Dell 12G (R720XD) generation, and looking at that document it states that with 8 HDD + 2 SSD we only need 1x H710p.
However I believe the R720XD backplane has a built in SAS Expander. (2x SFF-8087 cables for 12+2 drives).
We have populated the box with 12+2 as of course this makes most commercial sense.
Can you clarify on whether the Dell backplanes are supported SAS Expanders?
Duncan Epping says
http://partnerweb.vmware.com/programs/vsan/Virtual%20SAN%20Ready%20Nodes.pdf
According to the note in this paper you should be good.
Paul. says
uuuurgh I have just installed 10 x Cisco UCS C240 M3S hosts, which have one LSI 6G MegaRAID 9271-8i cards… each host has 2 x 400GB SAS SSD & 12 X 900GB 6G SAS spindles.. This kit was purchased before my arrival at the company.. so we need additional controllers right Duncan? Cheers @pshearduk
Duncan says
Good question Paul, not sure about this one. I will ask one of our PMs.
Paul. says
Much appreciated Duncan.
Paul. says
These hosts are “VSAN ready nodes” too btw
Paul. says
A quick update.. If I see the 14 disks (12 x SAS, 2 x SSD) via the LSI MegaRAID BIOS, does this mean that I already have a SAS expander installed? or do I still need the 2nd LSI card?
Thanks
Paul.
Duncan Epping says
Your scenario is fine, you are supported. It is with new 12G controllers that these restrictions apply.
Brad says
Yes, just to confirm Duncan’s above comments. We have our VSAN up and in the testing phase housed in 4 R730xds, single H730 in each, Two disk groups of 4 SAS10k and 1 SSD per group.
No problems yet.
Bob says
Duncan – so you are saying the 12G controllers, like the Cisco 12G SAS Modular Raid Controller, max out at 8 drives per controller?
Thanks,
Bob
Duncan Epping says
Yes that is what I am saying.