Over the past couple of weeks I have been watching the VMware VSAN Community Forum with close interest and also twitter. One thing that struck me was the type of hardware people used for to test VSAN on. In many cases this is the type of hardware one would use at home, for their desktop. Now I can see why that happens, I mean something new / shiny and cool is released and everyone wants to play around with it, but not everyone has the budget to buy the right components… And as long as that is for “play” only that is fine, but lately I have also noticed that people are looking at building an ultra cheap storage solution for production, but guess what?
Virtual SAN reliability, performance and overall experience is determined by the sum of the parts
You say what? Not shocking right, but something that you will need to keep in mind when designing a hardware / software platform. Simple things can impact your success, first and foremost check the HCL, and think about components like:
- Disk controller
- SSD / PCIe Flash
- Network cards
- Magnetic Disks
Some thoughts around this, for instance the disk controller. You could leverage a 3Gb/s on-board controller, but when attaching lets say 5 disks to it and a high performance SSD do you think it can still cope or would a 6Gb/s PCIe disk controller be a better option? Or even leverage 12Gb/s that some controllers offer for SAS drives? Not only can this make a difference in terms of number of IOps you can drive, it can also make a difference in terms of latency! On top of that, there will be a difference in reliability…
I guess the next component is the SSD / Flash device, this one is hopefully obvious to each of you. But don’t let these performance tests you see on Tom’s or Anandtech fool you, there is more to an SSD then just sheer IOps. For instance durability, how many writes per day for X years life can your SSD handle? Some of the enterprise grades can handle 10 full writes or more per day for 5 years. You cannot compare that with some of the consumer grade drives out there, which obviously will be cheaper but also will wear out a lot faster! You don’t want to find yourself replacing SSDs every year at random times.
Of course network cards are a consideration when it comes to VSAN. Why? Well because I/O will more than likely hit the network. Personally, I would rule out 1GbE… Or you would need to go for multiple cards and ports per server, but even then I think 10GbE is the better option here. Most 10GbE are of a decent quality, but make sure to check the HCL and any recommendations around configuration.
And last but not least magnetic disks… Quality should always come first here. I guess this goes for all of the components, I mean you are not buying an empty storage array either and fill it up with random components right? Think about what your requirements are. Do you need 10k / 15k RPM, or does 7.2k suffice? SAS vs SATA vs NL-SATA? Also, keep in mind that performance comes at a cost (capacity typically). Another thing to realize, high capacity drives are great for… yes adding capacity indeed, but keep in mind that when IO needs to come from disk, the number of IOps you can drive and your latency will be determined by these disks. So if you are planning on increasing the “stripe width” then it might also be useful to factor this in when deciding which disks you are going to use.
I guess to put it differently, if you are serious about your environment and want to run a production workload then make sure you use quality parts! Reliability, performance and ultimately your experience will be determined by these parts.
<edit> Forgot to mention this, but soon there will be “Virtual SAN” ready nodes… This will make your life a lot easier I would say.
</edit>
Many VSA vendors marketing try to convince you that no SAN shared storage will be cheaper than a normal implementation with a SAN array.
Even if this might not be the case for vSAN, people will always try to make it run on cheap hardware just because they can, this is the power of software only solutions; the use of SSD (good or bad ssd) will speed things up anyway… i hear all the time people already saying that with vSAN we will see big sata drives behind the ssd tier but they don’t realize that not everything will be in cache when reading.
I guess when the HCL will start being populated with more certified hardware (at the moment i can see only 2 Dell servers) it will be easier to understand; the option to choose a “ready made” hardware configuration node which will be activated later on in the vmware HCL will make it even more clear.
My consideration about costs also resides in the fact that vSAN relies on storage profiles to work which are a vsphere enterprise plus feature so i don’t think we will see many 3 nodes implementation…
I think vSAN should be considered an enterprise level replacement for SAN arrays, probably not suited for entry level scenarios, it’s just that VMware didn’t make this very clear in their vSAN messages so far.
Do note that in the Beta you can use VSAN with editions lower than E+ and get VM Storage Policies with it. So I would expect that functionality to be bundled, instead of requiring E+.
Duncan, I agree with the recommendation for enterprise grade equipment for quality storage. One of the first thing I think that will break a decent sized VSAN is the network. You and Cormac are absolutely right to push the message “the best practice is to use 10Gb links.”
The VSAN projet is still in Beta, so things still break and need fixing, and more importantly, VMware has not yet announced the pricing policy for VSAN.
I would not begin to assess VSAN for deployment until I can compare the costs of a VSAN deployment with enterprise grade components versus a Shared storage solution.
Thanks for covering so many elements of this new interesting technology.
My lab was in bad need of an uplift (HDS SMS was the SAN, QNAP for NAS) so I’m actually ignoring most of Duncan’s advice (Use quality parts, or fast drives) on a quest to build the cheapest moderately performing high capacity VSAN cluster money can buy for ~10K in hardware for the lab. http://vexpert.me/gV Get the final parts in tomorrow and should be benchmarking next week.
It also matters about the power usage. Anyways for large hard disks, i would suggest short stoking them. For example formatting only 50 percent of a drive or a raid group.
No usage of RAID in VSAN 🙂
Good recommendations. They point you to main issues in implementing VSAN, but… nothing about hardware. What if I would like to build something like this at home? Nothing written… Yes! Everybody clearly understand they need good SSDs, faster HDDs, a good /not cheap RAID card, but nothing specifically described about hardware can be used at home. Let say… go with a Samsung 840 EVO… Good SSD for you home lab. Don’t buy anything else. Anyway you will never have same IOPs you have at work in Production.
People don’t make benchmark test’s at home. They don’t have that budget that can afford enterprise hardware. They need just a roughly advice of some hardware that can be used, let say to build a VSAN environment to have an idea how it works.
You don’t think so?
I am not following you to be honest…
The article is Designing your hardware for VSAN.
I say to myself: Oh! This is exactly what I want. I want to build a VSAN environment at home, but I don’t know which hardware is good to start with.
And I assume there will be a recommendation to build a cheap lab. But the article starts with “first and foremost check the HCL” . This is good for people who have hardware and they want to check if their hardware is part of HCL. If not, to go and buy that hardware. Who ever read about VSAN knows very well that main points are:
Disk controller
SSD / PCIe Flash
Network cards
Magnetic Disks
You say 3Gb/s is good but not good. Go with 6Gb/s. What if I would like to go with 3Gb/s? Is it good to test VSAN? If yes which controller you recommend for a home lab. Something cheap and good to test a VSAN…
SSDs… which one is good? You say “we don’t have money to buy SSDs every year” . But this is impossible. Next year will be 9Gb/s or 12Gb/s . 6Gb/s SSD is still expensive for my budget, but I can afford 3Gb/s SSD from ebay. Which one is good if I can’t afford 6Gb/s SSDs?
Network card. Yeah!… I know that 10GbE is the best. Based on some enthusiasts recommendations I would go even with an Infiniband between servers (no switch)
Magnetic disks… WD Red is good but as a storage only and not good for the lab. 10k/15 RPM is not for a home budget definitely. So which one do you recommend and is not overkill?
Maybe I was expecting for something else from article? Maybe I was expecting for something like:How to design a VSAN at home? You have to look at disk controller – look at this or this model, you need to look at SSD drive – this one is good for a limited budget. If you have lot of money and Santa loves you, buy this SSD… and etc.
Hope I was clear =)
Here is a small design of a micro ESXi server with VSAN and vFRC. Hope this will give you some ideas…
http://www.bussink.ch/?p=1123 Homelab with vSphere 5.5 and VSAN.
How can I make recommendations for a cheap lab without components being on the HCL… I am not going down that rathole for sure, if it doesn’t work I am to blame. It is not like I have the budget to test all combinations.
If you find working cheap combo’s feel free to post them.
Alex, I agree. Real hardware benchmarks was something strongly missing from the VSA product release. It was ultimately Dell employees who rustled the hardware to get us some benchmarks (We ended up buying some fully loaded Cisco 2U’s at the time, to find out). The problem is normally vendors will lend gears or labs for this stuff, but sadly the major OEM of servers (IBM/Dell/HP/Cisco) have other storage solutions and have no invested interest in loaning 60-100K in storage. We are at least using Intel 3500 SSD’s (HP re-brands these for MLC’s I believe). If brocades gets me the VDX switches to replace my questionable 10Gig Netgears I might be able to actually build a more “enterprise” lab than a lot of what people are doing right now in beta. Hopefully we can try to push and find the limits, or validate that this system will scale.
Ahhh, just read read Alex’s comment. He’s looking for a quick and dirty guide I think to test this at home. Try this link, maybe this work 🙂
http://www.virtuallyghetto.com/2013/09/how-to-quickly-setup-and-test-vmware.html
Or click the VSAN link at the top, plenty articles there.
Thanks John.
This is what I was asking for. I know VSAN requirements, but I don’t know how to build it at home with less hardware. Company’s budget is very limited or they don’t need that technology, but for me as VCP is very important to be familiar with many VMware products. I want to play with many labs and rise my knowledges, but my salary is too low to buy equipment to play with those products. So the only one solution is a dirty solution. Like virtualghetto post to emulate SSD for VSAN. Good idea because I don’t have money to buy a SSD.
I’m not asking Duncan to give me specific cheap specs to build a VSAN. I understand his reaction to my question: how can I suggest you something cheap if VMware build products for SMBs and Enterprise environments and not for your home lab scant hardware.
I need and must to learn those technologies, but it doesn’t mean I have to stop there if my budget is limited and I can’t afford it.
Anyway this post is just a “Right Direction Guide” to take into consideration when you want to build a VSAN.
Good article on what to look for overall. Building a home lab is a challenge its the whole triangle, one point is speed/iops/latency, one point is cost, size(?)… you have to sacrifice one of them for sure to gain in the other…
Duncan, admittedly have not yet played with VSAN yet due to lack of hardware, but one of the things you brought up around SSD Drive Writes triggered a question.
Some RAID controllers now have a Wear Gauge to show when SSDs are nearing their end of useful life and need to be replaced.
VSAN however does not use RAID.
Does ESXi 5.5 have some mechanism included to notify you when you hit some threshold of the drive’s useful life so you can order a replacement and schedule the swap?
Also, using higher quality HDDs makes sense for some workloads, but I often hear storage reports saying something like 80% of data is relatively static and needs very little IOPs these days.
Is VMware looking at possibly doing a 3 tier approach down the road?
Enterprise SSD/PCIe
600GB 10K (Price/performance sweet spot for 2.5″ SAS last time I checked)
3/4TB 7200 RPM drives
PS: For a home lab, I saw this today and my jaw dropped:
http://www.amazon.com/Crucial-2-5-Inch-adapter-Internal-CT960M500SSD1/dp/B00BQ8RGL6/
Almost a terabyte of SSD for $500.
For now there are no direct plans to do 3 tiers. I would recommend using a proper SSD with potentially a higher “SSD to HDD” ratio ( > 10%) instead and go with 4TB sata drives. You will end up with more cache and higher capacity from an HDD perspective and it won’t cost that much more.
Realistically, for a 3-host cluster with less than 30 virtual machines, wouldn’t 1 GbE suffice for vSAN, which two vMK ports or more per host? The reason I say this, for a small or medium-sized business that wants to use VMware but does not want to purchase a separate SAN, the cost of adding one or two 10 GbE switches just for vSAN is prohibitive. Will there be performane numbers published comparing 1 GbE to 10 GbE? That would be helpful.