Removing a disk group from a VSAN host

4 December, 2013 by Leave a Comment

I had been playing around with my VSAN cluster for a bit the last couple of weeks and it literally has become messy. I created many VMs and many snapshots and removed many of those again, all of this while pulling cables of servers and pulling disks. Basically stress testing VSAN while injecting faults to see how it responds. It was time to clean up and upgrade to a later build as the beta refresh was just released. After deleting a bunch of VMs I noticed that not everything was removed, I had also uploaded ISOs and some other random stuff which I probably should not have. Anyway, I needed to clean one of my hosts up.

I figured I would use RVC for the exercise just to get a bit more familiar with it. First I wanted to check what the current state was of my cluster, I used the “vsan.disks_stats” command:

Then I figured I would want to just simply remove the disk group for Server “prmb-esx08” using “vsan.host_wipe_vsan_disks”:

Note that you can also do this using the UI:

  • Go to your cluster
  • Click “Manage” and “Virtual SAN” -> “Disk Management”
  • Select the “Disk Group” and click the “Remove the Disk Group” icon
Filed Under: Software Defined, Storage, Virtual SAN Tagged With: , , ,

Startup News Flash part 10

29 November, 2013 by 1 Comment

There we are, part 10 of the Startup News Flash. Someone asked me on Twitter last week why Company XYZ was never included in the news flash. Let it be clear that I am not leaving anyone out (unless I feel they aren’t relevant to this newsletter or my audience), I have limited time so typically do not do briefings… Which means that if the marketing team doesn’t sent me the details via email and I haven’t somehow stumbled across the announcement it will not appear on here. If you want your company to be listed, make sure they sent their press releases over.

Some new models announced by Nutanix. Funny to see how they’ve been pushing hard from a marketing perspective to remove the “pure VDI play” label they had and now launch a VDI focused model called the 7000 series. (Do not get me wrong, I think this is a brilliant move!) The 7000 series offers you the option to include NVIDIA K1 or K2 Grid cards. Primarily intended to accelerate graphics, so if you are for instance doing a lot of 3D rendering or just are a heavy graphical VDI user these could really provide a benefit over their (and other vendors) normal offerings. On top of that the 3000 and 6000 series has been overhauled. The NX-3061 and NX-3061 with 10 Core (2.8GHz) Ivy Bridge have been introduced and the NX6060 and NX6080 10 Core (2.8 and 3.0GHz respectively) have been introduced. Haven’t seen anything around pricing, so can’t comment on that.

No clue what it is exactly these guys do to be honest. I find their teaser video very intriguing. Not much detail to be found around what they are doing other than “re-imagine enteprise computing”. Hoping to hear more from these guys in the future as their teaser did make me curious.

I don’t care much about benchmarks, but it is always nice to see a smaller (or the underdog) company beat the big players. Kaminario managed to outperform Oracle, IBM and Fujitsu with their SPC-2 Performance Benchmark using their scale-out all flash array K2 v4. Just a couple of weeks after breaking the SPC-1 Benchmark World Record again. Like I said, I don’t care much about benchmarks  as it doesn’t typically say much about the operational efficiency etc. Still it is a nice indication of what can be achieved, though your results may vary depending on your IO pattern of course.

Filed Under: Server, Software Defined, Storage Tagged With: , , , , , ,

VSAN VDI Benchmarking and Beta refresh!

26 November, 2013 by 3 Comments

I was reading this blog post on VSAN VDI Benchmarking today on Vroom, the VMware Performance blog. You see a lot of people doing synthetic tests (max iops with sequential reads) on all sorts of storage devices, but lately more and more vendors are doing these more “real world performance tests”. While reading this article about VDI benchmarking, and I suggest you check out all parts (part 1, part 2, part 3), there was one thing that stood out to me and that was the comparison between VSAN and an All Flash Array.

The following quotes show the strength of VSAN if you ask me:

we see that VSAN can consolidate 677 heavy users (VDImark) for 7-node and 767 heavy users for 8-node cluster. When compared to the all flash array, we don’t see more than 5% difference in the user consolidation.

Believe me when I say that 5% is not a lot. If you are actively looking at various solutions, I would highly recommend to include the “overhead costs” to your criteria list as depending on the solution chosen this could make a substantial difference. I have seen other solutions requiring a lot more resources. But what about response time, cause that is where the typical All Flash Array shines… ultra low latency, how about VSAN?

Similar to the user consolidation, the response time of Group-A operations in VSAN is similar to what we saw with the all flash array.

Both very interesting results if you ask me. Especially the < 5% in user consolidation is what stood out to me the most! Once again, for more details on these tests read the VDI Benchmarking blog part 1, part 2, part 3!

Beta Refresh

For those who are testing VSAN, there is a BETA refresh available as of today. This release has a fix for the AHCI driver issue… and it increases the disk group limit from 6 to 7. From a disk group perspective this will  come in handy as many servers have 8, 16 or 24 disk slots allowing you to do 7HHDs + 1 SSD per group. Also some additional RVC commands have been added in the storage policy space, I am sure they will come in handy!

Nice side affect of the number of HDDs going up is increase in max capacity:

(8 hosts * (5 diskgroups * 7 HDDs)) * Size of HDD = Total capacity

With 2 TB disks this would result in:

(8 * (5 * 7)) * 2TB = 560TB

Now keep on testing with VSAN and don't forget to report feedback through the community forums or your VMware rep.

Filed Under: Software Defined, Storage, Virtual SAN Tagged With: , , , , , ,

Virtual SAN and maintenance windows…

25 November, 2013 by 4 Comments

After writing the article that “4 is the minimum number of hosts for VSAN” I received a lot of questions via email and on twitter etc about the cost associated with it and if this was a must. Let me start with saying that I wrote this article to get people thinking about Sizing their VSAN environment. When it comes to it, Virtual SAN and maintenance windows can be a difficult topic.

I guess there are a couple of things to consider here. Even in a regular storage environment you typically do upgrades in a rolling fashion meaning that if you have two controllers one will be upgraded while they other handles IO. In that case you are also at risk. The thing is though, as a virtualization administrator you have a bit more flexibility, and you expect certain features to work as expected like for instance vSphere HA. You need to ask yourself what is the level of risk I am willing to take, the level of risk I can take?

When it comes to placing a host in to Maintenance Mode, from a VSAN point of view you will need to ask yourself:

  • Do I want to move data from one host to another to maintain availability levels?
  • Do I just want to ensure data accessibility and take the risk of potential downtime during maintenance?

I guess there is something to say for either. When you move data from one node to another, to maintain availability levels, your “maintenance window” could be stretched extremely long. As you would potentially be copying TBs over the network from host to host it could take hours to complete. If your ESXi upgrade including a host reboot takes about 20 minutes, is it acceptable to wait for hours for the data to be migrated? Or do you take the risk, inform your users about the potential downtime, and as such do the maintenance with a higher risk but complete it in minutes rather than hours? After those 20 minutes VSAN would sync up again automatically, so no data loss etc.

It is impossible for me to give you advice on this one to be honest, I would highly recommend to also sit down with your storage team. Look at what their current procedures are today, what they have included in their SLA to the business (if there is one), and how they handle upgrades / periodic maintenance.

 

Filed Under: Server, Software Defined, Storage, Virtual SAN Tagged With: , , , ,

Startup News Flash part 9

18 November, 2013 by Leave a Comment

There we are, part 9 of the Startup News Flash. As mentioned last time, last week was “Storage Field Day” so typically a bit more news then normally this time of year. I would highly recommend to watch the videos. Especially the Coho video is very entertaining.

The original founders of Fusion IO (David Flynn and Rick White) just received 50 Million in funding for their new startup called Primary Data. I mentioned them briefly in Startup News Flash part 3 when they announced they started a new company and it seems that they have something on their hands! They haven’t revealed what they are working on, they are aiming to come out stealth around the second quarter of 2014. In the WSJ the following was mentioned in terms of the space these guys will be playing in: “The company is developing software–though it actually will come bundled on standard server hardware–that essentially connects all those pools of data together, offering what Flynn calls a “unified file directory namespace” visible to all servers in company computer rooms–as well as those “in the cloud” that might be operated by external service companies.” Indeed, something with storage / caching / software defined / scale-out…

I guess scale-out hypervisor based storage solutions are hot… Maxta just officially announced their new product called MxSP. Some rumors had already been floating around but now the details are out their. Marcel v/d Berg did a nice article on them which I recommend reading if you like to get some more details. Basically Maxta created a Virtual Storage Appliance which pools all local storage and presents it as NFS to your hypervisor. Today VMware vSphere is fully supported and KVM / Hyper-V in a limited fashion. It offers functionality like VM-level snapshots and zero-copy clones, Thin provisioning, Inline deduplication and more. It looks like licensing is capacity based but no prices have been mentioned.

When first looking at Avere is was intrigued by their solution but somehow it didn’t really click. Primary focus was a caching layer in between your NFS storage and your hosts… But I wondered why I would want an extra box for that and not just use something host local. Last week Avere made announcement around a solution that allows you to pool local and cloud storage resources and present them via a common namespace and move data between these tiers. FlashCloud is what Avere calls it. Their paper describes it best, so a shameless copy of that: “FlashCloud software running on Avere FXT Edge filers addresses this challenge by storing cold data on cost-effective cloud storage at the core of the network and automatically and efficiently moving active data to the edge near the users.” I like the concept… If you are interested, check out their site here.

Far from a startup, but cool enough to be listed here… The release of the X-Brick aka XtremIO by EMC. The XtremIO solution is a brand new all-flash array which delivers screaming performance in a scale-out fashion. Although there are limitations from a scaling point of view today, it is expected that these will be lifted soon. One of the articles I enjoyed reading is this one by Jason Nash. What is most interesting about the product is the following, and I am going to quote Jason here as he is spot on: “There is no setup and tuning of XtremIO.  No LUNs.  No RAID Groups.  No pools.  No stripe sizes.  No tiering.  Nothing.  You have a pool of very fast storage.  How big do you want that LUN to be?  That’s all you really need to do!”

Another round of funding for SimpliVity, Series C… 58 Million led by Kleiner Perkins Growth Fund and DFJ Growth with contributions by Meritech Capital Partners and Swisscom Ventures. I guess this GigaOM quote says it all: “CEO Doron Kempel, an EMC veteran, said the cash infusion will enable the company to execute on plans to triple its staff and boost sales growth five fold in 2014″.

Filed Under: Software Defined, Storage Tagged With: , , , , , , , ,

VSAN performance: many SAS low capacity VS some SATA high capacity?

14 November, 2013 by 23 Comments

Something that I have seen popping up multiple times now is the discussion around VSAN and spindles for performance. Someone mentioned on the community forums they were going to buy 20 x 600GB SAS drives for their VSAN environment for each of their 3 hosts. These were 10K SAS disks, which obviously outperform the 7200 RPM SATA drives. I figured I would do some math first:

  • Server with 20 x 600GB 10K SAS = $9,369.99 per host
  • Server with 3 x 4TB Nearline SAS = $4,026.91 per host

So that is about a 4300 dollar difference. Note that I did not spec out the full server, so it was a base model without any additional memory etc, just to illustrate the Perf vs Capacity point. Now as mentioned, of course the 20 spindles would deliver additional performance. Because after all you have additional spindles and better performing spindles. So lets do the math on that one taking some average numbers in to account:

  • 20 x 10K RPM SAS with 140 IOps each = 2800 IOps
  • 3 x 7200 RPM NL-SAS with 80 IOps each = 240 IOps

That is a whopping 2560 IOps difference in total. That does sound like an awe full lot doesn’t it? To  a certain extent it is a lot, but will it really matter in the end? Well the only correct answer here is: it depends.

I mean, if we were talking about a regular RAID based storage system it would be clear straight away… the 20 disks would win for sure. However we are talking VSAN here and VSAN heavily leans on SSD for performance. Meaning that each diskgroup is fronted by an SSD and that SSD is used for both Read Caching (70% of capacity) and write buffering (30%) of capacity. Illustrated in the diagram below.

The real question is what is your expected IO pattern? Will most IO come from read cache? Do you expect a high data change rate and as such could de-staging be problematic when backed by just 3 spindles? Then on top of that, how and when will data be de-staged? I mean, if data sits in write buffer for a while it could be the data changes 3 or 4 times before being destaged, preventing the need to hit the slow spindles. It all depends on your workload, your IO pattern, your particular use case. Looking at the difference in price, I guess it makes sense to ask yourself the question what $ 4300 could buy you?

Well for instance 3 x 400GB Intel S3700 capable of delivering 75k read IOps and 35k write IOps (~800 dollars per SSD). That is extra, as with the server with 20 disks you would also still need to buy SSD and as the rule of thumb is roughly 10% of your disk capacity you can see what either the savings are or the performance benefits could be. In other words, you can double up on the cache without any additional costs compared to the 20-disk server. I guess personally I would try to balance it a bit, I would go for higher capacity drives but probably not all the way up to 4TB. I guess it also depends on the server type you are buying, will they have 2.5″ drive slots or 3.5″? How many drive slots will you have and how many disks will you need to hit the capacity requirements? Are there any other requirements? As this particular user mentioned for instance he expected extremely high sustained IOs and potentially full backups daily, as you can imagine that could impact the number of spindles desired/required to meet performance expectations.

The question remains, what should you do? To be fair, I cannot answer that question for you… I just wanted to show that these are all things one should think about before buying hardware.

Just a nice little fact, today a VSAN host can hold 5 Disk Groups with 6 disks, so 30 disks in total. With 8 hosts in a cluster that is 240 disks… With GA the number of disks will potentially go up to 7 and even the number of hosts per cluster might be increased… You could potentially do ~560 disks in 1 one cluster. That is some nice capacity right with 4TB disks that are available today.

I also want to point out that a tool is being developed as we speak which will help you making certain decisions around hardware, cache sizing etc. Hopefully more news on that soon,

** Update, as of 26/11/2013 the VSAN Beta Refresh allows for 7 disks in a disk group… **

 

Filed Under: Server, Software Defined, Storage, Virtual SAN Tagged With: , , , ,

VSAN and Network IO Control / VDS part 2

12 November, 2013 by 2 Comments

About a week ago I wrote this article about VSAN and Network IO Control. I originally wrote a longer article that contained more options for configuring the network part but decided to leave a section out of it for simplicity sake. I figured as more questions would come I would publish the rest of the content I developed. I guess now is the time to do so.

In the configuration described below we will have two 10GbE uplinks teamed (often referred to as “etherchannel” or “link aggregation”). Due to the physical switch capabilities the configuration of the virtual layer will be extremely simple. We will take the following recommended minimum bandwidth requirements in to consideration for this scenario:

  • Management Network –> 1GbE
  • vMotion VMkernel –> 5GbE
  • Virtual Machine PG –> 2GbE
  • Virtual SAN VMkernel interface –> 10GbE

When the physical uplinks are teamed (Multi-Chassis Link Aggregation) the Distributed Switch load balancing mechanism is required to be configured as:

  1. IP-Hash
    or
  2. LACP

It is required to configure all portgroups and VMkernel interfaces on the same Distributed Switch using either LACP or IP-Hash depending on the type physical switch used. Please note all uplinks should be part of the same etherchannel / LAG. Do not try to create anything fancy here as a physically and virtually incorrectly configured team can and probably will lead to more down time!

  • Management Network VMkernel interface = LACP / IP-Hash
  • vMotion VMkernel interface = LACP / IP-Hash
  • Virtual Machine Portgroup = LACP / IP-Hash
  • Virtual SAN VMkernel interface = LACP / IP-Hash

As various traffic types will share the same uplinks we also want to make sure that no traffic type can push out other types of traffic during times of contention, for that we will use the Network IO Control shares mechanism.

We will work under the assumption that only have 1 physical port is available and all traffic types share the same physical port for this exercise. Taking a worst case scenario approach in to consideration will guarantee performance even in a failure scenario. By taking this approach we can ensure that Virtual SAN always has 50% of the bandwidth to its disposal while leaving the remaining traffic types with sufficient bandwidth to avoid a potential self-inflicted DoS. When both Uplinks are available this will equate to 10GbE, when only one uplink is available the bandwidth is also cut in half; 5GbE. It is recommended to configure shares for the traffic types as follows:

 

Traffic Type Shares Limit
Management Network  100 n/a
vMotion VMkernel Interface  250 n/a
Virtual Machine Portgroup  150 n/a
Virtual SAN VMkernel Interface  500 n/a

 

The following diagram depicts this configuration scenario.

Filed Under: Server, Software Defined, Storage, Virtual SAN Tagged With: , , , , , ,
Next Page»