Introduction to VMware vSphere Virtual SAN

Many of you have seen the announcements by now and I am guessing that you are as excited as I am about the announcement of the public beta of Virtual SAN with vSphere 5.5. What is Virtual SAN, formerly known as “VSAN” or “vCloud Distributed Storage” all about?

Virtual SAN (VSAN from now on in this article) is a software based distributed storage solution which is built directly in the hypervisor. No this is not a virtual appliance like many of the other solutions out there, this sits indeed right inside your ESXi layer. VSAN is about simplicity, and when I say simple I do mean simple. Want to play around with VSAN? Create a VMkernel NIC for VSAN and enable it on a cluster level. Yes that is it!

vSphere Virtual SAN

Before we will get a bit more in to the weeds, what are the benefits of a solution like VSAN? What are the key selling points?

  • Software defined – Use industry standard hardware, as long as it is on the HCL you are good to go!
  • Flexible – Scale as needed and when needed. Just add more disks or add more hosts, yes both scale-up and scale-out are possible.
  • Simple – Ridiculously easy to manage! Ever tried implementing or managing some of the storage solutions out there? If you did, you know what I am getting at!
  • Automated – Per virtual machine policy based management. Yes, virtual machine level granularity. No more policies defined on a per LUN/Datastore level, but at the level where you want it to be!
  • Converged – It allows you to create dense / building block style solutions!

Okay that sounds great right, but where does that fit in? What are the use-cases for VSAN when it is released?

  • Virtual desktops
    • Scale out model, using predictive (performance etc) repeatable infrastructure blocks lowers costs and simplifies operations
  • Test & Dev
    • Avoids acquisition of expensive storage (lowers TCO), fast time to provision
  • Big Data
    • Scale out model with high bandwidth capabilities
  • Disaster recovery target
    • Cheap DR solution, enabled through a feature like vSphere Replication that allows you to replicate to any storage platform

So lets get a bit more technical, just a bit as this is an introduction right…

When VSAN is enabled a single shared datastore is presented to all hosts which are part of the VSAN enabled cluster. Typically all hosts will contribute performance (SSD) and capacity (magnetic disks) to this shared datastore. This means that when your cluster grows, your datastore will grow with it. (Not a requirement, there can be hosts in the cluster which just consume the datastore!) Note that there are some requirements for hosts which want to contribute storage. Each host will require at least one SSD and one magnetic disk. Also good to know is that with this beta release the limit on a VSAN enabled cluster is 8 hosts. (Total cluster size 8 hosts, including hosts not contributing storage to your VSAN datastore.)

As expected, VSAN heavily relies on SSD for performance. Every write I/O will go to SSD first, and eventually they will go to magnetic disks (SATA). As mentioned, you can set policies on a per virtual machine level. This will also dictate for instance what percentage of your read I/O you can expect to come from SSD. On top of that you can use these policies to define availability of your virtual machines. Yes you read that right, you can have different availability policies for virtual machines sitting on the same datastore. For resiliency “objects” will be replicated across multiple hosts, how many hosts/disks will thus depend on the profile.

VSAN does not require a local RAID set, just a bunch of local disks. Now, whether you defined a 1 host failure to tolerate ,or for instance a 3 host failure to tolerate, VSAN will ensure enough replicas of your objects are created. Is this awesome or what? So lets take a simple example to illustrate that. We have configured a 1 host failure and create a new virtual disk. This means that VSAN will create 2 identical objects and a witness. The witness is there just in case something happens to your cluster and to help you decide who will take control in case of a failure, the witness is not a copy of your object let that be clear! Note, that the amount of hosts in your cluster could potentially limit the amount of “host failures to tolerate”. In other words, in a 3 node cluster you can not create an object that is configured with 2 “host failures to tolerate”. Difficult to visualize? Well this is what it would look like on a high level for a virtual disk which tolerates 1 host failure:

With all this replication going on, are there requirements for networking? At a minimum VSAN will require a dedicated 1Gbps NIC port. Of course it is needless to say that 10Gbps would be preferred with solutions like these, and you should always have an additional NIC port available for resiliency purposes. There is no requirement from a virtual switch perspective, you can use either the Distributed Switch or the plain old vSwitch, both will work fine.

To conclude, vSphere Virtual SAN aka VSAN is a brand new hypervisor based distributed platform that enables convergence of compute and storage resources. It provides virtual machine level granularity through policy based management. It allows you to control availability and performance in a way I have never seen it before, simple and efficient. I am hoping that everyone will be pounding away on the public beta, sign up today: http://www.vmware.com/vsan-beta-register!

Be Sociable, Share!

    Comments

    1. Parth says

      Sounds promising……
      But…
      1-What if i don’t have a hardware raid configured on bunch of disks which are a part of VSAN DISK
      then who gonna save my life.
      So if i got this correct
      Hardware disk failures will be handled by replicas and witness along with host failures?

      Next…… PURPOSE — > ? Confused ?
      Why all this dual or triple writes of data blocks on replicas,engaging witness,making compute more busy passing data through SSD to traditional disks.Increasing read/write IO and turnaround time for data blocks all just at the cost of “A SHARED DATASTORE ”

      Is it worth bringing this concept into existence, am afraid or might am not able to get this concept right.
      Increasing complexity,IO,Compute just to get a share data store.

      No doubt i like the idea about “hypervisor based distributed platform” might be i lack some more deep thought on this.

      Cheers
      Parth

      • says

        VSAN doesn’t require RAID, or better said… you don’t want to do RAID when you do VSAN as VSAN takes care of that by replicating objects when required. Hardware failures are handled by replicas and the witness indeed.

        Not sure I am following your other comments to be honest.

    2. says

      I like that news. You know I mentioned before ever since I heard of Nutanix a year ago I thought that VMware should buy them, little did I know that you guys had something similar brewing. Nutanix is still superior but VSAN is a good step forward in the direction of compute/storage residing on the same host. The great part is that it is embedded on the the hypervisor. Questions:
      – Is it base on ZFS?
      – Is it safe to assume that like Nutanix that the vm vmdk store will follow wherever the vm is hosted to get the most storage performance?
      – I use an enterprise HP 4GB SD card to store the hypervisor, Is there a problem if I use this feature? – Is the SD card future proof for installing ESXi hypervisor?

      • says

        1) not based on ZFS, fully in-house developed.
        2) we don’t move data around to follow the VM like Nutanix does. 10GbE infra will ensure low latency. Also, it is distributed caching layer and resilient so multi copies of data, no need to move stuff id you ask me
        3) I don’t see an issue with using that SD card.

    3. says

      Glad that this is finally built into vSphere and no longer just an appliance, makes it much more convenient to use. Ability to have 8 servers in cluster seems to make it a better option for businesses that may eventually need more than 3 node cluster offered by Storage Appliance.

    4. Bill says

      Is the size of a single VM bound by the max size of the disks of a single host?
      And where are the metadata of the virtual datastore saved? Thanks.

    5. says

      Hey Duncan,
      I build Virtual Server clusters using ASUS servers for my customers using VMware VSA. Will the Virtual SAN replace the VSA ? If so will the HCL match the HCL for the VSA ? What about upgrading from VSA to the Virtual SAN. Would it even be an upgrade? Will VMware keep the VSA as well or will this be dropped ? What are the performance advantages between the two. Price differences? VSA is part of ESS Plus, will the Virtual SAN be an extension of the VSA for customer swanting to scale past 3 hosts?
      Any insight wold be helpful.
      Cheers!
      Peter

      • says

        There will be no direct upgrade path, it means move out and move in. It is a completely different product, which means that the HCL will also differ. For instance for now it will require a RAID controller that supports pass-through.

        • says

          Hey Duncan – thanks for taking the time to respond to comments during vmworld!
          Question related with VSAN vs VSA: if you’ve implimented that clunky product, what is it’s license/support path moving forward? Example: Full support for VSA for X amount of time but no SNS license upgrade path to VSAN on upgrade to 5.5 (when out of beta)? I understand no upgrade path of the software itself, just checking to see if my client will need to purchase a different virtual SAN solution moving forward.

    6. Scott says

      Duncan,

      As always, thanks for the useful info you provide! You are a credit to VMware. With that, we’ll await the all important ‘limitations and caveats’ article that needs to accompany this. There’s a ton of questions floating around the community about some of the real-world implications of this technology.

      Such as:
      1. You mention “Every write I/O will go to SSD first, and eventually they will go to magnetic disks (SATA).”. Detail? When does this happen?
      2. Obviously this brings RAID into question — Does this replace RAID? Many people will be very, VERY hesitant to go without RAID, so can this be used in conjunction with RAID? RAID still brings other performance benefits, not just redundancy of course…
      3. Does this work or can this work with vFlash caching?
      4. How ‘intelligently’ does this tier the data? Can vSphere differentiate between 7.2K/10K/15K disks? My only fear is that this may be an ‘all or nothing’ type approach to tiering, whereas many 3rd parties have been doing tiering for quite a while.

      Overall, I’m really impressed with the announcements in 5.5. vSAN sounds very interesting. I’ld be willing to bet in 5 years, you won’t buy a server, hypervisor, and storage separately. Everything will basically be very Nutanix’esque.

    7. Blaine says

      With every single write going to SSD, will the HCL for SSD be fairly restrictive to models that have a higher number of write cycles? How will monitoring for SSD wear work?

    8. says

      I just wanted to say I appreciate your summary Duncan. It’s great to see the difference between the testing limit of 8 nodes and the goal of the product being beyond that – far beyond 8 nodes, I imagine.

      We’re a little ways away from it, but I’m curious to see adoption in the SMB market and the layered play of ViPR + VSAN for larger implementations.

      Cheers.

    9. Dmitry says

      Duncan,
      Tested it on real harware. Three hosts, 1 SSD + DAS each.
      Configured everything, but vsphere didn’t use them for datastore though showed both hosts and disks as eligible.

      • Torsten Mutayi says

        Did you add the license key for VSAN?
        If yes: go to your cluster -> Manage -> Virtual SAN -> Disk Mangement and try to “Claim disks” (the leftmost buttn below “Disk groups”) manually

    10. Greg says

      Just go lookup Nutanix.com where their core product does all of this out of the box (put more accurately, in their box…). VMware is late to this party and for the time being you’re better off getting your converged scale-out platform from folks that tailor their solutions around this concept, rather than offer it as another thing you COULD do with your virtual architecture.

      • says

        I guess you just said it all: “in their box” << that is key here. VSAN is a software solution, not tied to hardware.

        Anyway, there is a big market out there… And still much more to come.

    11. Matt says

      VSAN looks to be be very similar when compared to CEPH. I’m excited to see how this offering grows over time. I’m also very interested to see how the big box storage companies respond to this over the next few years.

    12. says

      Hi Ducan, is vSAN substitute for VSA? Does it work with local disk or shared sotrage? More one: what kind of datastore does it support? NFS, iSCSI, FC?
      Thanks for share your knowledge

      • says

        1) no it is not a substitute for VSA (for now)
        2) local disks only
        3) it will surface up a shared datastore using a VMware proprietary protocol, so not NFS/iSCSI/FC

    13. Ratnadeep says

      Thank you for the post Duncan. But could you please elaborate a bit on the role of the witness?

    14. Dirk Slechten says

      I’m taking a dive in to this technology since a few days and I have many questions/remarks…

      You need to create local disk groups with local disks. All these local disk groups will be combined to one datastore. (As far as I understand you can not create 1 disk group with disks from host 1 & host 2 or is this possible?)

      Since the disk group is JBOD and not a raid set, how will the data be written to this disk group?
      Will it start filling up first disk 1, then disk 2, disk 3 and so on? What happens if you delete a vm and create a new vm? This will lead to fragmentation?

      Lets say you have 5 disks of 300GB and you have one VM of 930GB, then this vm will consume 4 disks.
      There is no raid configured here since the raid is build on virtual machine level.
      So if there is one disk failure in this disk group, you will need to use the replicated VM on HOST2 in able to continue working. (Which means here that you need at least 8 disks for this vm only…)

      There is mentioned that if one disk failes in the disk group, only the data residing on that disks needs to be rebuild? So is vSan that intelligent that it knows which blocks it needs to replicate back from host 2 to host 1, or does the complete VM needs to be replicated back?

      Is the replication synchronous or asynchronous? Didn’t find any information about this

      The maximum size of a VMDK is 2TB. Does the disk group needs to be bigger or can you place one VMDK over 2 smaller disk groups?

      As hardware is so advanced nowadays you can get pre-deticve failures on disk level in your storage box. You can replace the disk and you will have a performance impact during the rebuild, but you can continue working.

      But when you have a predictive failure on a disk in vSan you can consider it as a disk failure as there is no raid protection in your disk group…..so you need to start working on the replicated vm on host 2.

      • duncan@yellow-bricks says

        as a start I would recommend the other posts on the topic of VSAN to be found here: vmwa.re/vsan