Yellow Bricks

by Duncan Epping

Introduction to vSphere Flash Read Cache aka vFlash

· ·

vSphere 5.5 was just announced and of course there are a bunch of new features in there. One of the features which I think people will appreciate is vSphere Flash Read Cache (vFRC), formerly known as vFlash. vFlash was tech previewed last year at VMworld and I recall it being a very popular session. In the last 6-12 months host local caching solutions have definitely become more popular and interesting as SSD prices keep dropping and thus investing in local SSD drives to offload IO gets more and more interesting. Before anyone asks, I am not going to do a comparison with any of the other host local caching solutions out there. I don’t think I am the right person for that as I am obviously biased.

As stated, vSphere Flash Read Cache is a brand new feature which is part of vSphere 5.5. It allows you to leverage host local SSDs and turn that in to a caching layer for your virtual machines. The biggest benefit of using host local SSDs of course is the offload of IO from the SAN to the local SSD. Every read IO that doesn’t need to go to your storage system means resources can be used for other things, like for instance write IO. That is probably the one caveat I will need to call out, it is “write through” caching only at this point, so essential a read cache system. Now, by offloading reads, potentially it could help improving write performance… This is not a given, but could be a nice side effect.

Just a couple of things before we get in to configuring it. vFlash aggregates local flash devices in to a pool, this pool is referred too as a “virtual flash resource” in our documentation. So in other words, if you have 4 x 200 GB SSD you end up with a 800GB virtual flash resource. This virtual flash resource has a filesystem sitting on top of it called “VFFS” aka “Virtual Flash File System”. As far as I know it is a heavily flash optimized version of VMFS, but don’t pin me on this one as I haven’t broken it down yet.

So now that I know what it is and does, how do I install it, what are the requirements and limitations? Well lets start with the requirements and limitations first.

Requirements and limitations:

  • vSphere 5.5 (both ESXi and vCenter)
  • SSD Drive / Flash PCIe card
  • Maximum of 8 SSDs per VFFS
  • Maximum of 4TB physical Flash-based device size
  • Maximum of 32TB virtual Flash resource total size (8x4TB)
  • Cumulative 2TB VMDK read cache limit
  • Maximum of 400GB of virtual Flash Read Cache per Virtual Machine Disk (VMDK) file

So now that we now the requirements, how do you enable / configure it? Well as with most vSphere features these days the setup it fairly straight forward and simple. Here we go:

  • Open the vSphere Web Client
  • Go to your Host object
  • Go to “Manage” and then “Settings”
  • All the way at the bottom you should see “Flash Read Cache Resource Management”
    • Click “Add Capacity”
    • Select the appropriate SSD and click OK
      Introduction to vSphere Flash Read Cache aka vFlash
  • Now you have a cache created, repeat for other hosts in your cluster. Below is what your screen will look like after you have added the SSD.

Now you will see another option below “Flash Read Cache Resource Management” and it is called “Cache Configuration” this is for the “Swap to host cache” / “Swap to SSD” functionality that was introduced with vSphere 5.0.

Now that you have enabled vFlash on your host, what is next? Well you enable it on your virtual machine, yes I agree it would have been nice to enable it for a full cluster or for a datastore as well but this is not part of the 5.5 release unfortunately. It is something that will be added at some point in the future though. Anyway, here is how you enable it on a Virtual Machine:

  • Right click the virtual machine and select “Edit Settings”
  • Uncollapse the harddisk you want to accelerate
  • Go to “Flash Read Cache” and enter the amount of GB you want to use as a cache
    • Note there is an advanced option, at this section you can also select the block size
    • The block size could be important when you want to optimize for a particular application

Not too complex right? You enable it on your host and then on a per virtual machine level and that is it… It is included with Enterprise Plus from a licensing perspective, so those who are at the right licensing level get it “for free”.

Reader Interactions

Comments

  1. So if you have to enable it on the VM level, does that mean it will not work for Horizon View with linked clones or if you enable it on the parent image will it be enabled on clones?

  5. Duncan,

    How does this work with HA/DRS? From what I understand, you can’t vMotion a vFRC’d VM unless the destination host has enough vFRC to accommodate that VM. So if you have 100GB of cache on source host with a VM using 50GB, and you have 100GB of cache on destination host with two VMs using 75GB, you can’t move the 50GB of cache VM to the destination. If you don’t have enough vFRC on any of your other hosts for that VM to live, how will HA be able to start up the VM on another host or DRS balance the load? I’d assume that DRS would have to do some more calculations and take that aspect into performance measures. It might be able to shuffle things around, but if it does take that aspect into consideration, it seems that it would be a major factor on how to distribute load across a cluster. If you have a two node cluster, or any x number of hosts, what if you don’t have enough room to accommodate a failed host during an HA event from a vFRC perspective? Does HA disable vFRC to ensure protection first over performance?

    Also, since this is per VM configured, you’d have to ensure that you keep track of all the VMs you allocate cache for and hope it doesn’t exceed the total amount you have for each host. While each host can accommodate 32TB of cache, if you have 2 nodes or 32 nodes, you are still limited to 32TB of cache in the cluster all the while having to buy 32TB for all your hosts and then keeping track of how much you dole out to ensure you don’t go over. This seems pretty clunky but I know its a 1.0 release and it can only get better. Sorry for the barrage but anything further will help. This is a feature I’ve been waiting for without having to spend more money going 3rd party for and it doesn’t seem all that impressive yet.

    • DRS does admission control for vFlash. It will move the cache along with the VM and it will do so based on what is available on the other hand. Yes the SSD device will need sufficient resources, but I would hope everyone makes sure that is the case. With regards to HA there are caveats, I have an article about all of this soon.

  6. If you are looking to implement this, make sure you have the latest compatibility version/virtual hardware otherwise you will not have an option to use the flash.

  9. So this seems like a good use case for Value SSD drives. For example Cisco UCS B200 M3 blades. You have two 100G flash options, the Performance (~1300 list) verses the Value (~525 list). The difference here is write performance on the Performance model. So for these small blades (just 2 hdd slots) you can get 200G read cache for around $1000.

  10. Hi Duncan

    If there is a bad block on the Flash Drive, What is the :
    1. Timeout detection for bad block
    2. How long before the IO request will be redirected to Storage unit

    (Assuming no raid on the local flash storage)

    Thanks.

  11. Great article Duncan!
    One question – is it possible to see/monitor how much/or if the cache is being used?
    Eg. via esxtop or vCenter reports?
    I’m just thinking – how do I know if vFRC is actually benefitting my vm and
    how do I know when to extend the cache for the vm? (rightsizing)

    Thanks!
    Enrico

  12. Great article Duncan!
    How the vFlash accelerate the VM? the whole VM’s VMDK is copied to the SSD? or the VM’s VMDK is monitored and the frequent read block is copied to the SSD? Thanks.

  13. Hi Duncan , 1 Q , how many vflash modules we can have on host ? like vfc , vfc1, suppose we are using one ssd and one pcie ssd , I want my critcal application to use vfrc of pcie ssd and other less critical to use normal SSD , can we can any entries in vmx file of Vm to specify the type of Vflash to be used by one vmdk

      • ok , so lets forget about modules for sometime , other part of Q , suppose I have one normal ssd and one pcie ssd drive on a host , both are added to capacity of vflash , so now i need to configure one Vmdk on a VM that can use flash provided from only pcie-ssd (As this device as more IOPS than normal SSD) and there is no option in web client to see that the Vfrc is using which drive flash in VM . so now can we see or edit .vmx file of Vm for selecting particular ssd drive configured for vflash.

  14. Duncan — Thanks for the article. I noticed if a person engages vFlash on the physical ESXi 5.5 host, confirms vFlash is working, then engages the caching from the physical ESXi host on the VMDK of a nested ESXi host (5.1 or 5.5) the nested ESXi host comes up with the datastores italicized and unmounted (and not able to mount).

    Assuming I have everything working on vFlash like it’s supposed to be working on the physical ESXi host, is there any workaround to getting the VMDKs of the nested ESXi host to take advantage of it?

    Thanks.

    Datto

  15. One thing that is important to note is that you must have your VM Virtual Hardware at Level 10 in order to see the Virtual Cache Read Cache in the Edit Settings of the Virtual Machine. If you have an earlier version of Virtual Hardware, the option won’t show up.

    Jim

  16. Duncan, Would i be right to assume that an SSD will not eligible if ESXi boots from it? I am unable to see the SSD that contains the ESXi boot partition in the list of available devices.

    Many thanks

    Mark

      • Duncan. in addition to this i found that the flash Eligibility was being “Detected as a remote SAS SSD”

        ~ # esxcli storage vflash device list
        Name Size Is Local Is Used in vflash Eligibility
        ———————————— —– ——– —————– ———————————
        naa.600508b1001c5c242c2805a4ed45beb6 95362 false false Detected as a remote SAS SSD

        I ran the following to resolve the issue.

        esxcli storage nmp satp rule add –satp VMW_SATP_LOCAL –device naa.600508b1001c5c242c2805a4ed45beb6 –option enable_local

        esxcli storage core claiming reclaim -d naa.600508b1001c5c242c2805a4ed45beb6

        • I am seeing the same thing. When I issue the same commands I get an error message.

          ~ # esxcli storage vflash device list
          Name Size Is Local Is Used in vflash Eligibility
          ———————————— —— ——– —————– —————————-
          naa.600508b1001c6242ed09cede84320808 763065 false false Detected as a remote SAS SSD
          ~ #

          Is Local is saying false, but the SSD is really local to the machine.

          I’ve ran:
          esxcli storage nmp satp rule add –satp VMW_SATP_LOCAL –device naa.600508b1001c6242ed09cede84320808 –option enable_local

          esxcli storage core claiming reclaim -d naa.600508b1001c6242ed09cede84320808

          Then I get an error message:
          Unable to unclaim path vmhba3:C0:T0:L8 on device naa.600508b1001c6242ed09cede84320808. Some paths may be left in an unclaimed state. You will need to claim them manually using the appropriate commands or wait for periodic path claiming to reclaim them automatically.

          Any thoughts as how to get past this?

  18. @Tr03bor

    Hey,

    I’ve been looking at vFlash and think its awesome! I have a few questions you might be able to help with?

    I had a VM located on SSD storage, the read speed for 4k blocks was around 540MB/sec. I then enabled vFlash just to see what would happen, interestingly the read speed decreased down to 280MB/sec (interesting the write speed of the SSD being used as the vFlash cache).

    I’ve read the documentation and it states “Data writes are dispatched to the backing storage, such as a SAN or NAS”, which makes sense, however from what I can see it strangely appears as if the writes are in fact traversing the vFlash device?

    Am I losing my mind? 🙂

    Doc – http://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.storage.doc%2FGUID-07ADB946-2337-4642-B660-34212F237E71.html

    • This sounds like it copies the data to the SSD upon reads.
      If that’s the case you will not have luck with value SSDs, but in fact need one that very quickly handles writes so it won’t lower the read performance.

      Same was the issue with Facebook Flashcache.

  19. First of all i just want to thank Duncan for this article and all of the other posts regarding this topic. i found all of this information incredibly helpful and was looking forward to using vFlash cache to help with some latency issues, but i wanted to point out an issue i encountered which took my vCenter down and caused a ton of issues so no one has to go through what i did because it was very unpleasant since my vCenter suffered from all of the symptoms outlined in the KB article and also prevented my backups from working since the inventory service DB got corrupted.

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2072392

    • Yeah, too bad we didn’t see your comment/kb article before.. we have the same issue at the moment..

  20. Hello Duncan,

    What happens if I enable vfrc on virtual disks which is already configured with customized iops limit on it. Will it cause any performance issues. First of all, is it recommended to use the vfrc on the virtual disk where the customized iops values are already assigned/configured on it.

    What happens if I add customized IOPs values onto the filed: Limit IOPs under VM edit settings which is right above the VFRC field. Because as we generally use the option unlimited.

    Suppose if I have the Limit IOPs value is already set to 100 for the virtual disk. Will it be fine/recommended if I enable VFRC on the same virtual disk?

  21. How would VFRC affect an environment running VSAN as the storage provider? Would it be a waste of resources, would it possibly cause complications with the way VSAN handles/provides buffering and/or is it compatible/tested?…thoughts?