• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Yellow Bricks

by Duncan Epping

  • Home
  • ESXTOP
  • Stickers/Shirts
  • Privacy Policy
  • About
  • Show Search
Hide Search

Frequently asked questions about vSphere Flash Read Cache

Duncan Epping · Sep 11, 2013 ·

Last week I received many emails on the topic of vFlash so I figured I would start collecting these questions and answer them in a “frequently asked questions about vSphere Flash Read Cache” article. That way the info it is out in the open and it should be fairly easy to find using google. If you haven’t done so yet, I would recommend starting by reading my introduction to vSphere Flash Read Cache. If you already have, here are the questions (and answers) I received so far:

  • In a cluster where some hosts have vFlash resources and others do not, can I vMotion a VM with vFlash assigned to a host without vFlash resources?
    • In order to successfully migrate a VM vFlash resources are required on the destination host
  • When migrating a VM I have various options, in essence I can move the cache or drop it. What is the impact of moving the cache?
    • When selecting to move the cache essential an “X-vMotion” (cross vMotion / non-shared-disk migration) is performed. Meaning that the “local” vFlash cache file that is stored on the VFFS filesystem is copied from source to destination. This results in the migration taking longer, but also means that the cache is hot instantly and does not need to be warmed up (impact of dropping the cache is the warm up that needs to happen again).
  • What are the requirements and considerations to run vSphere Flash Read Cache from a VM/Guest perspective?
    • The requirements from a guest perspective are minimal, other than VM hardware level 10 (5.5 compatiblity) there are no requirements. Considerations are also minimal, of course the IO pattern will matter, if you do 90% writes then the benefit will be minimal. If you have a specific IO size then a consideration could be changing the block size.
  • Will a VM be moved by DRS when it has vFlash enabled on a disk?
    • DRS treats a virtual machine with vFlash enabled as if “should vm-host” affinity rules are applied. Meaning that it will only move a virtual machine when absolutely needed, example being when entering maintenance mode or when it is the only way to solve over-utilization. (Similar to “low latency VM” functionality)
  • What additional considerations (if any) need to be made for backing up vFRC supported guests; at guest and/or VM layer?
    • There are no direct considerations with regards to backup. Considering vSphere Flash Read Cache is “write-through” there is no such a thing as “dirty data”. Both cache and storage will be in-sync at all times. One thing to realize is though that when you do a full VM image level backup, the VM will need to be restored to a host that has vFlash enabled in order for it to be powered on. Also note that with other solutions which do write-back back-up could be a potential caveat / concern. Discuss the potential impact with those vendors as each implementation differs.
  • What is the purpose of the “block size” which I can specify in the advanced settings? What is the minimum and maximum size?
    • The block size can be configured from 4KB to 1024KB. In order to optimize for I/O performance and minimize overhead try to align the vFlash blocksize to the blocksize used by your Guest OS / Application. You can leverage a tool like vscsiStats to identify the blocksize used by your virtual machine(s).
  • What happens to a virtual machine which is accelerated and the flash device on the host it is running on fails?
    • When a flash device fails which is being used for vFlash and a VM is actively using it then the VM will continue running but will experience a performance degradation as all reads will need to come from the storage device instead of SSD. In other words, an SSD failure does not cause any form of disruption to the workload but could potentially result in a degraded performance (similar to the performance before enabling vFlash)
  • What if I run out of capacity on my flash device?
    • When a flash device runs out of capacity you will simply not be able to reserve flash resources for new virtual machines or virtual machines for which you want to allocate flash resources.
  • Does vSphere Flash Read Cache do anything for writes like other solutions do?
    • Well I guess the short answer is “no, not directly”. It is called “vSphere Flash Read Cache” for a reason I guess… But when you think about it, although it doesn’t provide write-back caching the “read caching” portion will free up expensive array resources used for those reads. These expensive resources can now be used for… indeed… writes. So although vSphere Flash Read Cache does not provide write-back caching, only write-through, it might lead to accelerated writes simply because of the decrease of stress on your storage system.
  • Is it possible to configure vSphere Flash Read Cache on a virtual machine with Raw Device Mappings (RDM)?
    • vSphere Flash Read Cache does not support Physical RDMs vSphere Flash Read Cache does support Virtual RDMs.
  • Can I use vFlash on a virtual machine which is stored on a VSAN datastore?
    • No you cannot enable vFlash on a virtual machine which is stored on a VSAN datastore. Considering VSAN already provides read caching and write bufferering there is no real value in enabling that second layer of caching and subsequently add complexity along with it.
  • If I have VSAN enabled and also a NAS/SAN datastore, can my VM that is running on the NAS/SAN datastore be enabled for vFRC?
    • Yes, when the VM is stored on the NAS/SAN datastore you can enable vFRC for this particular VM.
  • Is there a minimum size for vSphere Flash Read Cache allocation to a virtual machine?
    • The minimum size for vSphere Flash Read cache is dependent on the selected blocksize. With a 4KB blocksize the minimum is 4MB, and with a 1MB blocksize the minimum is 276MB
  • What happens if I have my block size configured to 1024KB and I do a 4KB IO?
    • Within that 1024KB cache block your 4KB IO will be stored, the rest of the block (1020KB in this scenario) will be marked as “empty”. Hence the reason it is important to align your IO block size with your cache block size to avoid wastage.
  • What happens to my cache when I resize it on a powered-on VM?
    • When the cache is resized all cache is discarded. This results in a period of time where the cache will need to warm up again.
  • If a host fails and virtual machines with vFRC enabled need to be failed over and there are no sufficient resources available what happens?
    • vFRC allocations are reservations. The behavior is similar to that of a CPU or memory reservation. When there is not enough unreserved capacity available then HA will not be able to power-on the virtual machine.
  • When I configure vSphere Flash Read Cache the maximum cache size seems to be limited to 200GB, what is supported and how do I change this?
    • The maximum cache size per virtual machine is 400GB, however in the current release the cache size is limited to 200GB. You can however change the advanced setting called “VFLASH.MacCacheFileSizeMB” to 409600 if desired. Do note that you will need to change this setting on every host if you want the virtual machine to be able to vMotion between hosts or want vSphere HA to be able to restart a virtual machine.
  • How do I monitor a VM to see how much cache is being used? Is there anything in vCenter or ESXTOP to help me?
    • vCenter contains multiple metrics on a virtual machine level. Just look at the advanced section and more explicitly the virtual disk counters starting with “Virtual Flash …”. There are also various helpful metrics to be found using “esxcli storage vflash cache stats get -c <cache name>. This will for instance provide you with the cache hit rate, latency etc. I have not found any esxtop counters yet.
  • When a virtual machine with vSphere Flash Read Cache enabled is vMotioned including the cache which network is used to transfer the cache?
    • When a VM is migrated / vMotioned including the cache then the “XvMotion / enhance vMotion / non-shared migration” logic is used to move the cache content from source to destination. The data is transferred over the vMotion network. Ensure that sufficient bandwidth is available to shorten the migration time.
  • When flash content is migrated along with the virtual machine, will the migration process make a distinction between hot and cold cache?
    • XvMotion is responsible for migrating the cache from source to destination. Today it doesn’t know the difference and will copy the full file from source to destination. Meaning that if you have a cache size of 10GB the full 10GB will be copied regardless of it being used or not.
  • Can I use the same disk that is divided into partition for vSphere Flash Read Cache & VSAN SSD tier?
    • This is not supported! Both VSAN and vSphere Flash Read Cache require there own flash device.
  • The SSD in my host is being reported in vSphere as “non-SSD”. According to support this is a known issue with the generation of server I am using. Will this “mis-reporting” of the disk type affect my ability to configure a vFlash? <added 17/Sept/2013>
    • Yes it will, you will need to tag the SSD as local using (example below is what I use in my lab, your identifier will be different). And in this case I claim it as being “local” and as “SSD”.
      esxcli storage nmp satp rule add –satp VMW_SATP_LOCAL –device mpx.vmhba2:C0:T0:L0 –option “enable_local enable_ssd”
  • If I disable Strict Admission Control for vSphere HA, would it be able to power-on a VM with vFRC enabled? <added 17/Sept/2013>
    • No, admission control does not have anything to do with powering on virtual machines. Even if you disable admission control and there are no sufficient flash resources available then a virtual machine with vFRC enabled cannot be powered on.
    • Also note, that HA Admission Control does not take vFRC resources in to account whatsoever. So in an environment running vFRC you can technically use 100% of all flash resources, whilst Admission Control was setup in an N+1 fashion.
  • Is vFRC as good as the caching solution by vendor X? <added 18/Sept/2013>
    • I have had this question many times. It is difficult to answer as it depends. It depends on:
      • Your requirements from a functional perspective (read vs write back caching)
      • Type of workload you are running (read vs write, VDI vs Server)
      • Support requirements (Not all solutions out there are on the VMware compatibility list)
      • Operational requirements (vFRC needs to be enabled per virtual disk, some vendors do per datastore for instance etc)
      • Budget and even current vSphere licenses used (vFRC is included with Enterprise Plus)
  • x

If you have any other questions, don’t hesitate to drop them here and I will add them to the FAQ when applicable to the broader audience.

Related

Server, Software Defined, Storage vflash, vfrc, vsphere flash read cache

Reader Interactions

Comments

  1. Markus says

    11 September, 2013 at 17:30

    Is it possble ro add Read Cache to a RAW-Device?

    • Duncan Epping says

      11 September, 2013 at 20:26

      Good question, see article now for answer.

  2. Niran Even-Chen says

    11 September, 2013 at 22:21

    Duncan, When you say “One thing to realize is though that when you do a full VM image level backup, the VM will need to be restored to a host that has vFlash enabled in order for it to be powered on”, can you elaborate? If the cache is write through then why is the full VM image restore depended on it?

    • Duncan says

      11 September, 2013 at 23:24

      Because the VM configuration has vFlash enabled. Of course you can take it out, but that is an extra step. Just something to be aware off operationally speaking

      • Bas says

        12 September, 2013 at 16:28

        I guess the same problem will occur during a HA event if not all the host do have vFlash resources?

        • Duncan Epping says

          12 September, 2013 at 16:47

          HA will try to power-on the VM on a host which has flash resources available. If they are unavailable the power-on attempt will fail.

      • Brent says

        29 August, 2014 at 17:59

        “Because the VM configuration has vFlash enabled. Of course you can take it out, but that is an extra step. Just something to be aware off operationally speaking”
        — How do you take it out? Do I just edit the machine and set the vfrc to 0, then it will start? I’d like to evaluate vfrc on one host and if that host fails, I’d need to be able to start that vm on host without any vfrc.

  3. Luke D says

    13 September, 2013 at 04:24

    “vFRC allocations are reservations.” I’m concerned that this could be quite cumbersome to manage with large numbers of VMs with different workload characteristics.

    I’m thinking out loud.. how about using relative “shares” combined with analysis of “hot” blocks for a sort of weighted MRU approach. Even better — vFRC resource pools. I guess it’s still early days. Anyone know what the other vendors are doing?

    • Duncan Epping says

      13 September, 2013 at 06:06

      Agreed that that would be a more flexible solution, this is indeed something which is being considered for the future.

  4. Nyh says

    13 September, 2013 at 14:25

    I suspect that the answer is no, but… Can I use the same disk that is divided into partition for Read Cache & vSan SSD tier?

    • Duncan Epping says

      13 September, 2013 at 14:51

      No you cannot, but second time I received this question today… will add to the list 🙂

  5. iwan rahabok says

    15 September, 2013 at 16:30

    Thanks Duncan.
    If the ESXi host has both VSAN and vFlash, does it mean the block is cached 2x? I hope VSAN will be vFlash aware and intelligently use the vFlash for Read cache instead of duplicating it. This can help reduce the SSD cost as it’s not exactly cheap since each host will need 2.

    • Duncan says

      15 September, 2013 at 18:42

      You cannot enable vFlash on a VM which is stored on a VSAN datastore…

      • Remy Zandwijk says

        19 September, 2013 at 13:24

        So “No you cannot enable vFlash on a virtual machine which is stored on a virtual machine.” in the FAQ should be “No you cannot enable vFlash on a virtual machine which is stored on a VSAN datastore”. Right?

  6. Tim says

    18 September, 2013 at 22:45

    Does the caveat regarding image level backups apply to VMs replicated using vSphere Replication as well? IOW, would the target site require hosts to be vFlash enabled too?

    • Duncan Epping says

      18 September, 2013 at 23:17

      I haven’t tested it yet, but I would expect so. Of course you can simply reconfigure it if you want.

  7. Magnus says

    25 September, 2013 at 15:40

    It seems vscsistats can’t handle hosts with lots of VMs (75+). We get an overflow error trying to use vscsistats to get block size. It’s a shame since these are the type of hosts (4CPU, 1 TB RAM) that could benefit from the flash cache. We are on 5.1 build 1065491

    Example error:
    ~ # vscsiStats -l
    VscsiStats: Failed instance list get: worldGroups
    vscsiStats: exiting with error
    VscsiStats error code 0xbad0093: VSI GetList handler overflow

    • Duncan Epping says

      30 September, 2013 at 16:07

      I haven’t tested it at that scale to be honest… Will see if anyone here has.

  8. Tyson Then says

    27 September, 2013 at 15:37

    Hi Duncan,
    Thanks for the info.
    I read that vFRC support 8 SSDs per host. I can’t seem to find how multiple SSDs are aggregated. Is data striped across multiple drives like RAID0 with aggregated performance? Or just the combined storage across the drives with data “spilling” across drives.
    Thanks again for your informative posts.

    • Duncan Epping says

      30 September, 2013 at 16:26

      There is no striping of data with vFRC. All cache of a single VMDK will reside on a single SSD. (All flash caching solutions take this approach.)

  9. Johan Blom says

    30 September, 2013 at 13:32

    Can vFlash be used with VSA?

    • Duncan Epping says

      30 September, 2013 at 16:14

      Yes you can.

  10. Trace says

    3 October, 2013 at 18:55

    I know this is a silly question, but could vFRC use SAN attached SSD?

    • Duncan Epping says

      3 October, 2013 at 20:17

      You can, you will have to set it using “esxcli” as “local” though. Keep in mind though that the network latency will diminish some of the expected results.

  11. karan gandhi says

    8 October, 2013 at 08:20

    Hi Duncan , 1 Q , how many vflash modules we can have on host ? like vfc , vfc1, suppose we are using one ssd and one pcie ssd , I want my critcal application to use vfrc of pcie ssd and other less critical to use normal SSD , can we can any entries in vmx file of Vm to specify the type of Vflash to be used by one vmdk

  12. Bjørn-Tore says

    25 October, 2013 at 13:28

    Duncan, I dont get my local SSD disks up in the vFlash Add Capacity window. Is there anything I must do on server before they show up and becomes eligible SSD devices ? I could not find so much info on this.

    • Duncan Epping says

      25 October, 2013 at 14:44

      Look at the “esxcli storage nmp satp” example, your device is probably not listed as “local” or as “ssd”

      • Bjørn-Tore says

        27 October, 2013 at 16:23

        Thanx, that solved the problem. Should the SSD devices used for vFlash be setup in RAID1 config? What is best practice ?

        • Duncan Epping says

          27 October, 2013 at 16:35

          Most people don’t do raid with SSDs as flash, just serve it up directly

  13. sujith says

    7 November, 2013 at 15:22

    Can i enable/disable or manager vFRC via a vCenter windows version or is it mandator to use the web client.

    • Duncan Epping says

      9 November, 2013 at 13:25

      Webclient

  14. Sujith says

    14 November, 2013 at 08:52

    Thank you Duncan , one more thing please.
    My ‘esxcli storage core device list ‘ does return “Is SSD: true” I see it as “SSD” type in the vCenter now. But it doesn’t list in the vFRC add capacity window .

    Device Display Name: Local LSI Disk (naa.60030057014488c019165ec007d2de99)
    Storage Array Type: VMW_SATP_LOCAL
    Storage Array Type Device Config: SATP VMW_SATP_LOCAL does not support device configuration.
    Path Selection Policy: VMW_PSP_FIXED
    Path Selection Policy Device Config: {preferred=vmhba1:C2:T0:L0;current=vmhba1:C2:T0:L0}
    Path Selection Policy Device Custom Config:
    Working Paths: vmhba1:C2:T0:L0
    Is Local SAS Device: false
    Is Boot USB Device: false

    ( reboot dint help )
    Do you think there’s something in addition to this i need to do.I was initially planning on using a vendor PCI-e SSD flash card.

    – thanks

    • Duncan Epping says

      14 November, 2013 at 10:07

      Strange, I would suggest marking it as local and it should work. Which Flash/SSD is this?

      • Sujith says

        19 November, 2013 at 17:51

        Hey , thank you !! . That worked.
        its the local HDD

  15. Paul_P says

    14 November, 2013 at 10:54

    As vFRC enabled VMs which are backed up need to be restored to a vFRC enabled host, is this the same for SRM replicated VMs too? Either with vSphere replication or using an array replication technology?

    Thanks.

  16. Mark says

    14 November, 2013 at 20:12

    Hi Duncan,
    What command to make the SSD local ?
    I tried the following command, but don’t work.

    esxcli storage nmp satp rule add –satp VMW_SATP_LOCAL –device ‘diskname’ –option “enable_local enable_ssd”

    thanks

    • Mark says

      14 November, 2013 at 20:37

      Hi Duncan,
      It works after reboot the ESX.
      Thanks

  17. Frank says

    15 November, 2013 at 20:44

    Duncan,

    I have successfully added SSD capacity within VFRC, but the menu option to grant specific VMDKs access to the read cache is missing in my web client. Any ideas?

  18. Eibla says

    21 February, 2014 at 11:29

    Can I use my boot disk for VFRC? Its like 80GB of EFD and vSphere only consumes a little part of it.

  19. Mike says

    18 April, 2014 at 17:37

    What is the risk associated with an SSD failure when using VRFC ?

  20. Hung Vu says

    15 May, 2014 at 09:50

    Hi there,
    I read and understand Host Swap Cache to VFlash (VFFS) replace the function swap-to-SSD of earlier ESXi-5.0 to over subcribe of Memory of ESX-Host to VMs.
    But I configure Host-Swap-Cache and make a test of over subcribe to a VM, which das it’s datastore reside on a SAN-LUN, it doesn’ work as expected.
    Says My ESX-Host has 100GB Mem.
    I create a Host-Swap-Cache 30GB.
    My VM’s config hat as following resource allocations:
    Shares: 120GB
    Reservation: 100GB (As bis as the physical Mem od ESX-Host)
    Limit: 120GB
    After powered on the VM, I see a 20GB .vswp-file on the datastore of the VM in SAN.
    Why do the over subcribed 20GB not reside in the VFFS of Host-Swap-Cache ?

    Many thanks for hind.!
    HungVu

  21. Brandon says

    21 May, 2014 at 22:31

    Duncan,
    What would happen to the VM if the SSD where it’s read cache was stored failed? Would the VM still continue running just fine or would it fail? Would ESXi rebuild the cache on another SSD if one were available?

    • [email protected] says

      22 May, 2014 at 09:28

      The VM would continue to work just fine, with performance degraded to normal disk speeds

    • Tony says

      5 July, 2014 at 23:19

      If i have 3 SSD (eg3*450GB ) I could server up as raid5 so one failure keeps things nice and sweet. However reading the above it seems just leaving as none raided seems to be okay… If I have 3 drives in the host and one fails and there is enough space left, would the cache just be rewarmed on one of the other drives in the host? Thanks

  22. alket says

    11 July, 2014 at 20:58

    Hey Duncan,

    What’s the heuristic for how the cache gets populated and how blocks get evicted? I am running a simple test replaying an i/o trace but not able to make sense of the numbers yet.

    Thanks
    Alket

    • Duncan Epping says

      12 July, 2014 at 18:43

      Don’t know either Alket, will ask.

  23. Ash says

    27 August, 2014 at 17:24

    Duncan,

    Thank you for the info,

    This is what I am currently facing , once I allocate vFRC to a Vm(I gave about 10GB) , it started performing way too slow, Slower then without Having the vFRC, then I found the Vmware KB 2057219 which said
    “The cache had sufficient time to build. It can take quite some time for the Virtual Flash cache to build.” , So i gave it some time(Apporx 20 min ) then when I checked the performance It was considerably fast and better than what I expected.

    My question is , is there any way we can check with the Cache has completed build , is there any commands which can help in Monitor the cache build status ?

  24. jasonth says

    4 September, 2014 at 22:57

    Duncan, I couldn’t find a clear answer to this, so can you clarify? If I have a blade with two 250GB SSD, and I use them in a RAID 1 to boot ESXi from, can the remaining space that would normally be a VMFS be used for vFRC storage?

  25. David Culotta says

    23 September, 2014 at 18:03

    Duncan,

    In Enterprise HW it appears that you cannot serve it directly to the OS.

    Scenario: 1
    Hardware = HP BL460c G8 using P220I controller and two 800gb intel SSD’s for vFRC cache while booting from SD card. The P220i will not allow pass thru or direct connect. At least it doesn’t appear to.

    Scenario: 2
    Hardware = HP DL380 G8 using P420i controller and two 800gb intel SSD’s for vFRC cache and two 147gb SAS drives to boot the OS. Same story I cant find a way to directly present these to ESXi.

    So is there a method to directly connect the drives that I’m not aware of? What would be the ideal setup in the event that there isn’t a way to directly present the drives to ESXi? One array RAID0 with one 1.45tb Volume. One array RAID0 with Two 745gb Volumes? Or Two array’s RAID0 with Two 745gb Volumes?

    Thanks!

Primary Sidebar

About the author

Duncan Epping is a Chief Technologist in the Office of CTO of the Cloud Platform BU at VMware. He is a VCDX (# 007), the author of the "vSAN Deep Dive", the “vSphere Clustering Technical Deep Dive” series, and the host of the "Unexplored Territory" podcast.

Upcoming Events

May 24th – VMUG Poland
June 1st – VMUG Belgium

Recommended Reads

Sponsors

Want to support Yellow-Bricks? Buy an advert!

Advertisements

Copyright Yellow-Bricks.com © 2023 · Log in