Yellow Bricks

by Duncan Epping

vfrc

Disk Controller features and Queue Depth?

· ·

I have been working on various VSAN configurations and a question that always comes up is what are my disk controller features and queue depth for controller X? (Local disks, not FC based…) Note that this is not only useful to know when using VSAN, but also when you are planning on doing host local caching with solutions like PernixData FVP or SanDisk FlashSoft for instance. The controller used can impact the performance, and a really low queue depth will result in a lower performance, it is as simple as that.

** NOTE: This post is not about VSAN disk controllers, but rather about disk controllers and their queue depth. Always check the HCL before buying! **

I have found myself digging through documentation and doing searches on the internet until I stumbled across the following website. I figured I would share the link with you, as it will help you (especially consultants) when you need to go through this exercise multiple times:

http://forums.servethehome.com/index.php?threads/lsi-raid-controller-and-hba-complete-listing-plus-oem-models.599/

Just as an example, the Dell H200 Integrated disk controller is on the VSAN HCL. According to the website above it is based on the LSI 2008 and provides the following feature set: 2×4 port internal SAS, no cache, no BBU, RAID 0, 1 and 10. According to the VSAN HCL also provides “Virtual SAN Pass-Through”. I guess the only info missing is queue depth of the controller. I have not been able to find a good source for this. So I figured I would make this thread a source for that info.

Before we dive in to that, I want to show something which is also important to realize. Some controllers take: SAS / NL-SAS and SATA. Although typically the price difference between SATA and NL-SAS is neglectable, the queue depth difference is not. Erik Bussink was kind enough to provide me with these details of one of the controllers he is using as an example, first in the list is “RAID” device – second is SATA and third SAS… As you can see SAS is the clear winner here, and that includes NL-SAS drives.

mpt2sas_raid_queue_depth: int
     Max RAID Device Queue Depth (default=128)
  mpt2sas_sata_queue_depth: int
     Max SATA Device Queue Depth (default=32)
  mpt2sas_sas_queue_depth: int
     Max SAS Device Queue Depth (default=254)

If you want to contribute, please take the following steps and report the Vendor, Controller type and aqlength in a comment please.

  1. Run the esxtop command on the ESXi shell / SSH session
  2. Press d
  3. Press f and select Queue Stats (d)
  4. The value listed under AQLEN is the queue depth of the storage adapter

The following table shows the Vendor, Controller and Queue Depth. Note that this is based on what we (my readers and I) have witnessed in our labs and results my vary depending on the firmware and driver used. Make sure to check the VSAN HCL for the supported driver / firmware version, note that not all controllers below are on the VSAN HCL, this is a “generic” list as I want it to serve multiple use cases.

Generally speaking it is recommended to use a disk controller with a queue depth > 256 when used for VSAN or “host local caching” solutions.

Vendor Disk Controller Queue Depth
Adaptec RAID 2405 504
Dell (R610) SAS 6/iR 127
Dell PERC 6/i 925
Dell PERC H200 Integrated 600
Dell PERC H310 25
Dell PERC H330 256
Dell (M710HD) PERC H200 Embedded 499
Dell (M910) PERC H700 Modular 975
Dell PERC H700 Integrated 975
Dell (M620) PERC H710 Mini 975
Dell (T620) PERC H710 Adapter 975
Dell (T620) PERC H710p 975
Dell PERC H810 975
HP Smart Array B110i 1020
HP Smart Array B120i 31
HP Smart Array P220i 1020
HP Smart Array P400i 128
HP Smart Array P410i 1020
HP Smart Array P420i 1011
HP Smart Array P440ar 1020
HP Smart Array P700m 1200
IBM ServeRAID-M5015 965
IBM ServeRAID-M5016 975
IBM ServeRAID-M5110 975
Intel C602 AHCI (Patsburg) 31 (per port)
Intel C602 SCU (Patsburg) 256
Intel RMS25KB040 600
LSI 2004 25
LSI 2008 25 / 600 (firmware dependent!)
LSI 2108 600
LSI 2208 600
LSI 2308 600
LSI 3008 600
LSI 9271-8i 975
LSI 9300-8i 600

Something to know about vSphere Flash Read Cache

· ·

When I was looking in to vSphere Flash Read Cache (part of vSphere 5.5) there was one thing that had my interest, how does it interact with vSphere HA and DRS and more specifically, are there any caveats? It all started with the question what are the requirements for a virtual machine to be successfully restarted by vSphere HA?

The answer was simple, when you define a vSphere Flash Read Cache size for a virtual disk on a virtual machine, that amount of cache capacity defined for that virtual disk needs to be available on a local flash resource in order for the VM to be restarted / powered-on. So what does this mean? Well it means that when you set a flash read cache for a given virtual disk to 4GB, that 4GB needs to be available on your local host where the VM will be powered on. But what in the case of an HA initiated restart? Will HA ignore this requirement during restarts or will it try to guarantee the same performance? [Read more…] about Something to know about vSphere Flash Read Cache

vSphere Flash Read Cache and esxcli

· ·

As most features these days in vSphere you can configure them using the awesome esxcli command. I’ve already mentioned esxcli in my vSphere Flash Read Cache FAQ blog but I wanted to call it out explicitly here as I found it very useful. You can get some nice details using the esxcli command. So where do we start?

First thing would be:

esxcli storage vflash

This will return that there are 3 namespaces: cache, module and device. Lets start top down with device. The command “esxcli storage vflash device list” will show you a list of all flash devices and whether it has been configured for vFRC or not. The module namespace can provide you some more details around for instance cache blocksizes etc. If you run the command this is what the output looks like:

~ # esxcli storage vflash module get
 Min Supported Module Version: 1.0.0.0
 Revision: 1.0.0.0
 Supported Cache Block Size Max: 1048576
 Supported Cache Block Size Min: 4096
 Supported Cache Size Max: 214748364800
 Supported Cache Size Min: 4194304
 Supported Disk Size Max: 17592186044416
 Supported Mode Mask: WriteThru

[Read more…] about vSphere Flash Read Cache and esxcli

Frequently asked questions about vSphere Flash Read Cache

· ·

Last week I received many emails on the topic of vFlash so I figured I would start collecting these questions and answer them in a “frequently asked questions about vSphere Flash Read Cache” article. That way the info it is out in the open and it should be fairly easy to find using google. If you haven’t done so yet, I would recommend starting by reading my introduction to vSphere Flash Read Cache. If you already have, here are the questions (and answers) I received so far:

  • In a cluster where some hosts have vFlash resources and others do not, can I vMotion a VM with vFlash assigned to a host without vFlash resources?
    • In order to successfully migrate a VM vFlash resources are required on the destination host
  • When migrating a VM I have various options, in essence I can move the cache or drop it. What is the impact of moving the cache?
    • When selecting to move the cache essential an “X-vMotion” (cross vMotion / non-shared-disk migration) is performed. Meaning that the “local” vFlash cache file that is stored on the VFFS filesystem is copied from source to destination. This results in the migration taking longer, but also means that the cache is hot instantly and does not need to be warmed up (impact of dropping the cache is the warm up that needs to happen again).
  • What are the requirements and considerations to run vSphere Flash Read Cache from a VM/Guest perspective?
    • The requirements from a guest perspective are minimal, other than VM hardware level 10 (5.5 compatiblity) there are no requirements. Considerations are also minimal, of course the IO pattern will matter, if you do 90% writes then the benefit will be minimal. If you have a specific IO size then a consideration could be changing the block size.
  • Will a VM be moved by DRS when it has vFlash enabled on a disk?
    • DRS treats a virtual machine with vFlash enabled as if “should vm-host” affinity rules are applied. Meaning that it will only move a virtual machine when absolutely needed, example being when entering maintenance mode or when it is the only way to solve over-utilization. (Similar to “low latency VM” functionality)
  • What additional considerations (if any) need to be made for backing up vFRC supported guests; at guest and/or VM layer?
    • There are no direct considerations with regards to backup. Considering vSphere Flash Read Cache is “write-through” there is no such a thing as “dirty data”. Both cache and storage will be in-sync at all times. One thing to realize is though that when you do a full VM image level backup, the VM will need to be restored to a host that has vFlash enabled in order for it to be powered on. Also note that with other solutions which do write-back back-up could be a potential caveat / concern. Discuss the potential impact with those vendors as each implementation differs.
  • What is the purpose of the “block size” which I can specify in the advanced settings? What is the minimum and maximum size?
    • The block size can be configured from 4KB to 1024KB. In order to optimize for I/O performance and minimize overhead try to align the vFlash blocksize to the blocksize used by your Guest OS / Application. You can leverage a tool like vscsiStats to identify the blocksize used by your virtual machine(s).
  • What happens to a virtual machine which is accelerated and the flash device on the host it is running on fails?
    • When a flash device fails which is being used for vFlash and a VM is actively using it then the VM will continue running but will experience a performance degradation as all reads will need to come from the storage device instead of SSD. In other words, an SSD failure does not cause any form of disruption to the workload but could potentially result in a degraded performance (similar to the performance before enabling vFlash)
  • What if I run out of capacity on my flash device?
    • When a flash device runs out of capacity you will simply not be able to reserve flash resources for new virtual machines or virtual machines for which you want to allocate flash resources.
  • Does vSphere Flash Read Cache do anything for writes like other solutions do?
    • Well I guess the short answer is “no, not directly”. It is called “vSphere Flash Read Cache” for a reason I guess… But when you think about it, although it doesn’t provide write-back caching the “read caching” portion will free up expensive array resources used for those reads. These expensive resources can now be used for… indeed… writes. So although vSphere Flash Read Cache does not provide write-back caching, only write-through, it might lead to accelerated writes simply because of the decrease of stress on your storage system.
  • Is it possible to configure vSphere Flash Read Cache on a virtual machine with Raw Device Mappings (RDM)?
    • vSphere Flash Read Cache does not support Physical RDMs vSphere Flash Read Cache does support Virtual RDMs.
  • Can I use vFlash on a virtual machine which is stored on a VSAN datastore?
    • No you cannot enable vFlash on a virtual machine which is stored on a VSAN datastore. Considering VSAN already provides read caching and write bufferering there is no real value in enabling that second layer of caching and subsequently add complexity along with it.
  • If I have VSAN enabled and also a NAS/SAN datastore, can my VM that is running on the NAS/SAN datastore be enabled for vFRC?
    • Yes, when the VM is stored on the NAS/SAN datastore you can enable vFRC for this particular VM.
  • Is there a minimum size for vSphere Flash Read Cache allocation to a virtual machine?
    • The minimum size for vSphere Flash Read cache is dependent on the selected blocksize. With a 4KB blocksize the minimum is 4MB, and with a 1MB blocksize the minimum is 276MB
  • What happens if I have my block size configured to 1024KB and I do a 4KB IO?
    • Within that 1024KB cache block your 4KB IO will be stored, the rest of the block (1020KB in this scenario) will be marked as “empty”. Hence the reason it is important to align your IO block size with your cache block size to avoid wastage.
  • What happens to my cache when I resize it on a powered-on VM?
    • When the cache is resized all cache is discarded. This results in a period of time where the cache will need to warm up again.
  • If a host fails and virtual machines with vFRC enabled need to be failed over and there are no sufficient resources available what happens?
    • vFRC allocations are reservations. The behavior is similar to that of a CPU or memory reservation. When there is not enough unreserved capacity available then HA will not be able to power-on the virtual machine.
  • When I configure vSphere Flash Read Cache the maximum cache size seems to be limited to 200GB, what is supported and how do I change this?
    • The maximum cache size per virtual machine is 400GB, however in the current release the cache size is limited to 200GB. You can however change the advanced setting called “VFLASH.MacCacheFileSizeMB” to 409600 if desired. Do note that you will need to change this setting on every host if you want the virtual machine to be able to vMotion between hosts or want vSphere HA to be able to restart a virtual machine.
  • How do I monitor a VM to see how much cache is being used? Is there anything in vCenter or ESXTOP to help me?
    • vCenter contains multiple metrics on a virtual machine level. Just look at the advanced section and more explicitly the virtual disk counters starting with “Virtual Flash …”. There are also various helpful metrics to be found using “esxcli storage vflash cache stats get -c <cache name>. This will for instance provide you with the cache hit rate, latency etc. I have not found any esxtop counters yet.
  • When a virtual machine with vSphere Flash Read Cache enabled is vMotioned including the cache which network is used to transfer the cache?
    • When a VM is migrated / vMotioned including the cache then the “XvMotion / enhance vMotion / non-shared migration” logic is used to move the cache content from source to destination. The data is transferred over the vMotion network. Ensure that sufficient bandwidth is available to shorten the migration time.
  • When flash content is migrated along with the virtual machine, will the migration process make a distinction between hot and cold cache?
    • XvMotion is responsible for migrating the cache from source to destination. Today it doesn’t know the difference and will copy the full file from source to destination. Meaning that if you have a cache size of 10GB the full 10GB will be copied regardless of it being used or not.
  • Can I use the same disk that is divided into partition for vSphere Flash Read Cache & VSAN SSD tier?
    • This is not supported! Both VSAN and vSphere Flash Read Cache require there own flash device.
  • The SSD in my host is being reported in vSphere as “non-SSD”. According to support this is a known issue with the generation of server I am using. Will this “mis-reporting” of the disk type affect my ability to configure a vFlash? <added 17/Sept/2013>
    • Yes it will, you will need to tag the SSD as local using (example below is what I use in my lab, your identifier will be different). And in this case I claim it as being “local” and as “SSD”.
      esxcli storage nmp satp rule add –satp VMW_SATP_LOCAL –device mpx.vmhba2:C0:T0:L0 –option “enable_local enable_ssd”
  • If I disable Strict Admission Control for vSphere HA, would it be able to power-on a VM with vFRC enabled? <added 17/Sept/2013>
    • No, admission control does not have anything to do with powering on virtual machines. Even if you disable admission control and there are no sufficient flash resources available then a virtual machine with vFRC enabled cannot be powered on.
    • Also note, that HA Admission Control does not take vFRC resources in to account whatsoever. So in an environment running vFRC you can technically use 100% of all flash resources, whilst Admission Control was setup in an N+1 fashion.
  • Is vFRC as good as the caching solution by vendor X? <added 18/Sept/2013>
    • I have had this question many times. It is difficult to answer as it depends. It depends on:
      • Your requirements from a functional perspective (read vs write back caching)
      • Type of workload you are running (read vs write, VDI vs Server)
      • Support requirements (Not all solutions out there are on the VMware compatibility list)
      • Operational requirements (vFRC needs to be enabled per virtual disk, some vendors do per datastore for instance etc)
      • Budget and even current vSphere licenses used (vFRC is included with Enterprise Plus)
  • x

If you have any other questions, don’t hesitate to drop them here and I will add them to the FAQ when applicable to the broader audience.