I just noticed there has been an update to the VSAN HCL. When I now do a search for a disk controller (vmwa.re/vsanhcl) it immediately shows the queue depth of the controller. This will make life a lot easier, especially for those who prefer to build their own Virtual SAN node instead of using a Ready Node configuration. Although it is just a minor detail it is useful to know, and will definitely make life a lot easier when configuring your component built Virtual SAN nodes.
Search Results for: queue depth
Why Queue Depth matters!
A while ago I wrote an article about the queue depth of certain disk controllers and tried to harvest some of the values and posted those up. William Lam did a “one up” this week and posted a script that can gather the info which then should be posted in a Google Docs spreadsheet, brilliant if you ask me. (PLEASE run the script and lets fill up the spreadsheet!!) But some of you may still wonder why this matters… (For those who didn’t read some of the troubles one customer had with a low-end shallow queue depth disk controller, and Chuck’s take on it here.) Considering the different layers of queuing involved, it probably makes most sense to show the picture from virtual machine down to the device.
In this picture there are at least 6 different layers at which some form of queuing is done. Within the guest there is the vSCSI adaptor that has a queue. Then the next layer is VMkernel/VSAN which of course has its own queue and manages the IO that is pushed to the MPP aka muti-pathing layer the various devices on a host. On the next level a Disk Controller has a queue, potentially (depending on the controller used) each disk controller port has a queue. Last but not least of course each device (i.e. a disk) will have a queue. Note that this is even a simplified diagram.
If you look closely at the picture you see that IO of many virtual machines will all flow through the same disk controller and that this IO will go to or come from one or multiple devices. (Typically multiple devices.) Realistically, what are my potential choking points?
- Disk Controller queue
- Port queue
- Device queue
Lets assume you have 4 disks; these are SATA disks and each have a queue depth of 32. Total combined this means that in parallel you can handle 128 IOs. Now what if your disk controller can only handle 64? This will result in 64 IOs being held back by the VMkernel / VSAN. As you can see, it would beneficial in this scenario to ensure that your disk controller queue can hold the same number of IOs (or more) as your device queue can hold.
When it comes to disk controllers there is a huge difference in maximum queue depth value between vendors, and even between models of the same vendor. Lets look at some extreme examples:
HP Smart Array P420i - 1020
Intel C602 AHCI (Patsburg) - 31 (per port)
LSI 2008 - 25
LSI 2308 - 600
For VSAN it is recommended to ensure that the disk controller has a queue depth of at least 256. But go higher if possible. As you can see in the example there are various ranges, but for most LSI controllers the queue depth is 600 or higher. Now the disk controller is just one part of the equation, as there is also the device queue. As I listed in my other post, a RAID device for LSI for instance has a default queue depth of 128 while a SAS device has 254 and a SATA device has 32. The one which stands out the most is the queue depth of the SATA device, only a queue depth of 32 and you can imagine this can once again become a “choking point”. However, fortunately the shallow queue depth of SATA can easily be overcome by using NL-SAS drives (nearline serially attached SCSI) instead. NL-SAS drives are essentially SATA drives with a SAS connector and come with the following benefits:
- Dual ports allowing redundant paths
- Full SCSI command set
- Faster interface compared to SATA, up to 20%
- Larger (deeper) command queue [depth]
So what about the cost then? From a cost perspective the difference between NL-SAS and SATA is for most vendors negligible. For a 4TB drive the difference at the time of writing on different website was on average $ 30,-. I think it is safe to say that for ANY environment NL-SAS is the way to go and SATA should be avoided when possible.
In other words, when it comes to queue depth: spent a couple of extra bucks and go big… you don’t want to choke your own environment to death!
Disk Controller features and Queue Depth?
I have been working on various VSAN configurations and a question that always comes up is what are my disk controller features and queue depth for controller X? (Local disks, not FC based…) Note that this is not only useful to know when using VSAN, but also when you are planning on doing host local caching with solutions like PernixData FVP or SanDisk FlashSoft for instance. The controller used can impact the performance, and a really low queue depth will result in a lower performance, it is as simple as that.
** NOTE: This post is not about VSAN disk controllers, but rather about disk controllers and their queue depth. Always check the HCL before buying! **
I have found myself digging through documentation and doing searches on the internet until I stumbled across the following website. I figured I would share the link with you, as it will help you (especially consultants) when you need to go through this exercise multiple times:
Just as an example, the Dell H200 Integrated disk controller is on the VSAN HCL. According to the website above it is based on the LSI 2008 and provides the following feature set: 2×4 port internal SAS, no cache, no BBU, RAID 0, 1 and 10. According to the VSAN HCL also provides “Virtual SAN Pass-Through”. I guess the only info missing is queue depth of the controller. I have not been able to find a good source for this. So I figured I would make this thread a source for that info.
Before we dive in to that, I want to show something which is also important to realize. Some controllers take: SAS / NL-SAS and SATA. Although typically the price difference between SATA and NL-SAS is neglectable, the queue depth difference is not. Erik Bussink was kind enough to provide me with these details of one of the controllers he is using as an example, first in the list is “RAID” device – second is SATA and third SAS… As you can see SAS is the clear winner here, and that includes NL-SAS drives.
mpt2sas_raid_queue_depth: int
Max RAID Device Queue Depth (default=128)
mpt2sas_sata_queue_depth: int
Max SATA Device Queue Depth (default=32)
mpt2sas_sas_queue_depth: int
Max SAS Device Queue Depth (default=254)
If you want to contribute, please take the following steps and report the Vendor, Controller type and aqlength in a comment please.
- Run the
esxtop
command on the ESXi shell / SSH session - Press d
- Press f and select Queue Stats (d)
- The value listed under
AQLEN
is the queue depth of the storage adapter
The following table shows the Vendor, Controller and Queue Depth. Note that this is based on what we (my readers and I) have witnessed in our labs and results my vary depending on the firmware and driver used. Make sure to check the VSAN HCL for the supported driver / firmware version, note that not all controllers below are on the VSAN HCL, this is a “generic” list as I want it to serve multiple use cases.
Generally speaking it is recommended to use a disk controller with a queue depth > 256 when used for VSAN or “host local caching” solutions.
Vendor | Disk Controller | Queue Depth |
Adaptec | RAID 2405 | 504 |
Dell (R610) | SAS 6/iR | 127 |
Dell | PERC 6/i | 925 |
Dell | PERC H200 Integrated | 600 |
Dell | PERC H310 | 25 |
Dell | PERC H330 | 256 |
Dell (M710HD) | PERC H200 Embedded | 499 |
Dell (M910) | PERC H700 Modular | 975 |
Dell | PERC H700 Integrated | 975 |
Dell (M620) | PERC H710 Mini | 975 |
Dell (T620) | PERC H710 Adapter | 975 |
Dell (T620) | PERC H710p | 975 |
Dell | PERC H810 | 975 |
HP | Smart Array B110i | 1020 |
HP | Smart Array B120i | 31 |
HP | Smart Array P220i | 1020 |
HP | Smart Array P400i | 128 |
HP | Smart Array P410i | 1020 |
HP | Smart Array P420i | 1011 | HP | Smart Array P440ar | 1020 |
HP | Smart Array P700m | 1200 |
IBM | ServeRAID-M5015 | 965 |
IBM | ServeRAID-M5016 | 975 |
IBM | ServeRAID-M5110 | 975 |
Intel | C602 AHCI (Patsburg) | 31 (per port) |
Intel | C602 SCU (Patsburg) | 256 |
Intel | RMS25KB040 | 600 |
LSI | 2004 | 25 |
LSI | 2008 | 25 / 600 (firmware dependent!) |
LSI | 2108 | 600 |
LSI | 2208 | 600 |
LSI | 2308 | 600 |
LSI | 3008 | 600 |
LSI | 9271-8i | 975 |
LSI | 9300-8i | 600 |
Queue depth throttling
Most of you hopefully read about the new queue depth throttling feature in the release notes of ESX 3.5 U4 which has been released last week. A couple of customers asked me if this would be beneficial for them to set up.
Currently queue depth throttling is only supported for 3PAR Storage Arrays.
This, of course, doesn’t mean that it will not work with any of the other arrays. It actually does… but it probably hasn’t been tested to the full extent. Again, keep in mind that it’s currently not supported with any other array then 3PAR.
Now, what’s this queue depth throttling about? The knowledge base article actually has a good explanation of what it does:
VMware ESX 3.5 Update 4 introduces an adaptive queue depth algorithm that adjusts the LUN queue depth in the VMkernel I/O stack. This algorithm is activated when the storage array indicates I/O congestion by returning a BUSY or QUEUE FULL status. These status codes may indicate congestion at the LUN level or at the port (or ports) on the array. When congestion is detected,VMkernel throttles the LUN queue depth. VMkernel attempts to gradually restore the queue depth when congestion conditions subside.
In laymans terms: It’s an “algorithm” for handling queue sizes. When the array indicates it’s busy and/or that the queue is full it cuts the size of the queue in half so the array isn’t flooded with requests and can recover to a normal situation. When the array gives the green light the size of the queue will be increased again till the max specified queue depth has been reached.
Increasing the queue depth?
One of the most promising blogs of this moment is definitely Frank Denneman‘s blog. Frank is a freelance consultant with a focus on virtualization and storage. His latest addition “Increasing the queue depth” is an excellent article and really shows that Frank knows what he’s talking about!
When it comes to IO performance in the virtual infrastructure one of the most recommended “tweaks” is changing the Queue Depth (QD). But most forget that the QD parameter is just a small part of the IO path.
The IO path is made up of layers of hardware and software components, many of which can have a huge impact on the IO performance. The best results are achieved when the whole system is analysed and not just the ESX host alone.
I’m not going to copy and paste his entire article of course. Head over to his website and start reading. I can also recommend these articles: SRM and HP Continous Access DR Group Design and HP CA and the use of lun load balancing scripts. Don’t forget to bookmark the site or add it to your rss reader!