Well depending on what type of queues we are talking about of course, but in general no one likes queues. We are however confronted with queues on a daily basis, especially in compute environments. I was having a discussing with an engineer around storage queues and he sent me the following which I thought was worth sharing as it gives a good overview of how traffic flows from queue to queue with the default limits on the VMware side:
From top to bottom:
- Guest device driver queue depth (LSI=32, PVSCSI=64)
- vHBA (Hard coded limit: LSI=128, PVSCSI=255)
- disk.schedNumOutstanding=32 (VMKernel),
- VMkernel Device Driver (FC=32, iSCSI=128, NFS=256, local disk=32)
- Multiple SAN/Array Queues (Check Chad’s article for more details but it includes port buffers, port queues, disk queues etc (might be different for other storage vendors))
The following is probably worth repeating or clarifying:
The PVSCSI default queue depth is 64. You can increase it to 255 if required, please note that it is a per-device queue depth and keep in mind that this would only be truly useful when it is increased all the way down the stack and the Array Controller supports it. There is no point in increasing the queuedepth on a single layer when the other layers are not able to handle it as it would only push down the delay one layer. As explained in an article a year or three ago, disk.schednumreqoutstanding is enforced when multiple VMs issue I/Os on the same physical LUN, when it is a single VM it doesn’t apply and it will be the Device Driver queue that limits it.
I hope this provides a bit more insight to how the traffic flows. And by the way, if you are worried a single VM floods one of those queues there is an answer for that, it is called Storage IO Control!
Great post of hard to find info!
Great post!
I have particular VM(s) that occasionally flood the system with IO requests. I want to ensure they do not affect other systems by limiting the amount of IOps that can occur on that VM.
I tried enabling Storage IO Control and then setting a IOps limit on the particular LUN, however it seems like this limit may only be realized if the response time is exceeded for the LUN. Is there any way of using SIOC to Always limit the IOps(regardless if the response time is exceeded)? I have a bunch of other servers which use the same storage but not in VMWare, this is why I don’t believe “shares” is a good enough solution for me.
Thanks for your opinion on this matter.
I added a similar post here: http://communities.vmware.com/message/1711790
The limit on a VMDK is not related to SIOC at all. The limit is handled by the local disk scheduler and should always apply even if you don’t have SIOC enabled. have you looked at esxtop to verify it doesn’t work?
-D
I tried this simple test:
1. Ran the “dd” command to generate a 8GB file, this process finished with a Rate of approx 120MB/s
2. set the “Resources/Disk/Limit – IOPs” to 16 (the lowest number possible)
3. Ran the “dd” command again, it finishes with the same results as in step # 1.
From this test it appears that this limit is not applied/honoured by default.
What about the queus of the kernel on top of the PVSCI? Like /queue/nr_requests (128) vs. device/queue_depth (64)?