I had a question this week and I thought I wrote about this before but apparently, I did not. Hopefully by now most of you the I/O scheduler has changed over the past couple of years. Within ESXi, we moved to a new scheduler, which often is referred to as mClock. This new scheduler, and a new version of SIOC (storage i/o control) also resulted in some behavioral changes. Some which may be obvious, others which are not. I want to explicitly point out two things which I have discussed in the past which changed with vSphere 6.5 (and 6.7 as such) and both are around the use of limits.
First of all: Limits and SvMotion. In the past when a limit was applied to a VM, this would also artificially limit SvMotion. As of vSphere 6.5 (may apply to 6.0 as well but I have not verified this) this is no longer the case. Starting vSphere 6.0 the I/O scheduler creates a queue for every file on the file system (VMFS), this to avoid for instance a VM stalling other types (metadata) of IO. The queues are called SchedQ and briefly described by Cormac Hogan here. Of course, there’s a lot more to it than Cormac or I discuss here, but I am not sure how much I can share so I am not going to go there. Either way, if you were used to limits being applied to SvMotion as well you are warned… this is no longer the case.
Secondly, the normalization of I/Os changed with limits. In the past when a limit was applied IOs were normalized at 32KB, meaning that a 64KB I/O would count as 2 I/Os and a 4KB I/O would count as 1. This was confusing for a lot of people and as of vSphere 6.5 this is no longer the case. When you place a limit of 100 IOPS the VMDKs will be limited at 100 IOPS, regardless of the I/O size. This, by the way, was documented here on storagehub, not sure though how many people realized this.
goslackware says
Duncan,
In you link:
https://storagehub.vmware.com/t/vsphere-storage/vsphere-6-5-storage-1/storage-i-o-control-v2-2/
It says “In a custom setting, IOPS limit and IOPS reservation are both set to -1, implying unlimited.”
However, the picture, shows IOPS reservation set to “1”, not “-1″…
Erik says
Thanks for the info, I logged a support case nearly 2 months ago regarding mclock not normalizing based on 32KB, still unresolved. Maybe it was due to that case you got the question 🙂
I actually find it way more confusing not being able to normalize based on IO size. It feels like VMware is removing technical functionality and putting up an “Apple wall” to appear more user friendly. I thought users of VMware technology were techies and not one-click-guys.
David Manconi says
Hi Erik, What was the SR number you logged as I would like to investigate for another customer.
Cheers
David
Erik says
Hi David,
Sure, case number is 18801459205. It is still open, but I haven’t heard anything since July 13th. The conclusion is clear from Ducan post though, the feature has been removed.
David Manconi says
Thanks Erik. Yes saw the comments and conclusion in the article.
duncan@yellow-bricks says
I think the best thing to do is to file a feature request and explain the use case with the SIOC/SDRS team, i will drop them an email about this.