Yellow-Bricks

by Duncan Epping

Copyright © 2016

How does Mem.MinFreePct work with vSphere 5.0 and up?

With vSphere 5.0 VMware changed the way Mem.MinFreePct worked. I had briefly explained Mem.MinFreePct in a blog post a long time ago. Basically Mem.MinFreePct, pre vSphere 5.0, was the percentage of memory set aside by the VMkernel to ensure there are always sufficient system resources available. I received a question on twitter yesterday based on the explanation in the vSphere 5.1 Clustering Deepdive and after exchanging > 10 tweets I figured it made sense to just write an article.

Mem.MinFreePct used to be 6% with vSphere 4.1 and lower. Now you can imagine that when you had a host with 10GB you wouldn’t worry about 600MB being kept free, but that is slightly different for a host with 100GB as it would result in 6GB being kept free but still not an extreme amount right. What would happen when you have a host with 512GB of memory… Yes, that would result in 30GB of memory being kept free. I am guessing you can see the point now. So what changed with vSphere 5.0?

In vSphere 5.0 a “sliding scale” principle was introduced instead of Mem.MinFreePct. Let me call it “Mem.MinFree”, as I wouldn’t view this as a percentage but rather do the math and view it as a number instead. Lets borrow Frank’s table for this sliding scale concept:

Percentage kept free of –>
 Memory Range
6% 0-4GB
4% 4-12GB
2% 12-28GB
1% Remaining memory

What does this mean if you have 100GB of memory in your host? It means that from the first 4GB of memory we will set aside 6% which equates to ~ 245MB. For the next 8GB (4-12GB range) we set aside another 4% which equates to ~327MB. For the next 16GB (12-28GB range) we set aside 2% which equates to ~ 327MB. Now from the remaining 72GB (100GB host – 28GB) we set aside 1% which equates to ~ 720MB. In total the value of Mem.MinFree is ~ 1619MB. This number, 1619MB, is being kept free for the system.

Now, what happens when the host has less than 1619MB of free memory? That is when the various memory reclamation techniques come in to play. We all know the famous “high, soft, hard, and low” memory states, these used to be explained as: 6% (High), 4% (Soft), 2% (Hard), 1% (Low). FORGET THAT! Yes, I mean that… forget these as that is what we used in the “old world” (pre 5.0). With vSphere 5.0 and up these water marks should be viewed as a Percentage of Mem.MinFree. I used the example from above to clarify it a bit what it results in.

Free memory state Threshold in Percentage
 Threshold in MB
High water mark Higher than or equal to Mem.MinFree 1619MB
Soft water mark 64% of Mem.MinFree 1036MB
Hard water mark 32% of Mem.MinFree 518MB
Low water mark 16% of Mem.MinFree 259MB

I hope this clarifies a bit how vSphere 5.0 (and up) ensures there is sufficient memory available for the VMkernel to handle system tasks…

Comments

  2. Hi, in all the examples I’ve found, the 6 % value is used for the Mem.MemMinFreePct-setting. But what if you change this value? With the 6 %, the other percentages of the ranges are 4 %, 2 % and 1 %… But what if you change the 6 % to 10 % (for example!) ? Will the other percentages change too? I assume that the 4, 2 and 1 % are an outcome of 64 % of 6 % (which is 4 %), 32 % of 6 % (which is 2 %) and 16 % of 6 % (which is 1 %)…

    So changing the Mem.MemMinFreePct-setting to 10 % would, following the above assumption, result in:
    Range %
    4-12 GB => 6 % (64 % of 10 %)
    12-28 GB => 3 % (32 % of 10 %)
    remaining => 1.6 ~ 2 % (16 % of 10 %)

    Is this assumption correct? Or how does it work?

    • you are correct in a way, mixing two things here. those ranges are only used with 5.0 which doesn’t use the “fixed” 6% any longer, it uses 6/4/2/1 all the way.

      With 4.1 and prior, changing that 6% would indeed lead to (4/2/1 aka 64/32/16) changing as well.

      • Thanks for your quick response. Ok, so changing the 6 % in 5.0 and after, changes the rest of the sliding scale too.

        Does the counter ‘Mem.LowFreeThreshold’ has a relation to the MemMinFreePct? When I use the 4.1-method for calculating the high threshold, the result equals the value mentioned in the LowFreeTreshold counter in stead of the total high treshold with the >5.0-sliding scale calculation …

  3. Would you clarify when each state is invoked? I have four memory states, which means I should only need three thresholds.

    High : greater than minfreepct
    Soft : between 64% minfreepct and minfreepct
    Hard : between 32% minfreepct and 64% minfreepct
    Low : less than 32% minfreepct

    Does anything different happen for memory reclamation around 16% minfree pct versus below 32%? What’s different in terms of memory reclamation techniques at 20% minfreepct versus 10% minfreepct? The table in the article doesn’t clarify the the “between this and that” for memory state.

    Thanks,

    Joel

      • Thanks Duncan (btw I’m a fan of “HA and Clustering Deepdive”). I left a request at Frank’s blog too. The math still doesn’t work (as far as I can tell):

        High is greater than MinFreePct
        Soft is between 64% MinFreePct and MinFreePct
        Hard is between 32% MinFreePct and 64% MinFreePct
        Low is below 32% MinFreePct
        What memory state is below 16% MinFreePct? I think I am out of choices.

        • Not sure I am following it… but this is what happens and when:
          Between High and Soft (64%): Ballooning
          When “Hard” water mark(32%) is reached: Ballooning, Compression, Swapping
          When Low water mark (16%) is reached: Ballooning, Compression, Swapping + “blocking of execution of VMs that consume more than their target size.”

          • Thanks for your patience. I think we are almost there. I think my host is in a HIGH memory state when above MinFreePct. When I drop below MinFreePct, then I go to SOFT. When I drop below 64% MinFreePct, I should be in HARD. When I drop below 32% MinFreePct, I should be in LOW. You have another threshold at 16%, but I think I am out of memory states. You said ‘when HARD water mark (32%) is reached’. Didn’t I reach HARD when I fell below 64%? So when I get to the next threshold (32%), aren’t I due to be in LOW?

            I don’t see how I can start in HIGH, pass three thresholds and not be in the fourth and final state.

            Again, I appreciate you sticking with me. One of us is having a counting problem (maybe I need more fingers).

          • Maybe my mistake is in understanding the transition from HIGH to SOFT. Am I in HIGH through the MinFreePct all the way to 64% of MinFreePct? I’ve been assuming the transition takes place at MinFreePct.

  4. Hi Duncan,

    do the same principles (Mem.MinFreePct & Thresholds) apply to resource pools ?