Yellow Bricks

by Duncan Epping

tps

How many pages can be shared if Large Pages are broken up?

· ·

I have written multiple articles(1, 2, 3, 4) on this topic so hopefully by now everyone knows that Large Pages are not shared by TPS. However when there is contention the large pages will be broken up in small pages and those will be shared based on the outcome of the TPS algorythm. Something I have always wondered and discussed with the developers a while back is if it would be possible to have an indication of how many pages can possibly be shared when Large Pages would be broken down. (Please note that we are talking about Small Pages backed by Large Pages here.) Unfortunately there was no option to reveal this back then.

While watching the VMworld esxtop session “Troubleshooting using ESXTOP for Advanced Users, TA6720” I noticed something really cool. Which is shown in the quote and the screenshot below which is taken from the session. Other new metrics that were revealed in this session and shown in this screenshot are around Memory Compression. I guess the screenshot speaks for itself.

  • COWH : Copy on Write Pages hints – amount of memory in MB that are potentially shareable,
  • Potentially shareable which are not shared. for instance when large pages are used, this is a good hint!!

There was more cool stuff in this session that I will be posting about this week, or at least adding to my esxtop page for completeness.

KB Article 1020524 (TPS and Nehalem)

· ·

Scott Lowe pointed to KB Article 1020524 in his short take article. Although I agree with Scott that it is a useful article it is actually technically incorrect. I wanted to point it out as when Scott points to something you know many will pick up on it.

On Nehalem systems with Hardware assisted Memory Management Unit (MMU), TPS improves performance from 10 – 20% by using large pages.

Since TPS is done per small page, the VMkernel delays page sharing until host memory becomes over-committed. In the background, the ESX host gathers information about TPS, then uses it when it needs more memory.

It might be just a detail, but it is important to realize that it is not TPS that improves performance but Large Pages. TPS has got absolutely nothing to do with it and does not impact performance anywhere near the mentioned 10-20%.

One thing to note is that TPS does identify the pages which can be shared, however as 2MB pages are used they are not actively shared. When a system gets overcommitted those Large Pages (2MB) will be broken down in Small Pages (4KB) and the already identified pages will shared.

I just love TPS….

Disabling TPS hurting performance?

· ·

On the internal mailinglist there was a discussion today around how disabling TPS (Transparent Page Sharing) could negativitely impact performance. It is something I hadn’t thought about yet but when you do think about it it actually does make sense and is definitely something to keep in mind.

Most new servers have some sort of NUMA architecture today. As hopefully all of you know TPS does not cross a NUMA node boundary. This basically means that pages will not be shared between NUMA nodes. Another thing that Frank Denneman already described in his article here is that when memory pages are allocated remotely there is a memory penalty associated with it. (Did you know there is an “esxtop” metric, N%L,which shows the percentage of remote pages?) These pages are accessed across an interconnect bus which is always slower than so called local memory.

Now you might ask what is the link between NUMA, TPS and degraded performance? Think about it for a second… TPS decreases the amount of physical pages needed. If TPS is disabled there is no sharing and chances of going across NUMA nodes are increased and as stated this will definitely impact performance. Funny how disabling a mechanism(TPS) which is often associated with “CPU overhead” can have a negative impact on memory latency.