A while ago (2014) I wrote an article on TPS being disabled by default in future release. (Read KB 2080735 and 2097593 for more info) I described why VMware made this change from a security perspective and what the impact could be. Even today, two years later, I am still getting questions about this and what for instance the impact is on swap files. With vSAN you have the ability to thin provision swap files, and with TPS being disabled is this something that brings a risk?
Lets break it down, first of all what is the risk of having TPS enabled and where does TPS come in to play?
With large pages enabled by default most customers aren’t actually using TPS to the level they think they are. Unless you are using old CPUs which don’t have EPT or RVI capabilities, which I doubt at this point, it only kicks in with memory pressure (usually) and then large pages get broken in to small pages and only then will they be TPS’ed, if you have severe memory pressure that usually means you will go straight to ballooning or swapping.
Having said that, lets assume a hacker has managed to find his way in to you virtual machine’s guest operating system. Only when memory pages are collapsed, which as described above only happens under memory pressure, will the hacker be able to attack the system. Note that the VM/Data he wants to attack will need to be on the located on the same host and the memory pages/data he needs to breach the system will need to be collapsed. (actually, same NUMA node even) Many would argue that if a hacker gets that far and gets all the way in to your VM and capable of exploiting this gap you have far bigger problems. On top of that, what is the likelihood of pulling this off? Personally, and I know the VMware security team probably doesn’t agree, I think it is unlikely. I understand why VMware changed the default, but there are a lot of “IFs” in play here.
Anyway, lets assume you assessed the risk and feel you need to protect yourself against it and keep the default setting (intra-VM TPS only), what is the impact on your swap file capacity allocation? As stated when there is memory pressure, and ballooning cannot free up sufficient memory and intra-VM TPS is not providing the needed memory space either the next step after compressing memory pages is swapping! And in order for ESXi to swap memory to disk you will need disk capacity. If and when the swap file is thin provisioned (vSAN Sparse Swap) then before swapping out those blocks on vSAN will need to be allocated. (This also applies to NFS where files are thin provisioned by default by the way.)
What does that mean in terms of design? Well in your design you will need to ensure you allocate capacity on vSAN (or any other storage platform) for your swap files. This doesn’t need to be 100% capacity, but should be more than the level of expected overcommitment. If you expect that during maintenance for instance (or an HA event) you will have memory overcommitment of about 25% than you could ensure you have 25% of the capacity needed for swap files available at least to avoid having a VM being stunned as new blocks for the swap file cannot be allocated and you run out of vSAN datastore space.
Let it be clear, I don’t know many customers running their storage systems in terms of capacity up to 95% or more, but if you are and you have thin swap files and you are overcommitting and TPS is disabled, you may want to re-think your strategy.
Default setting is intra-VM TPS only, no inter-VM TPS, right? VMware terminology is: intra-VM is within one VM, inter-VM is between different VMs. By the way, inter-VM TPS is done per NUMA node, so the attack risk is limited to VMs within the same NUMA node, not just the same host.
Good point, just realized it should be intra indeed, edited the article. And yes TPS considers NUMA indeed, so even less likely. (As Frank documented here for those interested: http://frankdenneman.nl/2016/08/22/numa-deep-dive-part-5-esxi-vmkernel-numa-constructs/)
Interesting question…. are EPT and RVI worth it? Is there a significant performance advantage for most applications of having large pages, or do we sacrifice TPS needlessly?
Higher levels of memory overcommit would be useful, that I see, most environments wind up being heavy on memory/RAM consumption as the limiting factor for growth and consolidation — CPU usage under most conditions is low on well-stocked hosts.
I have looked at the security/hacking risk with TPS, and it seems to be extremely low, almost laughable. Perhaps “disable/group TPS by hash for especially-sensitive high-security VMs” would have been a more sensible default.
Perhaps VMware could have added a slide bar to the Virtual Machine configuration page to make a number of performance sacrifices for enhanced security for some VMs. E.g. “Certain VMs require No TPS, Encrypted VMotion, Encrypted storage traffic, Encrypted VMDKs, ” but the vast majority of VMs do not, so only select certain VMs for the sacrifice……
Sure, VMware is correct to take the overall concept seriously, as side-channel attacks could be of concern particularly for banking customers with small high-value secrets and multi-tenant/mixed-security-level environments; However, the attack difficulty of side-channel key discovery is astronomically high,
as the TPS hacking method essentially requires another VM has encryption calculations or keys in CPU cache, and the attacking VM makes guesses to collect statistical timings over a long period of time.
I see the proper fix is to use hardware AES offloads or different cryptography resistant to these issues, not to disable performance features such as TPS, however. After all…… even with TPS disabled, a VM will still be susceptible to this attack by other programs running in the same VM.
I have disabled large pages and salting as long as those options have existed.
It makes a huge difference as we can use a lot less hardware. Nowadays we use UCS B200 M4 blades with 2×18 core and 768GB RAM in our clusters, and on average TPS save around 300GB per host. As we have VMs in the five figures this saves us many millions in hardware expenses, and its also better for the environment…
As I see it there is as good as zero risk that we will get exploited. The “row hammer” exploit needs “vulnerable” hardware too as I understand it, and UCS is not vulnerable according to Cisco.