Yellow Bricks

by Duncan Epping

security

Disk format change going from 6.x to vSAN 6.6?

· ·

Internally I received a comment around the upgrade to 6.6 and the disk format version change. When you upgrade to 6.6 also the version of  disk changes, it goes to version 5. In the past with these on-disk format changes a data move was needed and the whole disk group would be reformat. When going from vSAN 6.2 (vSphere 6.0 U2 that is!) to vSAN 6.6 there is no data move needed. The update is simply a metadata update, and on the average cluster will take less than 20 minutes.

When introducing encryption in to the environment you will need to evac data though as this will be a reformat of the disk. Reason for it being is that the disk will need to be encrypted and so will the data. This doesn’t mean however that if you want to “rekey” your environment a full format and data move is needed, vSAN Encryption has the ability to do a so called “shallow rekey” which means that the Key Encryption Key (KEK) will be replaced (see animated gif below), but not the Data Encryption Key (DEK). It is possible to do a deep rekey, but this will mean a full reformat and data evac of all disk groups. I hope that clears things up.

The difference between VM Encryption in vSphere 6.5 and vSAN encryption

· ·

More and more people are starting to ask me what the difference is between VMCrypt aka VM Encryption and the beta feature we announced not to long ago called vSAN Encryption. (Note, we announced a beta, no promises were made around dates or actual releases or releasing of the feature.) Both sounds very much the same and essential both end up encrypting the VM but there is a big difference in terms of how it is implemented. There are advantages and disadvantages to both solutions. Lets look at VM Encryption first.

VM Encryption is implemented through VAIO (vSphere APIs for IO Filters). The VAIO framework allows a filter driver to do “things” to/with the IO that a VM sends down to a device. One of these things is encryption. Now before I continue, take a look at this picture of where the filter driver sits.

As you can see the filter driver is implemented in the User World and the action against the IO is taken at the top level. If this for instance is encryption then any data send across the wire is already encrypted. Great in terms of security of course. And all of this can be enabled through policy. Simply create the policy, select the VM or VMDK you want to encrypt and there you go. So if it is that awesome, why vSAN Encryption?

Well the problem is that all IO is encrypted at the top level. This means that it is received in the vSAN write buffer fully encrypted, then the data at some point needs to be destaged and is deduplicated and compressed (in all-flash). As you can imagine, encrypted blocks do not dedupe (or compress) well. As such in an all-flash environment with deduplication and compression enabled any VM that has VM Encryption through VAIO enabled will not provide any space savings.

With vSAN Encryption this will be different. The way it will work is that it will provide “encryption at rest”. The data travels to the destination unencrypted then when it reaches its destination it is written encrypted to the cache tier, then it is decrypted before it is destaged, and it will be encrypted after it is deduplicated and/or compressed again. This means that you will benefit from space saving functionality, however encryption in this case is a cluster wide option, which means that every VM will be encrypted, which may not be desirable.

So in short:

  • VM Encryption (VAIO)
    • Policy based (enable per VM)
    • Data travels encrypted
    • No/near zero dedupe
  • vSAN Encryption
    • Enabled on a cluster level
    • Data travels unencrypted, but it is written encrypted to the cache layer
    • Full compatibility with vSAN data services

I hope that clarifies why we announced the beta of vSAN Encryption and what the difference is with VM Encryption that is part of vSphere 6.5.

(Inter-VM) TPS Disabled by default, what should you do?

· ·

We’ve all probably seen the announcement around inter-VM(!!) TPS (transparent page sharing) being disabled by default in future releases of vSphere, and the recommendation to disable it in current versions. The reason for this is the fact there was a research paper published which demonstrates how it is possible to get access to data under certain highly controlled conditions. As the KB article describes:

Published academic papers have demonstrated that by forcing a flush and reload of cache memory, it is possible to measure memory timings to determine an AES encryption key in use on another virtual machine running on the same physical processor of the host server if Transparent Page Sharing is enabled. This technique works only in a highly controlled environment using a non-standard configuration.

There were many people who blogged about what the potential impact is on your environment or designs. Typically in the past people would take a 20 to 30% memory sharing in to account when sizing their environment. With inter-VM TPS disabled of course this goes out of the window. Frank described this nicely in this post. However, as Frank also described and I mentioned in previous articles when large pages are being used (usually the case) then TPS is not used by default and only under pressure…

The under pressure part is important if you ask me as TPS is the first memory reclaiming technique used when a host is under pressure. If TPS cannot sufficiently reduce the memory pressure then ballooning is leveraged, followed by compression and swapping ultimately. Personally I would like to avoid swapping at all costs and preferably compression as well. Ballooning typically doesn’t result in a huge performance degradation so it could be acceptable, but TPS is something I prefer as it just breaks up large pages in to small pages and collapses those when possible. Performance loss is hardly measurable in that case. Of course TPS would be way more effective when pages between VMs can be shared rather then just within the VM.

Anyway, the question remains should you have (inter-VM) TPS disabled or not? When you assess the risk you need to ask yourself first who has access to your virtual machines as the technique requires you to login to a virtual machine. Before we look at the scenarios, not that I mentioned “inter-VM” a couple of times now, TPS is not completely disabled in future versions. It will be disabled for inter-VM sharing by default, but can be enabled. More to be found on that in this article on the vSphere blog.

Lets explore 3 scenarios:

  1. Server virtualisation (private)
  2. Public cloud
  3. Virtual Desktops

In the case of “Server virtualisation”, in most scenarios I would expect that only the system administrators and/or application owners have access to the virtual machines. The question then is, why would they go to this level when they have access to the virtual machines anyway? So in the scenario where Server Virtualization is your use case, and access to your virtual machines is restricted to a limited number of people, I would definitely reconsider enabling inter-VM TPS.

In a public cloud environment this however is different of course. You can imagine that a hacker could buy a virtual machine and try to retrieve the AES encryption key. What he (the hacker) does with it next of course is even then still the question. Hopefully the cloud provider ensures that that the tenants are isolated from each other from a security/networking point of view. If that is the case there shouldn’t be much they could do with it. Then again, it could be just one of the many steps they have to take to break in to a system so I would probably not want to take the risk, although the risk is low. This is one of the scenarios where I would leave inter-VM TPS disabled.

Third and last scenario is Virtual Desktops. In the case of a virtual desktop many different users have access to virtual machines… The question though is if you are running any applications or accessing applications which are leveraging AES encryption or not. I cannot answer that for you, so I will leave that up in the air… you will need to assess that risk.

I guess the answer to whether you should or should not disable (inter-VM) TPS is as always: it depends. I understand why inter-VM TPS was disabled, but if the risk is low I would definitely consider enabling it.

VMware patches for #shellshock

· ·

Last night a whole bunch of patches for the shellshock security issue were released. Although I am hoping that all of you have your datacenter secured for outside threads and inside threads by isolating networks, firewalls etc… It would be wise to install these patches ASAP. Majority of linux based VMware appliances were impacted, but luckily patching them is not a huge thing. Below you can find a list of the patches and links to the downloads for your convenience.

Note that the downloads are in the middle of the list, so you need to scroll down before you see them. There are also patches for products like the VMware VSA, vSphere Replication, VC Ops etc. Make sure to download those as well!

New beta of the vSphere 5.5 U1 Hardening Guide released

· ·

Mike Foley just announced the new release of the vSphere 5.5 U1 Hardening Guide. Note that it is still labeled as a “beta” as Mike is still gathering feedback, however the document should be finalized first week of June.

For those concerned about security, this is an absolute must read! As always, before implementing ANY of these recommendations make sure to test them on a test cluster and test expected functionality of both the vSphere platform and the virtual machines and applications running on top of it.

Nice work Mike!