Yesterday I showed how to recover from a vShield App crash. Now bare in mind that this scenario is very rare. Today I decided to lock down my environment to the level where it was impossible to login to vCenter Server or vShield Manager. I added L2 and L3 “Any – Any” block rules to the Datacenter which hosts vCenter and vShield Manager. I needed to get access back to my vCenter host so I started digging and this is how I managed to get it back… it was a lot easier than expected:

1. http://<ip-address-of-vShield-manager>
2. remove/change rule

Was it really that simple? Yes it was, even after applying block rules I could still access vShield Manager. I wondered why so I started digging in to it.

If you look at the vShield Manager UI you will see that all VMs are listed except for vShield Manager and the vShield App FW VMs. The reason for this is that the vShield VMs are considered to be Service VMs. You can actually see this when you go to your Cluster in the vShield Manager UI and check the “Summary” as it will list the amount of Service VMs as shown in the screenshot below.

I wondered what caused these VMs to be listed as Service VMs so I looked at the .vmx file of the vShield VMs and spotted the following entries:

• vShield Manager.vmx:
  vshield.vmtype = "Manager"
vshield.vmversion = "5.0"
vshield.vmbuild = "473791"
• vShield App FW.vmx:
  vshield.vmtype = "Zones"
vshield.vmversion = "3.0"
vshield.vmbuild = "473791"

Another thing that I noticed in the .vmx file for vShield Manager is that it did not have a filter applied, in other words traffic goes straight to the VM. This was the reason traffic was not blocked by the rules we created. The next thing I wanted to test is what would happen if I would remove the filters from the vCenter VM and simply add the three .vmx entries that the vShield Manager had? The reason I wanted to test this is because I wanted to know if a filter would be applied or not.

Instead of (ab)using my vCenter VM for this (I might need it later on) I created a test VM. I booted up the VM to see if it would get the filter and made sure the rules I created were applied. I couldn’t access the VM as expected as the filter and the rules were applied. I powered it off, removed the filter, added the three entries (vShield Manager) and booted up the VM… No changes were made to the VM and I could still access it. Is this useful for your production environment? No it is not, as it is definitely not recommended to make changes like these as it is totally unsupported and could lead to unexpected results. It is nice to know though…

I wanted to know what happened when my vShield App virtual machine would fail. So I killed it and of course I couldn’t reach vCenter anymore. The reason for this being is the fact that a so-called dvfilter is used. The dvfilter basically captures the traffic, sends it to the vShield App VM which inspects it and then sends it to the VM (or not depending on the rules). As I killed my vShield App VM there was no way it would work. If I would have had my vCenter available I would just vMotion the VMs to another host and the problem would be solved, but it was my vCenter which was impacted by this issue. Before I started digging myself I did a quick google and I noticed this post by vTexan. He had locked himself out by creating strict rules, but my scenario was different. What were my options?

Well there are multiple options of course:

1. Move the VM to an unprotected host
2. Disarm the VM
3. Uninstall vShield

As I did not have an unprotected host in my cluster and did not want to uninstall vShield I had only 1 option left. I figured it couldn’t be too difficult and it actually wasn’t:

1. Connect your vSphere Client to the ESXi host which is running vCenter
2. Power Off the vCenter VM
3. Right click the vCenter VM and go to “Edit Settings”
4. Go to the Options tab and click General under Advanced
5. Click Configuration Parameters
6. Look for the “ethernet0.filter0″ entries and remove both values
7. Click Ok, Ok and power on your vCenter VM

As soon as your vCenter VM is booted you should have access to vCenter again. Isn’t that cool? What would happen if your vShield App would return? Would this vCenter VM be left unprotected? No it wouldn’t, vShield App would actually notice it is not protected and add the correct filter details again so that the vCenter VM will be protected. If you want to speed this process up you could of course also vMotion the VM to a host which is protected. Now keep in mind that while you do the vMotion it will insert the filter again which could cause the vCenter VM to disconnect. In all my tests so far it would reconnect at some point, but that is no guarantee of course.

Tomorrow I am going to apply a security policy which will lock out my vCenter Server and try to recover from that… I’ll keep you posted.

** Disclaimer: This is for educational purposes, please don’t try this at home… **

The VMware Cloud Infrastructure aims to reduce operational overhead and lower Total Cost of Ownership (TCO) by simplifying management tasks and abstracting complex processes. The focus of this architecture, as indicated by our customer requirements, is resource aggregation and isolation through the use of pools for each of the crucial pillars: network, storage and compute. Each of the three pillars will be carved in to multiple units of consumption with priority allocated based on their service level agreement. This will be achieved by leveraging core functionality offered by vSphere 5.0. Subsequently vShield App will be used to isolate each of the different type of workloads. As a hypervisor-based application-aware firewall solution, vShield App allows defining policies to logical, dynamic application boundaries (security groups) instead of physical boundaries.

This resource and security layering method will allow for a fast and safe deployment of new workloads.

Each of the different types of resources are carved up in to different groups for each of the respective workload types. A virtual machine, or vApp, will be deployed in one of the three different compute and security groups after which a specific networking group will be selected and a storage tier. Compute, Security and Network  group types are currently defined based on the different type of workloads this virtual infrastructure will host. In the future additional blocks may be added based on the requirements of the internal customers and the different types of workloads being deployed…

I guess there is no point in rehashing what is written in the whitepaper of what Richard wrote, I just want to point out the whitepaper as I believe it is a good read. As always results may vary but it is pretty obvious that from an architectural and operational perspective End Point Security is most definitely worth looking into and I cannot wait for more vendors to jump on the bandwagon. Download the tolly report here. (I personally found the disk results very interesting…)

Download:
http://www.vmware.com/files/pdf/techpaper/VMW_10Q3_WP_vCloud_Director_Security.pdf

Description

The VMware® vCloud™ Director Security Hardening Guide helps users who are embarking into the journey of cloud computing understand key security elements and technologies found in VMware’s vCloud Director product. It also provides guidelines and best practices for installation, configuration and operation of secure clouds based on VMware’s vCloud Director.

The KB article that was published yesterday contains a workaround to change this behaviour. I recommend everyone to read the article and implement this workaround when your password policy describes passwords longer than 8 characters.

Hytrust is a policy driven appliance which enhances security and auditing for virtualized environments. Although I had seen multiple demos I had never actually played around with it. I must say I was pleasantly surprised at Tech Summit.

Hytrust sits in between you, the user/admin, and the vCenter/ESX. Basically it proxies the requests based on your role. If the role has no permissions on the specific “task” it will return a message stating “permission denied by Hytrust”.

Now that sounds cool doesn’t it? I guess what was even more impressing was the fact that with Hytrust this also works on ESXi. Yes you are reading that correct, role based “unsupported” mode access to ESXi, that’s something VMware doesn’t even offer at the moment. I tested it, it works great! (Yeah I know it is still not supported, but it does offer a solution to those who need it.)

Another cool thing is the configuration templates for Hosts. It basically enables assessment of security configuration. Hytrust contains several pre-built templates including for instance VMware’s Security Hardening Best Practices. Not only assessment but also the option to remediate when needed.

And I haven’t even talked about the auditing functionality yet. As Hytrust proxies all commands, it is just a small step for them to log all the info and make it audit-able….

After playing around with in Hytrust I fully understand why Cisco invested, it rocks. Just try it out. The Community Edition, free for up to three hosts is available here: Hytrust Appliance v2.0 Community Edition

source article

This document is the official release of the vSphere 4.0 Security Hardening Guide. This version is based on feedback collected during the public draft comment period. We will still be collecting feedback on this document — if there are any typos, errors, or changes, please add them to the comments below.

Overall, there are more than 100 guidelines, with the following major sections:

• Introduction
• Virtual Machines
• Host (both ESXi and ESX)
• vNetwork
• vCenter
• Console OS (for ESX only)

vSphere Hardening Guide April 2010.pdf (951.0 K) View Download

For security reasons some customers have the requirement to insert specific account information for every user. It appears that when you modify the details for “vpxuser” in /etc/passwd and the vpxuser password is refreshed these account details are overwritten. (Every 30 days the vpxuser password gets refreshed.) According to my former colleague this has been fixed in vCenter 4.0 but the “issue” does exist in vCenter 2.5 Update 6 today.

vShield Manager login(page 24): admin/default
Configure IP Address with following command (page 35): setup