Yellow Bricks

by Duncan Epping

vshield

vShield App broke down on the host that is running vCenter now what?

· ·

I was playing around with vShield App and I locked out my vCenter VM which happened to be hosted on the cluster which was protected by vShield App. Yes I know that it is not recommended, but I have a limited amount of compute resources in my lab and I can’t spare a full server just for vCenter so I figured I would try it anyway and by breaking stuff I learn a lot more.

I wanted to know what happened when my vShield App virtual machine would fail. So I killed it and of course I couldn’t reach vCenter anymore. The reason for this being is the fact that a so-called dvfilter is used. The dvfilter basically captures the traffic, sends it to the vShield App VM which inspects it and then sends it to the VM (or not depending on the rules). As I killed my vShield App VM there was no way it would work. If I would have had my vCenter available I would just vMotion the VMs to another host and the problem would be solved, but it was my vCenter which was impacted by this issue. Before I started digging myself I did a quick google and I noticed this post by vTexan. He had locked himself out by creating strict rules, but my scenario was different. What were my options?

Well there are multiple options of course:

  1. Move the VM to an unprotected host
  2. Disarm the VM
  3. Uninstall vShield

As I did not have an unprotected host in my cluster and did not want to uninstall vShield I had only 1 option left. I figured it couldn’t be too difficult and it actually wasn’t:

  1. Connect your vSphere Client to the ESXi host which is running vCenter
  2. Power Off the vCenter VM
  3. Right click the vCenter VM and go to “Edit Settings”
  4. Go to the Options tab and click General under Advanced
  5. Click Configuration Parameters
  6. Look for the “ethernet0.filter0” entries and remove both values
  7. Click Ok, Ok and power on your vCenter VM

As soon as your vCenter VM is booted you should have access to vCenter again. Isn’t that cool? What would happen if your vShield App would return? Would this vCenter VM be left unprotected? No it wouldn’t, vShield App would actually notice it is not protected and add the correct filter details again so that the vCenter VM will be protected. If you want to speed this process up you could of course also vMotion the VM to a host which is protected. Now keep in mind that while you do the vMotion it will insert the filter again which could cause the vCenter VM to disconnect. In all my tests so far it would reconnect at some point, but that is no guarantee of course.

Tomorrow I am going to apply a security policy which will lock out my vCenter Server and try to recover from that… I’ll keep you posted.

** Disclaimer: This is for educational purposes, please don’t try this at home… **

vShield App and layering your design

· ·

I started diving in to vShield App and one thing that I like about vShield App is that it allows you to use different types of objects to apply your policies to. Never really put too much thought in to it, but considering the world is more and more changing to policy based management this fits right in. I just wanted to share something that I was working on, any feedback / thoughts are welcome…

The VMware Cloud Infrastructure aims to reduce operational overhead and lower Total Cost of Ownership (TCO) by simplifying management tasks and abstracting complex processes. The focus of this architecture, as indicated by our customer requirements, is resource aggregation and isolation through the use of pools for each of the crucial pillars: network, storage and compute. Each of the three pillars will be carved in to multiple units of consumption with priority allocated based on their service level agreement. This will be achieved by leveraging core functionality offered by vSphere 5.0. Subsequently vShield App will be used to isolate each of the different type of workloads. As a hypervisor-based application-aware firewall solution, vShield App allows defining policies to logical, dynamic application boundaries (security groups) instead of physical boundaries.

This resource and security layering method will allow for a fast and safe deployment of new workloads.

Each of the different types of resources are carved up in to different groups for each of the respective workload types. A virtual machine, or vApp, will be deployed in one of the three different compute and security groups after which a specific networking group will be selected and a storage tier. Compute, Security and Network  group types are currently defined based on the different type of workloads this virtual infrastructure will host. In the future additional blocks may be added based on the requirements of the internal customers and the different types of workloads being deployed…

Management Cluster / vShield Resiliency?

· ·

I was reading Scott’s article about using dedicate clusters for management applications. Which was quickly followed by a bunch of quotes turned into an article by Beth P. from Techtarget. Scott mentions that he had posed the original question on twitter if people were doing dedicated management clusters and if so why.

As he mentioned only a few responded and the reason for that is simple, hardly anyone is doing dedicated management clusters these days. The few environments that I have seen doing it were large enterprise environments or service providers where this was part of an internal policy. Basically in those cases a policy would state that “management applications cannot be hosted on the platform it is managing”, and some even went a step further where these management applications were not even allowed to be hosted in the same physical datacenter. Scott’s article was quickly turned in to a “availability concerns” article by Techtarget to which I want to respond. I am by no means a vShield expert, but I do know a thing or two about the product and the platform it is hosted on.

I’ll use vShield Edge and vShield Manager as an example as in Scott’s article vCloud Director is mentioned which leverages vShield Edge. This means that vShield Manager needs to be deployed in order to manage the edge devices. I was part of the team who was responsible for the vCloud Reference Architecture but also part of the team who designed and deployed the first vCloud environment in EMEA. Our customer had their worries as well about resiliency of vShield Manager and vShield Edge, but as they are virtual they can easily be “protected” by leveraging vSphere features. One thing I want to point out though, if vShield Manager is down vShield Edge will continue to function so no need to worry there. I created the following table to display how vShield Manager and vShield Edge can be “protected”.

Product vShield Manager VMware HA VM Monitoring VMware FT
vShield Manager Yes (*) Yes Yes Yes
vShield Edge Yes (*) Yes Yes Yes

Not only would you be able to leverage these standard vSphere technologies there is more that can be leveraged:

Please don’t get me wrong here, there are always methods to get locked out but as Edward Haletky stated “In fact, the way vShield Manager locks down the infrastructure upon failure is in keeping with longstanding security best practices”. (Quote from Beth P’s article) I also would not want my door to be opened up automatically when there is something wrong with my lock. The trick though is to prevent a “broken lock” situation from occurring and to utilize vSphere capabilities in such a way that the last known state can be safely recovered if it would.

As always an architect/consultant will need to work with all the requirements and constraints  and based on the capabilities of a product come up with a solution that offers maximum resiliency and with the mentioned options above you can’t tell me that VMware doesn’t provide these

Creating a vCD Lab on your Mac/Laptop

· ·

I was just building a vCD Lab and thought I would document the process. I know Hany has done something similar recently but mine is slightly different. I wanted to have a slim config from a memory perspective and virtual machine count perspective. Before I start, let’s give a warning… ***this is totally unsupported***

Pre-requisites:

We will be creating multiple VMs but for the sake of simplicity will be combining functionality where possible. First you will need to install multiple ESXi hosts and a vCenter server. I am assuming all of you know how to do this so I won’t go into detail here. If you don’t drop me a comment. I did list some of the recommendations/requirements:

vCenter / DNS / ESXi

  • Create a VM with 1 vCPU and 1 GB of memory. I used a 20GB thin disk, which should be more than sufficient as we will not be using VUM.
  • Connect the Windows 2008 – 64 Bit ISO and walk through the standard installation process. I will not describe every step, as all of you should be able to install an OS. However the following is recommended:
    • Fixed IP Address
    • I changed the host name to “vcenter”
    • Install DNS
      • pre-populate DNS with records for your two esxi hosts, vShield Manager and your vCD server.
  • I will not tell you how to install ESXi or vCenter for that matter. Just ensure you have two ESXi hosts with shared storage in a DRS enabled cluster, those are the requirements. Preferably with some memory resource. I gave both my ESXi hosts 3GB. There are a couple of options for shared storage:
    • You could use Openfiler as your iSCSI target for ESXi hosts (preferred), if you don’t know how to set it up read this excellent this article by Kiwi_Si.
    • You could enable NFS on your CentOS which also hosts your vCD and Oracle database
    • If you are using VMware Workstation enable “clustering” of disks… I haven’t tested this in a while though.

Result: vCenter Server, 1 Cluster containing at least 2 ESXi hosts with DRS enabled.

vShield Manager

You could run vShield Manager as a VM within your virtualized ESXi host, but from a performance perspective that is probably not the smartest thing to do. So we are going to import it into Fusion. For those using Windows VMware Workstation is also fine, or even Player.

I guess this is the most tricky part of the whole setup, you will need to convert the vShield OVA to a VM. Now this is not a must, you can also run the vShield on your virtual ESXi hosts, but I like to avoid this for performance reasons. So this is how I converted it:

  • Go to the folder which contains the OVA and go into the OVA and copy all files included into a separate folder
  • Download the OVF Tool to convert the vShield Manager OVF Files to a format that Fusion supports
    • Open a terminal window and “cd” to the folder which contains “VMware-ovftool-2.0.1-260188-mac.i386.sh”
    • Make the script executable by typing the following:
      chmod +x VMware-ovftool-2.0.1-260188-mac.i386.sh
    • Run the installer script by typing the following:
      ./VMware-ovftool-2.0.1-260188-mac.i386.sh
    • Confirm the installation with “yes”
    • Accept the EULA with “yes”
    • Confirm the path by pressing enter/return
    • The install should complete literally within seconds
    • Go to the folder that contains the “OVF” file and type the following:
      /opt/vmware/ovftool/ovftool.bin “VSM.ovf” .
    • Accept the EULA by typing “yes”
    • The conversion should now start and when it is completed a new folder should be created which contains your VMX file and your VMDK files. These can be imported into Fusion.
    • Copy the VSM Folder to the place you store your local VMs and open the VM within Fusion and fire it up
  • Now that you have VSM running on your Laptop/Macbook you will need to configure it. These steps are pretty straight forward, but they will need to happen in order for VSM to function correctly:
    • Open the vShield Manager console and login with user “admin” and password “default”
    • Type “enable”, enter the password “default” again and type “setup” to configure your VSM
    • Enter your IP, Subnet, Gateway and DNS details and exit to ensure these are active
  • That is it! Now you can use your internet browser to see if you can login to your VSM “https://<ipaddress”

Result: vShield Manager running within Fusion.

vCD VM

  • Create a VM with 1 vCPU and 1 GB of memory. I used a 20GB thin disk, which should be more than sufficient.
  • Connect the CentOS 5 – 64 Bit ISO and walk through the standard installation process. I will not describe every step, as all of you should be able to install an OS. However the following is recommended:
    • Default partitioning scheme
    • Fixed IP Address
    • Disable IP v6
    • Server GUI install
  • After the install is done you will need to reboot the VM and configure the OS. I recommend the following:
    • Disable the Firewall
    • Disable SELinux
    • Enable NTP
    • Create an additional user
  • Now that the VM has rebooted again we will need to upgrade all packages to the latest version and install VMware Tools all the required packages:
    • Install VMware Tools (extract the files from the archive and run the installer via a terminal window by going to the path where you extracted it and type:
      ./vmware-install.sh
      use all the default settings
    • Open a terminal window and type the following:
      yum update
      yum upgrade
    • Now install all the Oracle and vCD required packages:
      yum install alsa-lib bash chkconfig compat-libcom_err coreutils findutils glibc grep initscripts krb5-libs libgcc libICE libSM libstdc libX11 libXau libXdmcp libXext libXi libXt libXtst module-init-tools net-tools pciutils procps sed tar which
  • Install Oracle 10g Express (again note that this isn’t officially supported):
    • Copy the Oracle RPM file to your vCD VM
    • Open a terminal window and go to the path where you copied the Oracle RPM file
    • rpm -i oracle-xe-10.2.0.1-1.0.i386.rpm
    • /etc/init.d/oracle-xe configure
    • Use the default ports (8080 and 1521)
    • Enter the password twice
    • Select “y” to ensure the database daemon is started when the VM restarts
  • After the Oracle 10g Express server has been installed test if you can actually access it by opening a web browser. Try http://<ipaddress>:8080/apex
  • I would recommend to create a new user for the vCD environment:
    • Click “Administration”
    • Go to “Database Users” and click “Create User”
    • I would recommend to give it the name “vcloud” and an easy to remember password. Also make sure you tick the “DBA” tick box.
    • Click “Create”
  • Now it is time to install vCD (copy the bin file to your vCD VM)
    • First we need to create a virtual interface so that we have two IP addresses that vCD can use. Of course you can also add a second NIC, but I use this method to keep the VM configuration as simple as I possibly can:
      • Open a terminal windows and type the following:
        nano /etc/sysconfig/network-scripts/ifcfg-eth0:1
      • Add the following to the file you just opened, of course add the approriate IP address and net mask!
        BOOTPROTO=static
        DEVICE=eth0:1
        IPADDR=<ip address>
        NETMASK=<net mask>
        ONBOOT=yes
      • Save the file and restart the network by typing the following:
        service network restart
      • When you do an “ifconfig” it should show you two devices…
    • Open a terminal window and go to the path where you copied the vCD BIN file and make the bin file executable:
      chmod +x vmware-cloud-director-1.0.0-285979.bin
    • type the following to do the install
      ./vmware-cloud-director-1.0.0-285979.bin
    • It will ask you if you want to run the installer on an unsupported distro, type “y”
    • It will ask you if you want to run the configuration script, type “n”
    • Next we will create self signed certificates, open a terminal window and do the following:
    • Go to /etc and copy and paste the following:
      /opt/vmware/cloud-director/jre/bin/keytool -keystore certificates.ks -storetype JCEKS -storepass password -genkey -keyalg RSA -alias http -dname “cn=vcloud,  ou=vmware, o=vmware, c=US” -keypass password
      /opt/vmware/cloud-director/jre/bin/keytool -keystore certificates.ks -storetype JCEKS -storepass password -genkey -keyalg RSA -alias consoleproxy -dname “cn=vcloud,  ou=vmware, o=vmware, c=US” -keypass password
    • Now you should have a file called “certificates.ks” in /etc
    • Next we will need to configure vCD, type the following to start the configuration:
      /opt/vmware/cloud-director/bin/configure
    • Select your first IP address, this will be the IP address which is used for vCD Portal access
    • Select your second IP address, this will be the IP address which is used for the VM Remote Console
    • Type the path to your certificates store, which is “/etc/certificates.ks
    • Type the password, which is password
    • Press enter to skip the “syslog server”
    • Enter the host (or IP address) for the database
      127.0.0.1
    • Press enter/return to use default database port (1521)
    • Type the database service name
      xe
    • Type the database username, in my case:
      vcloud
    • Type the database password, in my case:
      vmware
    • Now the database will be initialized and the vCD install will be  completed
    • Type “y” to start the vCD service
    • You can monitor the progress of the vCD service start up as follows
      tail -f /opt/vmware/cloud-director/log/cell.log
    • It will show you the percentage of the initialization of the application that has completed. Of course it should say “Application Initialization: Complete. Server is ready in” at some point.

Result: VM with both Oracle 10g Express and vCloud Director 1.0.

Final Steps

That is it for the command-line stuff… All we need to do now is configure vCD through the web interface… here we go:

  • Open a browser and point it to “https://<vCloud Director Address>/cloud/
  • Click “Next” on the welcome screen
  • “Accept” the License Agreement
  • Type your license key and click “Next”
  • Create an Administrator account and type a password and click “Next”
  • Give the system a name, I called it “vCD”, and click “Next”
  • Review your settings and click “Finish” if they look okay

Now you should be presented with the following screen and you should be good to go!

So what’s next? Hany has listed a nice set of videos in his article that will describe how to create a Provider vDC, how to attach a vCenter server etc. Go ahead play around, have fun… enjoy the vCloud!