vSphere 5.0 HA: Application Monitoring intro

I don’t think anyone has blogged about App Monitoring yet so I figured I would do a “what’s new / intro” to App Monitoring in vSphere 5.0. Prior to vSphere 5 App Monitoring could only be leveraged by partners which had access to the SDK/APIs. A handful of partners leveraged those of which probably Symantec’s ApplicationHA is the best example. The “problem” with that though is that you would still need to buy a piece of software while you might have in-house development who could easily bake this into their application… well with vSphere 5 you can. I grabbed one of the latest code drops and started playing around. Note that I am not going to do an extensive article on this. Just showing what you have after installing the package. In my case I installed it on a Windows VM.

Now first of all after installing the package you will have new executable. This executable allows you to control App Monitoring offers without the need to compile a full binary yourself. This new command, vmware-appmonitoring.exe, takes the following arguments, which are not coincidentally similar to the functions I will show in a second:

Enable
Disable
markActive
isEnabled
getAppStatus

When running the command the following output is presented:

C:\VMware-GuestAppMonitorSDK\bin\win32>vmware-appmonitor.exe
Usage: vmware-appmonitor.exe {enable | disable | markActive | isEnabled | getApp Status}

Now I guess most parameters speak for itself. “Enable” will allow you to switch on App Monitoring and “Disable” turns it off again. “IsEnabled” will give you the current status, is it on or off? The “getAppStatus” tells what the status is of your app, is it healthy and has it been sending heartbeats regularly, well than the result will be green if there is a real issue than it will be red. (There’s also gray which means HA just picked up on the VM it’s status needs to be cleared and monitoring should be started soon) Now the one that is most important is “markActive”. This parameter needs to be called at least every 30 seconds. This is the heartbeat parameter. In other words “markActive” is what informs HA that the application is still alive!

I am sure that as soon as William Lam gets his hands on the package he will go wild and release a bunch of scripts which will allow you to enhance resiliency for application/service. These parameters can also be used by your development team, but in the form of a function. The Application Awareness API allows for anyone to talk to it using different types of languages like C++ and Java for instance. Currently there are 6 functions defined:

VMGuestAppMonitor_Enable()
Enables Monitoring
VMGuestAppMonitor_MarkActive()
Mark application as active, recommend to call this at least every 30 seconds
VMGuestAppMonitor_Disable()
Disable Monitoring
VMGuestAppMonitor_IsEnabled()
Returns status of Monitoring
VMGuestAppMonitor_GetAppStatus()
Returns the current application status recorded for the application
VMGuestAppMonitor_Free()
Frees the result of the VMGuestAppMonitor_GetAppStatus() call

These functions could be used by your development team to enhance resiliency in a simple way. This is just the start however, I personally would like to see some sort of rolling patch process added on top and for instance the ability to restart service or have a partial VM failure. Or even the hint the hypervisor that there is a partial failure and request a vMotion to a different host to validate if that solves the problem… If you feel there’s something that needs to be added to App Monitoring let me know and I’ll make sure the PM/Dev Team reads this thread.

** disclaimer: some of this info was taken from the vSphere 5.0 Technical Deepdive book **

I briefly touched on this topic already, but I guess on Yellow-Bricks the primary focus should be technical. I downloaded Symantec’s Application HA and decided to give it a spin, I already started documenting the installation of the ApplicationHA Console and the Guest Component but Mr Sloof beat me to it and documented his finding in the following two articles.

No need to repeat it in my opinion as Eric did an excellent job. I do want to point two things out and they are around the requirements/constraints of the product:

Symantec ApplicationHA Console requires Windows Server 2008 or Windows Server 2008 R2.
- The Console provides the integration between vCenter and ApplicationHA from a UI perspective. An ApplicationHA tab will be added to the vSphere Client which can be used to configure and control ApplicationHA.
A 64Bit Guest OS is required for the ApplicationHA Windows Guest Component.
- This is the actually component that enables application level protection.

This basically means that if you are primarily using 32Bit OS’s, like I for instance do in my homelab, it won’t work as the installer literally throws an error that the VM doesn’t meet the requirements. That is for now, as I bet that when demand grows Symantec will add support for 32Bit OSs.

So what is this ApplicationHA product?

Symantec ApplicationHA is an extension of VM / Application Monitoring. Symantec simplified Veritas Cluster Server (VCS) to enable application availability monitoring including of course responding to issues. Just to be absolutely clear, the foundation still is VCS based the unnecessary bits and pieces were taken out. Note that it is not a multi-node clustering solution like VCS itself but a single node solution.

The question remains how does it integrate with vCenter and more specifically with HA?

Lets start with the integration with vCenter. This is where the “ApplicationHA Console” comes into play. It basically adds a tab on a VM level that gives you the details of this VM in terms of ApplicationHA of course depending on the fact if you installed the Guest Component or not the Tab will display different info. Ultimately it should look as follows:

Now this example shows IIS being monitoring, but not only IIS also a disk mount. By the way, besides having a standard agent for IIS; ApplicationHA also includes an agent for MS SQL and MS Exchange. My primary focus for this article is Windows by of course Symantec also offers agents for Linux which includes Oracle, SAP and Weblogic. Just because your application is not included today it doesn’t mean you are out of luck. Symantec regularly updates the ApplicationHA Agent pack. However there is also a standard Guest Component which enables you to monitor standard services, mountpoints, processes and is what Symantec calls an Infrastructure Agent.

That brings me to the next point. From a GUI perspective it seems like there is a single component that monitors the state of your application. That is not the case, as stated above it can monitor the availability of services, process and much more. As stated this is what Symantec calls the Infrastructure Agent and yes it has multiple components. I have listed the most crucial agents below with a short description:

Heartbeat Agent – The Heartbeat agent monitors the configured application service group.
MountMonitor Agent – The MountMonitor agent monitors the mount path of the configured storage. It is independent of how the underlying storage is managed (whether SFW disk groups or LDM disks or any other storage management software). The mount path can be a drive letter or a folder mount.
GenericService Agent – The GenericService agent brings services online, takes them offline, and monitors their status. A service is an application type that is supported by Windows, and conforms to the interface rules of the Service Control Manager (SCM).
Process Agent – The Process agent brings processes online, takes them offline, and monitors their status. You can specify different executables for each process routine.

Each of these agents has very specific configuration attributes. An example for instance would be the “DelayBeforeAppFault” for the Heartbeat Agent. This specifies the number of seconds the agent must wait in case of an error before communicating application fault to VMware HA. I guess I just hit the nail on the head. The Heartbeat Agent is the agent that is responsible for communication with VMware HA. Basically it provides a means to trigger VMware HA to kick in when Symantec can not handle the issue with a restart of the service.

So what happens if an Application that has been configured for protection fails?

First Symantec ApplicationHA is triggered to try to get the Application up and running after the failure occurred by restarting the application. It should be noted that Symantec’s ApplicationHA is aware of dependencies and knows in which order services should be started or stopped. If however for whatever reason this fails for an “X” amount of times VMware HA will be asked to take action. (X is configurable by changing the config attribute “AppRestartAttempts”.) The action that HA takes will be a restart of the virtual machine.

But wait is it really VMware HA? With that meaning is it the good old “AAM” agent who is responsible for this action? No it isn’t. In this case both “vpxa” and “hostd” are responsible for the action taken. I guess visualizing the process will make it a bit easier to digest:

I hope this makes it a bit more clear on how the product works and how it integrates with VMware HA itself.

applicationHA