Yellow Bricks

by Duncan Epping

VMworld esxtop advanced session

· ·

During my flight from Boston back to the Netherlands I listened to the VMworld esxtop session “Troubleshooting using ESXTOP for Advanced Users, TA6720“. As always an excellent session with a lot of in-depth info. Most of it was already documented, however there were a couple of key points that I hadn’t documented yet. I just added those to my esxtop page which I wanted to stress as I personally believe it is very useful info. It seems pretty random but it rolled up nicely into the esxtop page in my opinion.

  • %SYS should be less than 20, %SYS is the percentage of time spent by system services on behalf of the world. The possible system services are interrupt handlers, bottom halves, and system worlds.
  • -b = batch mode, adding “-a” will force all metrics to be gathered
  • Limit display to a single group (l)
    • enables you to focus on a specific VM
  • Limiting the number of entities (#)
    • this enables you for instance to watch the top 5 worlds for

I have also added thresholds for ZIP/s, UNZIP/s and CACHEUSD. These should of course be 0 from a performance perspective as anything larger than 0 means the host was overcommitted on memory and had to resort to memory compression.

If anyone has more metrics/thresholds to contribute which they used in the past to troubleshoot issues let me know!

How many pages can be shared if Large Pages are broken up?

· ·

I have written multiple articles(1, 2, 3, 4) on this topic so hopefully by now everyone knows that Large Pages are not shared by TPS. However when there is contention the large pages will be broken up in small pages and those will be shared based on the outcome of the TPS algorythm. Something I have always wondered and discussed with the developers a while back is if it would be possible to have an indication of how many pages can possibly be shared when Large Pages would be broken down. (Please note that we are talking about Small Pages backed by Large Pages here.) Unfortunately there was no option to reveal this back then.

While watching the VMworld esxtop session “Troubleshooting using ESXTOP for Advanced Users, TA6720” I noticed something really cool. Which is shown in the quote and the screenshot below which is taken from the session. Other new metrics that were revealed in this session and shown in this screenshot are around Memory Compression. I guess the screenshot speaks for itself.

  • COWH : Copy on Write Pages hints – amount of memory in MB that are potentially shareable,
  • Potentially shareable which are not shared. for instance when large pages are used, this is a good hint!!

There was more cool stuff in this session that I will be posting about this week, or at least adding to my esxtop page for completeness.

The Dutch VMUG Event, 10th of December 2010!

· ·

The Dutch VMUG event has always been infamous for the massive amount of attendees and the excellent agenda. Of course the 2010 edition will be no different. The VMUG Team secured a great selection of speakers, and with roughly 700 attendees I guess you could say that it is a mini-VMworld.

Just for those wondering who will be speaking and what the topics are I made a short summary which is listed below. For the Dutch speaking you can find it as well on the VMUG website.

  • VMware Keynote by Richard Garsthagen
  • VMware vCloud Director by Willem van Engeland (VMware, Specialist SE Cloud) and Duncan Epping
  • vSphere 4 Backup by Doug Hazelman (VEEAM)
  • vSphere Disaster Recovery by Gabrie van Zanten (Openline / Gabesvirtualworld.com)
  • vSphere Introduction by Viktor van den Berg (XTG / VMUG Leader)
  • Advanced Troubleshooting by Eric Sloof (NTPro.nl)
  • HA & DRS Q&A by Frank Denneman (VMware) and Duncan Epping
  • Project VRC Phase III by Ruben Spruijt (PQR) and Jeroen v.d. Kamp (Login)
  • PowerCLI is for Administrators by Alan Renouf (EMC) / Luc Dekens
  • VDI & Storage = Deep Impact by Herco van Brug
  • Workshop: Customizing vEcoshell for your Environment by Scott Herrold (Quest)

These are not all sessions, but these are the ones I personally wanted to highlight. Just look at those names…. it won’t get any better than that. If you don’t want to miss out on these sessions make sure you sign up now because I bet it will be sold out quickly.

By the way, as Frank and I will be doing a Q&A session on HA & DRS I will be publishing a blog post soon to start gathering questions… So if there is anything regarding HA or DRS you always wanted to have explained let us know. This is your chance.

How does Symantec ApplicationHA integrate/work with VMware HA?

· ·

I briefly touched on this topic already, but I guess on Yellow-Bricks the primary focus should be technical. I downloaded Symantec’s Application HA and decided to give it a spin, I already started documenting the installation of the ApplicationHA Console and the Guest Component but Mr Sloof beat me to it and documented his finding in the following two articles.

No need to repeat it in my opinion as Eric did an excellent job. I do want to point two things out and they are around the requirements/constraints of the product:

  • Symantec ApplicationHA Console requires Windows Server 2008 or Windows Server 2008 R2.
    • The Console provides the integration between vCenter and ApplicationHA from a UI perspective. An ApplicationHA tab will be added to the vSphere Client which can be used to configure and control ApplicationHA.
  • A 64Bit Guest OS is required for the ApplicationHA Windows Guest Component.
    • This is the actually component that enables application level protection.

This basically means that if you are primarily using 32Bit OS’s, like I for instance do in my homelab, it won’t work as the installer literally throws an error that the VM doesn’t meet the requirements. That is for now, as I bet that when demand grows Symantec will add support for 32Bit OSs.

So what is this ApplicationHA product?

Symantec ApplicationHA is an extension of VM / Application Monitoring. Symantec simplified Veritas Cluster Server (VCS) to enable application availability monitoring including of course responding to issues. Just to be absolutely clear, the foundation still is VCS based the unnecessary bits and pieces were taken out. Note that it is not a multi-node clustering solution like VCS itself but a single node solution.

The question remains how does it integrate with vCenter and more specifically with HA?

Lets start with the integration with vCenter. This is where the “ApplicationHA Console” comes into play. It basically adds a tab on a VM level that gives you the details of this VM in terms of ApplicationHA of course depending on the fact if you installed the Guest Component or not the Tab will display different info. Ultimately it should look as follows:

Now this example shows IIS being monitoring, but not only IIS also a disk mount. By the way, besides having a standard agent for IIS; ApplicationHA also includes an agent for MS SQL and MS Exchange. My primary focus for this article is Windows by of course Symantec also offers agents for Linux which includes Oracle, SAP and Weblogic. Just because your application is not included today it doesn’t mean you are out of luck. Symantec regularly updates the ApplicationHA Agent pack. However there is also a standard Guest Component which enables you to monitor standard services, mountpoints, processes and is what Symantec calls an Infrastructure Agent.

That brings me to the next point. From a GUI perspective it seems like there is a single component  that monitors the state of your application. That is not the case, as stated above it can monitor the availability of services, process and much more. As stated this is what Symantec calls the Infrastructure Agent and yes it has multiple components. I have listed the most crucial agents below with a short description:

  • Heartbeat Agent – The Heartbeat agent monitors the configured application service group.
  • MountMonitor Agent – The MountMonitor agent monitors the mount path of the configured storage. It is independent of how the underlying storage is managed (whether SFW disk groups or LDM disks or any other storage management software). The mount path can be a drive letter or a folder mount.
  • GenericService Agent – The GenericService agent brings services online, takes them offline, and monitors their status. A service is an application type that is supported by Windows, and conforms to the interface rules of the Service Control Manager (SCM).
  • Process Agent – The Process agent brings processes online, takes them offline, and monitors their status. You can specify different executables for each process routine.

Each of these agents has very specific configuration attributes. An example for instance would be the “DelayBeforeAppFault” for the Heartbeat Agent. This specifies the number of seconds the agent must wait in case of an error before communicating application fault to VMware HA. I guess I just hit the nail on the head. The Heartbeat Agent is the agent that is responsible for communication with VMware HA. Basically it provides a means to trigger VMware HA to kick in when Symantec can not handle the issue with a restart of the service.

So what happens if an Application that has been configured for protection fails?

First Symantec ApplicationHA is triggered to try to get the Application up and running after the failure occurred by restarting the application. It should be noted that Symantec’s ApplicationHA is aware of dependencies and knows in which order services should be started or stopped. If however for whatever reason this fails for an “X” amount of times VMware HA will be asked to take action. (X is configurable by changing the config attribute “AppRestartAttempts”.) The action that HA takes will be a restart of the virtual machine.

But wait is it really VMware HA? With that meaning is it the good old “AAM” agent who is responsible for this action? No it isn’t. In this case both “vpxa” and “hostd” are responsible for the action taken. I guess visualizing the process will make it a bit easier to digest:

I hope this makes it a bit more clear on how the product works and how it integrates with VMware HA itself.