Intro
Thresholds
Howto – Run
Howto – Capture
Howto – Analyze
References
Changelog

This page is solely dedicated to one of the best tools in the world for ESX; esxtop.

I am a huge fan of esxtop! I read a couple of pages of the esxtop bible every day before I go to bed. Something I however am always struggling with is the “thresholds” of specific metrics. I fully understand that it is not black/white, performance is the perception of a user in the end.

There must be a certain threshold however. For instance it must be safe to say that when %RDY constantly exceeds the value of 20 it is very likely that the VM responds sluggish. I want to use this article to “define” these thresholds, but I need your help. There are many people reading these articles, together we must know at least a dozen metrics lets collect and document them with possible causes if known.

Please keep in mind that these should only be used as a guideline when doing performance troubleshooting! Also be aware that some metrics are not part of the default view, I’ve added the character you need to add to the default view for your convenience. You can add fields to an esxtop view by clicking “f” on followed by the corresponding character.

I used VMworld presentations, VMware whitepapers, VMware documentation, VMTN Topics and of course my own experience as a source and these are the metrics and thresholds I came up with so far. Please comment and help build the main source for esxtop thresholds.

Although understanding all the metrics esxtop provides seem to be impossible using esxtop is fairly simple. When you get the hang of it you will notice yourself staring at the metrics/thresholds more often than ever. The following keys are the ones I use the most.

open console session or ssh to ESX(i) and type:

esxtop

By default the screen will be refreshed every 5 seconds, change this by typing:

s 2

Changing views is easy type the following keys for the associated views:

c = cpu
m = memory
n = network
i = interrupts
d = disk adapter
u = disk device (includes NFS as of 4.0 Update 2)
v = disk VM
y = power states

V = only show virtual machine worlds
2 = highlight a row, moving down
8 = highlight a row, moving up
4 = remove selected row from view
e = statistics broken down per world
6 = statistics broken down per world

Ad/Remove fields:

f
<type appropriate character>

Changing the order:

o
<move field by typing appropriate character uppercase = left, lowercase = right>

Saving all the settings you’ve changed:

W

Keep in mind that when you don’t change the filename it will be saved and used as default settings.
Help:

?

In very large environments esxtop can high CPU utilization due to the amount of data that will need to be gathered and calculations that will need to be done. If CPU appears to highly utilized due to the amount of entities (VMs / LUNs etc) a command line option can be used which locks specific entities and keeps esxtop from gathering specific info to limit the amount of CPU power needed:

esxtop -l

More info about this command line option can be found here.

First things first. Make sure you only capture relevant info. Ditch the metrics you don’t need. In other words run esxtop and remove/add(f) the fields you don’t actually need or do need! When you are finished make sure to write(W) the configuration to disk. You can either write it to the default config file(esxtop4rc) or write the configuration to a new file.

Now that you have configured esxtop as needed run it in batch mode and save the results to a .csv file:

esxtop -b -d 2 -n 100> esxtopcapture.csv

Where “-b” stands for batch mode, “-d 2″ is a delay of 2 seconds and “-n 100″ are 100 iterations. In this specific case esxtop will log all metrics for 200 seconds.

You can use multiple tools to analyze the captured data.

1. perfmon
2. excel
3. esxplot

Let’s start with perfmon as I’ve used perfmon(part of Windows also know as “Performance Monitor”) multiple times and it’s probably the easiest as many people are already familiar with it. You can import a CSV as follows:

1. Run: perfmon
2. Right click on the graph and select “Properties”.
3. Select the “Source” tab.
4. Select the “Log files:” radio button from the “Data source” section.
5. Click the “Add” button.
6. Select the CSV file created by esxtop and click “OK”.
7. Click the “Apply” button.
8. Optionally: reduce the range of time over which the data will be displayed by using the sliders under the “Time Range” button.
9. Select the “Data” tab.
10. Remove all Counters.
11. Click “Add” and select appropriate counters.
12. Click “OK”.
13. Click “OK”.

The result of the above would be:

With MS Excel it is also possible to import the data as a CSV. Keep in mind though that the amount of captured data is insane so you might want to limit it by first importing it into perfmon and then select the correct timeframe and counters and export this to a CSV. When you have done so you can import the CSV as follows:

1. Run: excel
2. Click on “Data”
3. Click “Import External Data” and click “Import Data”
4. Select “Text files” as “Files of Type”
5. Select file and click “Open”
6. Make sure “Delimited” is selected and click “Next”
7. Deselect “Tab” and select “Comma”
8. Click “Next” and “Finish”

All data should be imported and can be shaped / modelled / diagrammed as needed.

Another option is to use a tool called “esxplot“. You can download version 1.0 here.

1. Run: esxplot
2. Click File -> Import -> Dataset
3. Select file and click “Open”
4. Double click host name and click on metric

As you can clearly see in the screenshot above the legend(right of the graph) is too long. You can modify that as follows:

1. Click on “File” -> preferences
2. Select “Abbreviated legends”
3. Enter appropriate value

The following documents / articles have been used as a reference:

• Interpreting esxtop Statistics
• Performance Troubleshooting for VMware vSphere 4 and ESX 4.0
• Hypervisor.fr – Easter Eggs esxtop
• Performance Training
• Using Perfmon for esxtop
• esxplot
• remove vertical lines from perfmon

07-01-2010 | decreased %RDY from 20 to a value of 10
22-01-2010 | added CPU –> TIMER/S
22-01-2010 | added MEM –> N%L
24-01-2010 | added sections (howto)
02-02-2010 | expanded analyze section and included screenshots
10-02-2010 | decreased %CSTP from 100 to 5
10-02-2010 | decreased KAVG from 5 to 2
23-03-2010 | increase %SWPWT from 1 to 5
23-03-2010 | added “e”, “V”, “i”, “2″, “4″, “6″, “8″ in the “views” section
16-06-2010 | added “-l” functionality and stressed NFS added option