This is what the log file would typically look like:

iscsid: cannot make a connection to 192.168.1.20:3260 (101,Network is unreachable)
iscsid: Notice: Reclaimed Channel (H34 T0 C1 oid=3)
iscsid: session login failed with error 4,retryCount=3
iscsid: Login Target Failed: iqn.1984-05.com.dell:powervault.md3000i.6002219000a14a2b00000000495e2886 if=iscsi_vmk@vmk8 addr=192.168.1.20:3260 (TPGT:1 ISID:0xf) err=4
iscsid: Login Failed: iqn.1984-05.com.dell:powervault.md3000i.6002219000a14a2b00000000495e2886 if=iscsi_vmk@vmk8 addr=192.168.1.20:3260 (TPGT:1 ISID:0xf) Reason: 00040000 (Initiator Connection Failure)

This is explained in KB 2007108 which also contains the download link. Make sure to download it and update your environment if you are running iSCSI.

We checked multiple whitepapers before we went onsite and our primary source was this excellent article by Scott Drummonds. After start vscsiStats and receiving a “successful started”  we waited for 15 minutes and verified if we could see any data at all. Unfortunately we did not see anything. What is happening here? We checked the build/patch level and it was ESX 3.5 Update 4. Nothing out of the ordinary I would say. After trying several VMs we still did not see anything with “vscsiStats -s -w <worldID>”. For some weird reason, in contrary to what all blog articles are stating and what Scott Drummonds states we had to use the following command:

vscsiStats -s -t -w <worldID>

This might not be the case in most situations, but again we had to add “-t” to capture any data. You can find the world ID of the VM you want to monitor the performance by using the following command:

vscsiStats -l

After a couple of minutes you can verify if any data is being collected by using the following command:

vscsiStats -p all -w <worldID>

If you want to save your data in a CSV file to import it in Excel use the following:

vscsiStats -p all -c -w <worldID> > /tmp/vmstats-<vmname>.csv

Don’t forget to stop the monitoring:

vscsiStats -x -w <worldID>

So what’s the outcome of this all? Well with vscsiStats you can create great diagrams which for instance show the latency. This can be very useful in NFS environments as esxtop does not show this info:

If you don’t want to do this by hand, check out this article by Gabe.

COMSTAR is a software framework that enables you to turn any OpenSolaris host into a SCSI target that can be accessed over the network by initiator hosts. COMSTAR breaks down the huge task of handling a SCSI target subsystem into independent functional modules. These modules are then glued together by the SCSI Target Mode Framework (STMF).

Mike wrote a really extensive article on how-to setup COMSTAR. I must admit it looks promising, and it’s open source… freely available! Also be sure to check out these pages on the OpenSolaris section of SUN’s website.

A couple of outtakes:

I’ve been working with 10 joint VMware/EMC customers this week in NY, NJ and Houston (phew!), and was in Australia the week before last where there were 2 more. Out of those 12, 4 asked me questions about the applicability of “stretching” their ESX clusters across geographic distances – that’s 33%, and absolutely above the “man, I should write a blog on the topic” threshold.

So, what are we talking about?

A stretched cluster is the practice of having ESX member servers in a cluster that are geographically separated. The reason this is generally done is to provide the ability to dynamically move workloads from one datacenter to another. Often, the customer is also considering it for disaster recovery purposes (“I’ll just VMotion in case of a disaster”). Can this be done – ABSOLUTELY – but not considered lightly.

read more…

I guess it was inevitable, but it’s still depressing. Traveling around the world means I read a LOT of magazines – there’s that 15 minutes of airplane ascent and decent where my usual toys (PSP, iPod, DS) are verboten. Some stuff (like the Economist) I read to expand my horizons, some stuff (like Maximum PC) I read as the nerd equivalent of Maxim (completely vacuous brain mush).

I couldn’t resist the headline of this month’s Windows IT Pro: “Virtualization Wars: Hyper-V vs. ESX Server ”

read more…

I am so not into protocol and transport wars – BUT that still doesn’t change the fact that the future is Ethernet-connected. So, then what about protocol? iSCSI, NFS, or FCoE? Well – NFS will continue to do well – it works well, there’s nothing wrong with it – and it will always have the strengths that it has in the VMware context (so easy to create massive datastores that span ESX clusters or even sites). iSCSI will continue to grow wildly (it is the fastest growing in the market at large, and in EMC’s portfolio) and is (IMHO – I’m still in love) the future of the block storage market en masse. BUT, I’m starting to come around on FCoE.

read more…

The cool thing about the Celerra is the possibility to use “network load balancing”. Where most iSCSI SAN’s work with a group address, which forces ESX to always walk the same path, the Celerra SAN’s use multiple interfaces with multiple ip-adresses. Would like to see this in real life…

There are three so called “transport modes”:

1. SAN – Is the default option and requires iSCSI or a SAN based VMFS which must be directly accessible from both the ESX and the VCB proxy host.
2. NBD – Is the over-the-network protocol to access the virtual harddisks. Data is sent unencrypted over the network to the VCB proxy. Does not require a SAN or iSCSI and is supported within a VM.
3. NBDSSL – Well the name already says enough, same as option 2 but data is transferred with the use of SSL, encrypted. Note that this will be slower and will put more stress on the CPU of the ESX Host. Only supported on ESX 3.5 and Virtual Center 2.5.

For more info check out the PDF about VCB 1.1 on the VMware website.

Jumbo frames for a vSwitch are supported. Most people would want to enable this for the iSCSI initiator, but:

Jumbo Frames allow ESX Server 3 to send larger frames out onto the physical network. The network must support Jumbo Frames end‐to‐end for Jumbo Frames to be effective. Jumbo Frames up to 9kB (9000 Bytes) are supported. iSCSI with Jumbo Frames is not supported!

In other words, your vSwitch and VMkernel support Jumbo Frames but the iSCSI initiator doesn’t. I don’t really understand why VMware did this, but it probably has got something to do with the release date of 3.5 and proper testing this new functionality. If you want to take the the risk here’s how you enable jumbo frames on your vSwitch or VMkernel:

• VMkernel command: esxcfg-vmknic -a -i <ip-address vmkernel> -n <netmask vmkernel> -m 9000 <portgroupname>
• vSwitch command: esxcfg-vswitch -m 9000 <vSwitchx>

for more info check the server configuration guide.

This is caused by the fact that the “Software iSCSI Client” isn’t enabled on the ESX firewall. November the 15th VMware released a couple of patches which addressed a bug. This bug made it possible to connect via the iSCSI software initiator without enabling the service on the ESX firewall. To open up the port open a console session to the ESX host and type “esxcfg-firewall -e swISCSIClient” or enable it via VirtualCenter “Configuration -> Security Profile -> Properties -> Check Software iSCSI Client”.