Pluggable Storage Architecture, exploring the next version of ESX/vCenter

The next version of ESX has a totally different architecture for storage. The new architecture is called “Pluggable Storage Architecture”. For my own understanding I wanted to write down how this actually works and what all the different abbreviations/acronyms mean:

• PSA = Pluggable Storage Architecture
• NMP = Native Multipathing
• MPP = Multipathing Plugin
• PSP = Path Selection Plugin
• SATP = Storage Array Type Plugin

At the top level we have “Pluggable Storage Architecture”. This is just the name of the new concept, but it’s a well chosen name cause that’s what it is… a new storage architecture that uses plugins. Let’s start with the native VMware plugins.

The Native Multipathing(NMP) module is the default module that ESX(as of vSphere) uses. If your array is listed on the compatibility list then VMware pre-decides which multipathing “algorithm” will be used. ESX will natively do: Fixed, MRU, Round-Robin. Of course you can change this “algorithm” if you want or need to, but be careful.

The NMP associates paths with the LUN and/or array. The NMP uses rules to decide which Storage Array Type Plugin(SATP) will be used and which Path Selection Plugin(PSP) will be used. If you want to edit these rules you can use “esxcli”, and you can find the rules in the “/etc/vmware/esx.conf”. For instance the default PSP for the EMX Symmetrix array is Fixed:

/storage/plugin/NMP/config[VMW_SATP_SYMM]/defaultpsp = "VMW_PSP_FIXED"

Now that these basics have been decided the two “sub” plugins take over. The SATP handles path fail-overs, it’s as simple as that. This of course will be reported back to the NMP, because the NMP is responsible for the i/o flow. The SATP will monitor the health of a path and take the required action depending on the type of array. Nothing more, nothing less.

Now you are probably wondering what the PSP does, or when it will be utilized. The PSP determines which path will be used for an I/O request, in other words the PSP deals with the Fixed, MRU and Round-Robin algorithms. Keep in mind that the type of PSP that will be used has already been pre-configured, it’s not the PSP that decides if it will do Fixed or MRU, it’s the NMP that decides this based on pre-defined rules of which an example has been showed above. Both the SATP and PSP are plugins of the NMP and are controlled by the NMP.

You can of course change the default behavior. In my post on iSCSI load balancing we’ve seen the “esxcli” in action. This command also interacts with the NMP. For instance you can easily check which SATPs are available at the moment:

[[email protected] ~]# esxcli nmp satp list
Name                 Default PSP    Description
VMW_SATP_ALUA_CX     VMW_PSP_FIXED  Supports EMC CX that use the ALUA protocol
VMW_SATP_SVC         VMW_PSP_FIXED  Supports IBM SVC
VMW_SATP_MSA         VMW_PSP_MRU    Supports HP MSA
VMW_SATP_EQL         VMW_PSP_FIXED  Supports EqualLogic arrays
VMW_SATP_INV         VMW_PSP_FIXED  Supports EMC Invista
VMW_SATP_SYMM        VMW_PSP_FIXED  Supports EMC Symmetrix
VMW_SATP_LSI         VMW_PSP_MRU    Supports LSI and other arrays
VMW_SATP_EVA         VMW_PSP_FIXED  Supports HP EVA
VMW_SATP_DEFAULT_AP  VMW_PSP_MRU    Supports non-specific active/passive arrays
VMW_SATP_CX          VMW_PSP_MRU    Supports EMC CX that do not
                                    use the ALUA protocol
VMW_SATP_ALUA        VMW_PSP_MRU    Supports non-specific arrays that use
                                    the ALUA protocol
VMW_SATP_DEFAULT_AA  VMW_PSP_FIXED  Supports non-specific active/active arrays
VMW_SATP_LOCAL       VMW_PSP_FIXED  Supports direct attached devices

This example also illustrates that a specific SATP is linked to a specific PSP. You can also check which PSPs are available at the moment:

[[email protected] ~]# esxcli nmp psp list
Name           Description
VMW_PSP_MRU    Most Recently Used Path Selection
VMW_PSP_RR     Round Robin Path Selection
VMW_PSP_FIXED  Fixed Path Selection

At the moment there are only three PSPs available, but you can imagine that vendors develop their own PSP to optimize load balancing for their array. The vendors can even write their own SATPs. You can also use esxcli to change the rules or add a rule, although I would not recommend doing this.

What’s next? That would be a Multipathing Plugin(MPP). An MPP can be seen as “NMP+SATP+PSP”. An MPP combines all the three functionalities into one module which, these MPP’s will be developed by the vendors of the arrays. EMC is one of the vendors that already demonstrated their MPP. You can see a cool demo on the blog of Chad Sakac, which is EMC’s MPP also known as Powerpath for ESX.