Yellow Bricks

by Duncan Epping

Software Defined

Introducing startup PernixData – Out of stealth!

· ·

There are many startups out there that do something with storage these days. To be honest, many of them do the same thing and at times I wonder why on earth everyone focuses on the same segment and tries to attack it with the same product / feature set. One of the golden rules for any startup should be that you have a unique solution that will sell itself. Yes I realize that it is difficult, but if you want to succeed you will need to stand out.

About a year ago Satyam Vaghani (former VMware principal engineer who was responsible for VMFS, VAAI, VVOLs etc.) and Poojan Kumar (former VMware Data products lead and ex-Oracle Exadata founder) decided to start a company – PernixData. PernixData was conceptualized based on their experiences working on the intersection of virtualization, flash based storage and data. Today PernixData is revealed to the world. For those who don’t know, Pernix means “agile”. But what is PernixData about?

How many of you haven’t experienced storage performance problems? It probably is, in fact, the number one bottleneck in most virtualized environments. Convincing your manager (director / VP) that you need a new ultra-fast (and expensive) storage device is not easy; far from it. On top of that, data will always hit the network first before being acknowledged and every read will go over your storage network. How cool would it be if there was a seamless software solution that solves all your storage performance problems without you requiring to rip and replace your existing storage assets?

Server-side flash overcomes problems associated with network based storage and server-side caching solutions provide some respite. Yet, server-side caching solutions usually neither satisfy enterprise class requirements for availability nor transparently support clustered hypervisor features such as VMware vMotion. In addition, while they accelerate reads they fail to do much for writes. Customers are then stuck between either overhauling their entire storage infrastructure or going with caching solutions that work for limited use cases. PernixData is about to release a cool new product – a flash virtualization platform – that bridges this gap. By picking up where hypervisors left off, PernixData is planning to become the VMware of server flash and is aiming to do to server flash what VMware did to CPU and memory. So, what is this flash virtualization platform and why would you need it?

PernixData’s flash virtualization platform virtualizes all flash resources across all server nodes in a vCenter Server cluster into a single high-performance, enterprise class data tier. The great thing is that this happens in a transparent way. PernixData sits completely within the hypervisor and in the data-path of your virtual machine. Note that there are no requirements to install anything in the guest (virtual machine). PernixData is not a virtual appliance because virtual appliances introduce performance overhead and would need to be managed with all costs and complexity associated.

PernixData is also flash technology agnostic. It can leverage SSD or PCIe flash (or both) within the platform. The nice thing is that PernixData uses a scale-out architecture. As you add hosts with flash they can be dynamically added to the platform. On top of that, PernixData does both read and write acceleration while providing full data protection and is fully compatible with VM mobility solutions like vMotion, Storage vMotion, HA, DRS and Storage DRS.

Even more exciting PernixData will support both Write-through and Write-back modes. The cool part is that PernixData also ensures IO is replicated for high availability purposes. You don’t want to run your VM in Write-back mode when you cannot guaranteed data is highly available right?! I guess that is one of the unique selling points of the solution. A distributed, scale out, flash virtualization platform which is not only flash agnostic but also non-disruptive for your virtual workloads.

I would imagine this is many times cheaper than buying a new storage array. Even without knowing what the cost of PernixData will be, or which flash device (PCIe or SSD) you would decide to use… I bet when it comes to overall costs of the solution (product + implementation costs) it will be many many times cheaper.

As I started off with, the golden rule for any startup should be that they have a unique solution that sells itself. I am confident that PernixData FVP has just that by being a disruptive technology that solves a big problem in virtualized environments  in a scale-out and transparent manner while leveraging your existing storage investments.

If you want to be kept up to date, make sure to follow Satyam, Poojan , Charlie and PernixData on twitter. If you are interested in joining the PernixData FVP Beta, make sure to sign up!

Make sure to also read Frank’s article on PernixData.

<update>

I recommend watching the Storage Field Day videos for more details from Satyam Vaghani himself, note the playlist this is 4 videos!

</update>

VMware to acquire Virsto; Brief look at what they offer today

· ·

Most of you have seen the announcement around Virsto by now, for those who haven’t read this blog post: VMware to acquire Virsto. Virsto is a storage company which offers a virtual storage solution. I bumped in to Virsto various times in the past and around VMworld 2012 got reminded about them when Cormac Hogan wrote an excellent article about what they have to offer for VMware customers. (Credits go to Cormac for the detailed info in this post)  When visiting Virsto’s website there is one thing that stands out and that is “software defined storage”. Lets take a look at what Virsto offers and what software defined storage means to them.

Lets first start with the architecture. Virsto has developed an appliance and a host level service which together forms an abstraction layer for existing storage devices. In other words, storage devices are connected directly to the Virsto appliance and Virsto aggregates these devices in to a large storage pool. This pool is in its turn served up to your environment as an NFS datastore. Now I can hear you think, what is so special about this?

As Virsto has abstracted storage and raw device are connected to their appliance they control the on-disk format. What does this mean? Devices that are attached to the Virsto appliance are not formatted with VMFS. Rather Virsto has developed their own filesystem which is highly scalable and what makes this solution really interesting. This filesystem is what allows Virsto to offer specific data services, to increase performance and scale and reduce storage capacity consumption.

Lets start with performance, as Virsto sits in between your storage device and your host they can do certain things to your IO. Not only does Virsto increase read performance, but their product also increases write performance. Customers have experienced performance increases between 5x and 10x. For the exact technical details read Cormac’s article. For now let me say that they sequentialise IO in a smart way and de-stage writes to allow for a more contiguous IO flow to your storage device. As you can imagine, this also means that the IO utilisation of your storage device can and probably will go down.

From an efficiency perspective Virsto optimizes your storage capacity by provisioning every single virtual disk as a thin disk. However, this thin disk does not introduce the traditional performance overhead associated with thin disks preventing the waste of precious disk space just to avoid performance penalties. What about functionality like snapshotting and cloning, this must introduce overhead and slow things down is what I can hear you think… Again, Virsto has done an excellent job of reducing overhead and optimizing for scale and performance. Virsto allows for hundreds, if not thousands, of clones of a gold template without sacrificing performance while saving storage capacity. Not surprising Virsto is often used in Virtual Desktop and large Test and Development environments as it has proven to reduce the cost of storage with as much as 70%.

Personally I am excited about what Virsto has to offer and what they have managed to achieve in a relatively short time frame. The solution they have developed, and especially their data services framework promises a lot for the future. Hopefully I will have time on my hands soon to play with their product and provide you with more insights and experience.

vCloud Suite equals a Software Defined Datacenter

· ·

I was on the VMTN podcast this week with Frank Denneman and Rawlinson Rivera, hosted by John Troyer. One of the discussions we had was around the Software Defined Datacenter and the vCloud Suite. Often people make a direct connection between a Software Defined Datacenter and the vCloud Suite and I can understand why. I have heard some people comment that because some components are not fully integrated yet; the vCloud Suite does not allow you to build a full Software Defined Datacenter.

On the call I mentioned that a Software Defined Datacenter is not just about the vCloud Suite. Using the vCloud Suite does not magically provide you with a Software Defined Datacenter. I guess the same could be said for a cloud, using the vCloud Suite does not magically provide you with a cloud.

What a lot of people tend to forgot is that a cloud or an SDDC is not about the infrastructure or the individual components. (Lets from now on use SDDC instead of the full name or the word cloud) An SDDC is about how you are providing services to your customers. In this case, customers could be external / internal customers of course. An SDDC is about software defined services, about flexibility and agility. What does that mean? There are two points of view, the consumer of the platform and the platform administrator. Lets explain from both views what is means, or at least what I think it means…

  1. The consumer of the platform
    The consumer should be able to select a specific service level for their workload, or select a specific service for their workload. When they they select a service or service level the platform should sort things out for them fully automated, whether it is DR / Backup / Resources / Storage Tiering / Security… it should be automatically applied to the workload when either of those software defined service characteristics are selected and applied.
  2. The platform administrator
    The platform administrator should be able to define services and policies which can be consumed. These services or policies could be as simple as “enabling vSphere Replication” on a virtual machine, or as complex as deploying a 3 tier vApp including a full application stack and security services using vCloud Automation Center in combination with Application Director and vCloud Networking and Security.

In some cases that means you will need to deploy the full vCloud Suite and potentially more, in other cases it might mean you will deploy less but use 3rd party solutions to provide a fully automated solution stack and experience to your consumers . In the end it is about having the ability to define and offer services in a specific way and enabling your customers to consume these in a specific way.

Although the SDDC could be architected and build using the vCloud Suite, using the vCloud Suite does not automagically provide you with an SDDC. An SDDC is about your operating model and service offering, not about the components you are using.

Feel free to chip in,

VMware Storage APIs for VM and Application Granular Data Management

· ·

Last year at VMworld there was a lot of talk about VMware vStorage APIs for VM and Application Granular Data Management aka Virtual Volumes aka VVOL / VVOLs. The video of this session was just posted up on youtube and I am guessing people will have questions about it after watching it. What I wanted to do in this article is explain what VMware is trying to solve and how VMware intends to solve it. I tried to keep this article as close to the information provided during the session as possible. Note that this session was a Technology Preview, in no shape or form did VMware commit to ever delivering this solution let alone mention a timeline. Before we go any further… if you want to hear more and are attending VMworld, sign up for this session by Vijay Ramachandra and Tom Phelan!

INF-STO2223 – Tech Preview: vSphere Integration with Existing Storage Infrastructure

Background

The Storage Integration effort started with the vSphere API for Array Integration, also known as VAAI. VAAI was aimed to offload data operations to the array to reduce the load and overhead on the hypervisor but more importantly to allow for greater scale and better performance. In vSphere 5.0 the vSphere Storage APIs for Storage Awareness (aka VASA) were introduced which allowed for an out-of-band communications channel to discover storage characteristics. For those who are interested in VASA, I would like to recommend reading Cormac’s excellent article where he explains what it is and he shows how VMware partners have implemented it.

Although these APIs have bridged a huge gap they do not solve all of the problems customers are facing today.

What is VMware trying to solve?

In general VMware is trying to increase agility and flexibility of the VMware storage stack through providing a general framework where any current and future data operations can be implemented with minimal effort for both VMware and partners. Customers have asked for a solution which allows them to differentiate services to their customers on a per application level. Currently when provisioning LUNs, typically large LUNs, this is impossible.

Another area of improvement is granularity. For instance, it is desired to have per VM level fail-over or for instance to allow deduplication on a per VMDK level. This is currently impossible with VMFS. A VMFS volume is usually a LUN and data management happens at a LUN / Volume granularity. In other words a LUN is the level at which you operate from a storage perspective but this is shared by many VMDKs or VMs which might have different requirements.

As stated in mentioned in last years VMworld presentation the currently wish list is:

  1. Ability to offload to storage system on a per VMDK level
  2. Snapshots / cloning / replication / deduplication etc
  3. A framework where any current or future storage system operation can be leveraged
  4. No disruption to the existing VM creation workflows
  5. Highly scalable

These 4 should maximize your ROI on hardware investment, reduce operational effort associated with storage management and virtual machine deployment. It will also allow you to enforce application level SLAs by specifying policies on a VMDK or VM level instead of a datastore level. The granularity that it will allow for is in my opinion the most important part here!

How does VMware intend to solve it?

During the research phase many different options were looked at. Many of these however did not take full advantage of the capabilities of the storage system and they introduced more complexity around data layout. The best way of solving this problem is leveraging well known objects… volumes / LUNs.

These objects are referred to as VM Volumes, but also sometimes referred to as vVOLs. A VM Volume is a VMDK (or it derivative) stored natively inside a storage system. Each VM Volume will have a representative on the storage system. By creating a volume for each VMDK you can set policies on the lowest possible level. Not only that, the SAN vs NAS debate is over. This however does implies that when every VMDK is a storage object there could be thousands of VM Volumes. Will this require a complete redesign of storage systems to allow for this kind of scalability? Just think about the current 256 LUNs per host limit for instance. Will this limit the amount of VMs per host/cluster?

In order to solve this potential problem a new concept is introduced which is called an “IO De-multiplexer” or “IO Demux”. This is one single device which will exist on a storage system and it represents a logical I/O channel from the ESXi hosts to the entire storage system. Multi-pathing and path policies will be defined on a per IO Demux basis, which typically would be once. Behind this IO Demux device there could be 1000s of VM volumes.

This however introduces a new challenge. Where in the past the storage administrator was in control, now the VM administrator could possible create hundreds of large disks without discussing it with the storage admin. To solve this problem a new concept called Capacity Pools is introduced. A Capacity Pool is an allocation of physical storage space and a set of allowed services for any part of that storage space. Services could be replication, cloning, backup etc. This would be allowed until the set threshold is exceeded. It is the intention to allow Capacity Pools to span multiple storage systems and even across physical sites.

In order to allow to set specific QoS parameters another new concept is introduced called Profiles. Profiles are a set of QoS parameters (performance and data services) which apply to a VM Volume, or even a Capacity Pool. The storage administrator can create these profiles and assign these to the Capacity Pools which will allow the tenant of this pool to assign these policies to his VM Volumes.

As you can imagine this shifts responsibilities within the organization between teams, however it will allow for greater granularity, scale, flexibility and most importantly business agility.

Summarizing

Many customers have found it difficult to manage storage in virtualized environments. VMFS volumes typically contain dozens of virtual machines and VMDKs making differentiation on a per application level very difficult. VM Volumes will allow for more granular data management by leveraging the strength of a storage system, the volume manager. VM Volumes will simplify data and virtual infrastructure management by shifting responsibilities between teams and removing multiple layers of complexity.