I have been reading many articles over the last weeks on Software Defined Storage and wrote an article on this topic a couple of weeks ago. While reading up one thing that stood out was that every single storage/flash vendor out there has jumped on the bandwagon and (ab)uses this term where ever and when ever possible. In most of those cases however the term isn’t backed by SDS enabling technology or even a strategy, but lets not get in to the finger pointing contest as I think my friends who work for storage vendors are more effective at that.
The article which triggered me to write this article was released a week and a half a go by CRN. The article was a good read, so don’t expect me to tear it down. The article just had me thinking about various things, and what better way to clear your head then to write an article about it. Lets start with the following quote:
While startups and smaller software-focused vendors are quick to define software-defined storage as a way to replace legacy storage hardware with commodity servers, disk drives and flash storage, large storage vendors are not giving ground in terms of the value their hardware offers as storage functionality moves toward the software layer.
Let me also pull out this comment by Keith Norbie in the same article, as I think Keith hit the nail on the head:
Norbie said to think of the software-defined data center as a Logitech Harmony remote which, when used with a home theater system, controls everything with the press of a button.
If you take a look at how Keith’s quote relates to Software Defined Storage it would mean that you should be able to define EVERYTHING via software. Just like you can simply program the Logitech Harmony remote to work with all your devices; you should be able to configure your platform in such a way that spinning up new storage objects can be done by the touch of one button! Now getting back to the first quote, whether functionality moves out of a storage system to a management tool or even to the platform is irrelevant if you ask me. If your storage system has an API and it is allows you to do everything programmatically you are half way there.
I understand that many of the startups like to make potential customers believe different, but the opportunity is there for everyone if you ask me. Yes that includes old-timers like EMC / NetApp / IBM (etc) and their “legacy” arrays. (As some of the startups like to label them.) Again, don’t get me wrong… playing in the SDS space will require significant changes to most storage platforms as most were never architected for this usecase. Most are currently not capable of creating thousands of new objects programmatically. Many don’t even have a public API.
However, what is missing today is not just a public API on most storage systems, it is also the platform which doesn’t allow you to efficiently manage these storage systems through those APIs. When I say platform I refer to vSphere, but I guess the same applies to Hyper-V, KVM, Xen etc. Although various sub-components are already there like the vSphere APIs for Array Integration (VAAI) and the vSphere APIs for Storage Awareness (VASA), there are also still a lot capabilities missing. A good example would be defining and setting specific data-services on a virtual disk level granularity, or end to end Quality of Service for virtual disks or virtual machines, or automatic instantiation of storage objects during virtual machine provisioning without manual action required from your storage admin. Of course, all of this from a single management console…
If you look at VMware vSphere and what is being worked on in the future you know those capabilities will come at some point, in this case I am referring to what was previewed at VMworld as “virtual volumes” (sometime also referred to as VVOLs), but this will take time… Yes I know some storage vendors already offer some of this granularity (primarily the startups out there), but can you define/set this from your favorite virtual infrastructure management solution during the provisioning of a new workload? Or do you need to use various tools to get the job done? If you can define QoS on a per VM basis, is this end-to-end? What about availability / disaster recovery, do they offer a full solution for that? If so, is it possible to simply integrate this with other solutions like for instance Site Recovery Manager?
I think exciting times are ahead of us; but lets all be realistic… they are ahead of us. There is no “Logitech Harmony” experience yet, but I am sure we will get there in the (near) future.
Good stuff D.
Good read Duncan – I think there is one more aspect that lots of startup/emerging storage vendors often overlook – supportability. Does the solution have a built-in framework/architecture to monitor system/process health, and alert the end user when h/w, s/w components fail, or when performance has degraded? Let’s hope all storage vendors (legacy or emerging) continue to focus on RAS (Reliability, Availability, Serviceability), and not just tout crazy IOPS/super low latency to make a sale 🙂
Good point Wen. Fully agree that this is often overlooked. It seems that there is an IOps war going on, but RAS and also recover-ability is often neglected unfortunately.
Hi
Great read – just a question – is SMI-S supposed to be the public standard for storage management?
Wen Yu – agree with your point about headline marketing about IOPS etc against RAS and also integration.
Phil
I like the idea of controlling everything with the click of a button, as per the Logitech remote analogy. However, how many datacenters are really prepared to deploy such a solution?
-Greg
There is no doubt that SDS is a critical component of an overall SDDC strategy. Certainly as critical as SDN. Part of the challenge, I think, goes beyond tooling and enablement. Most of the legacy technologies don’t fit well in a SDS box; they’re too complex and inflexible. When you consider the promise of SDDC, the idea is to unlock agility through the ability to programmatically repartition resources. So routers that can freely move IPs around, compute management intelligence that can orchestrate host capacity as a global pool, etc.
Storage is where the rubber meets the road though and presents a few fundamental problems. It’s where the workloads ultimately actually *live*, its the most sensitive to performance shortfalls that the user will see, and it is the least flexible. It’s also the resource domain most prone to component failure and the most complex to scale out.
Looking at how large public cloud IaaS platforms have solved for this space is interesting because you definitely see a recurring theme of commodity white box clustered storage foundations with a really intelligent control plane providing abstraction and network attach as a rule.
It’s hard to imagine what a SDS FC SAN would really look like. It’s just a fundamentally difficult thing to grow so almost certainly any SDS enhancement to it would be pretty limited to basic provisioning and operations automation.
Ideally attached to smart APis on the host should be APIs on the front end of super extensible storage systems (clustered, network attached, distributed filers)
In other words with SDS I think the foundation really matters. For EMC, for example, I can see a build up around Isilon, or even better VSAs, then just abstracting the control plane out from traditional arrays.
Great article as usual Duncan. Glad to see some bloggers take the high road as well on the ever tempting competition bashing. Nice work.
I absolutely love the Harmony remote analogy. Possibly because I am a sucker and have three of them since each room has different entertainment toys. This has me thinking… Multiple harmony remotes is analogous to today’s search for true software defined nirvana. Granted, the big vendors have an SDDC strategy however, in many cases it requires you only use their toolsets or platforms. Which leads you to more than one “all-in-one remote” or software defined nothing. The harder thing to do, is actually build software for the purpose of integration. Take the vVols example… Vendors ranging from EMC, NetApp, IBM, etc… were all engaged through the process – resulting in true software defined storage.
Just my random thoughts…
Hey Duncan – great perspective on a noisy topic. Despite being one of those all-flash vendors you referred to in the post, we take a very similar view of SDS and wrote about it here. http://www.wired.com/insights/2012/11/setting-the-record-straight-on-software-defined-storage/
Any storage system vendor, emerging or legacy, promoting their product as SDS is only doing so to attach themselves to the marketing hype. Storage systems deliver capabilities and functionality in support of SDS but by themselves are not SDS. It is the control plane (e.g. VVOLs) in conjunction with the underlying storage hardware that will come together via API-level integration to support SDS.
The article is spot on, thanks for you comment! Looks like I will have to spent some time looking in to SolidFire soon 🙂
Nice Post