Lately I have been thinking about the future of servers and more specifically the design around servers. Servers are more and more heading towards these massive beasts with all sorts of options that many might not need, but end up paying for as they are already bolted on. On the other hand you have these massive blade chassis that will allow for 10 / 14 blades, whatever your vendor decides is a nice form factor. While thinking about that I wondered why we have the 1U and 2U servers stuffed with options and the possibility to add disks when all we actually want, in many cases, is to run ESXi as a hypervisor. Even if we want to have local disks do we really need a 2U server?
After doing some research on the internet I bumped into something which I thought was a cool concept. Although it isn’t was I envisioned, it is close enough to share with you. I haven’t seen these types of servers used for virtualization so far and I wonder why that is. There are multiple vendors with offerings like these but I wanted to point out the following two as they offer more than others in my opinion and are VMware Certified. These servers are traditionally used in HPC environments (High-performance computing), but if you look at what they offer they could be suitable for virtualization as well. They are very dense but don’t bring along the requirement to buy a full chassis if you just need 3 or 4 servers. Of course you cannot directly compare them to blade servers and chassis, but think about the possibilities for a second and I will expand on that as well in a second.
Now in this case, the Super Micro 2U Twin2 has 4 nodes. Each node has a set of 6 SAS drives to its disposal and can hold up to 192GB or RAM. On top of that it can hold 2 Intel Nehalem/Westmere CPUs and has an Infiniband 20Gbps on board. This by itself is a very cool concept, but what if we would simplify it? These servers typically have:
- Expansion slots
- Sata / sas controllers
- Disks
- CD/DVD
- Multiple 1GbE links
- IPMI Lan port
But do we really need all of that? Wouldn’t a fully stripped down server make more sense for a virtualized environment? Do we really need a Sata/SAS controller? Do we need a CD/DVD Drive? Do we need multiple 1Gbe links plus 20GbE Infiniband and on top of that an IPMI Lan port? What if someone would come out with a server that wasn’t geared towards HPC but to virtualization. Yes we have seen many vendors taking their traditional servers and positioning them as Virtualization Ready but are they? So what would I like to see?
Well for starters I kinda like the form factor above, but I would like to see one without those disks. In most environments there will be shared storage available so there is no need for local disks. It would be nice if they had an on-board dual SD slot, allowing for ESXi to be installed locally. So what if someone could crank out, maybe someone already did if so let me know, a configuration like this:
- 2U “Chassis”
- Max 4 nodes
- Each node supporting max 2 sockets
- Each node supporting 192GB (probably overkill)
- Single 10GbE CNA
- Single IPMI LAN port
- SAS/SATA controllers
But what if we could go even more crazy like that, kinda like what Dell developed with their C5125 Microservers, what if you could host 12 Server nodes in 3u? Would that be something that you would be interested in? Yes, you might be limited to a single processor but without the requirement for a disk and lets say 96GB of memory max it should be possible. Yes I understand their would be implications to a design like that, but that is not the point right now.
I don’t design hardware or servers, but it seems to me that many options have been explored for all kinds of workloads but we haven’t reached the full potential for virtualization. Out in the field we see many people creating home labs with barebone casings, we see people running very stripped down configurations but when you walk into a random datacenter you see DL380’s, Dell R710s etc fully stocked with all bells and whistles while half of these features are not used. Wouldn’t dense and virtualization purpose built servers be nice? Seamicro created a nice solution with 512 servers in a 10 Us, but the CPUs are not powerful enough unfortunately for our purpose. Still I feel there are opportunities out their to really innovate, to lower the cost, lower the chances of failure and to ease management and maintenance!
Which server vendor out there is going to take the next step?
I saw some of these at VMworld either last year or the year before. I think at that time they were just trying to get certified for the HCL. I agree about the local storage issue. So much space wasted for bays that are never used. Even in the new C200s I just got which are supposedly next gen servers!
Duncan
I think IBM could easily do this as they have a similar blade already – IBM BladeCenter QS22. The HX5 model in the same series only takes 1.8″ disks so may fit your description.
I’ve always wondered when the market would start shifting towards the likes of PICMG based systems – fully diskless, network-boot systems in a PCI-card-size form factor. You could put a load of those into a 2U chassis very easily.
The 2U twin2 servers are great. You can get them without Infiniband, and install a low profile 10GbE card of your choice in the open PCI-E 2.0 x16 slot on each node. I can’t imagine it’ll be long before Supermicro releases a flavor with 10GbE onboard, they already have Intel Niantic 10GbE LOMs in their 1U superservers.
I wouldn’t consider the local storage useless…they provide great disk density (certainly better than blades). Enterprise SATA drives are cheap when you don’t need to pay up for the rebadged “insert server vendor here” drive. A number of workloads run just fine on local disk, some are in fact designed to – and you can pick the increased spindle count of 2.5″ drives (six per server), or capacity-for-price 3.5″ option (three per).
Just imagine if your disk scaled along with your compute, memory, and network whenever you added a unit of server capacity. On that note, it’d sure be nice if VMware added some basic features embracing local storage, e.g. not prevent maintenance mode when running VMs on local disk (configurable ‘shutdown/suspend OK’ flag per VM should be pretty easy, for starters).
P.S. no, I’m not in Supermicro sales 😀
The local storage currently in there can most definitely be used in the near future :). And yes I am already working on that but cannot spill details yet.
Hint: VMware have already released a design paper on something that would make use of it…. I wouldnt be suprised if it was very close to release by now….
http://labs.vmware.com/publications/lithium
I’m not sure it’s such a simple option to remove the unwanted components – it would most likely require a redesign of the architecture from the ground up, and that’s not cheap either. Manufacturers aren’t going to be keen to jump onto a “single use” architecture when they can sell current designs as both traditional servers and virtual hosts. I agree that it would be great to have streamlined, all grunt virtual host servers like this, but the opportunity cost for manufacturers might be too great.
@Milson – Local storage definitely isn’t useless, but if the requirement is for local storage then it’s already well catered for with existing server designs!
I would say if they can invest in HPC they surely can invest in virtualization! Different type of workload different needs in my opinion. But than again I am just rambling and randomly spitting out an idea, which doesn’t mean it is feasible for vendors automatically.
With these servers you have some shared disk avail no? Why limit youself to one os in chassis? I see two boot from usb esxi hosts, and two nexenta nodes in an HA configuration. A really tiny HA vmware environment in one box.
Why not install ESXi on all 4 hosts and run Nexenta in a VM on each, re-exposing its local disk? You could have 2 HA configurations and actually make good use of the 12 cores worth of CPU in each node.
For bonus hotness, load 5 of the 6 2.5″ bays with 1T drives and the 6th with an SSD an use it for a L2ARC device under Nexenta.
Or if you don’t like spending money for Nexenta licenses, you could consider Linux + DRBD instead. 🙂
Hi Duncan – Interesting article and a subject I keep coming back to as a point of personal interest.
The article below was in The Register a few days back, makes for good reading and with some tweeks could be ideal.
http://www.theregister.co.uk/2011/04/22/virtual_machine_company_appliances/
Cheers,
Perry.
I’ve read the article, it’s nice but too much in my opinion.
This idea has in fact crossed my mind several times although in a slightly different way: in my line of work I usually only see small scale virtualization projects, often limited to 3 hosts.
HP has launched several models over the last few months that could have potential if modified for use as a small VMware cluster. For instance with their new E5500 Exchange cluster you could have a very potent 2-node VMware cluster and plenty storage (35 SAS drives) in one box and their DL1000 has the potential of up to four nodes at the cost of decent on-board storage.
I think only to often the SMB is forgotten when it comes to hardware design.
Those HP 4node systems are nice indeed 🙂
HP now has a DL2000 model which is even better that fits 4 nodes in 2U of space.
There’s also an HP blade called the 2×220 which can fit two systems in a single half-height blade slot, giving you 32 servers in 10U! But the problem with that one is it’s starved for IO – you get a single QDR infiniband/10GbE connection and 2x1GbE, so unless you have infiniband you’re limited to 12Gb/sec of ethernet thruput total, no FC option.
Plus that you will need to pay up front for a possible 32 slots. Considering the size of the hosts these days many will not hit that level. You can easily fit 150VMs on 4 nodes.
HP does have a smaller 8 slot enclosure, so if you know you won’t be needing a full 16 slots you can go that route.
But even then for smaller businesses that is too much 🙂
Some of the principles (stripping down servers to bare minimum requirement and removing unnecessary items) are being followed by the opencompute initiative (use your favourite search engine to find out more) – however taking it to the extreme and removing disk controllers off the motherboard and having a really stripped down cpu/memory/network and nothing else may be a bit extreme
to get mainstream adoption – there will always be someone who wants a sata port here/ an usb port there and soon there is a “basic” requirement that becomes default across the range
many vendors these days will sell a barebones config with no disks / peripherals – hp even do skinless or their SL series hardware where they cram as many servers into a chassis as possible with some limitations on power redundancy etc
Also – how much money would someone really save not putting a simple sata controller in? most vendoras provide a very cut down version onboard and controller upgrades as extras – same goes with networks
personally – i would like to see more /faster on-board ports than provided today (usually 2 x 1gig elcheapo chipset)
my 2pence worth!
That is right, however when we are talking about formfactor and cramming more in less space you will need to remove components at some point. for instance your sata / sas controller and connectors could also provide space for additional memory slots :). But I agree there is a cost associated with it, I am glad I am not responsible for executing or even calculating costs etc… just spitting ideas and see what other people feel/think about this.
Make it dual CNA’s and I think you’re on the right track, though a lot of place’s aren’t ready for or willing to go converged infrastructure yet so the market might be a bit limited.
Why dual CNA? Many UCS deployments also only have a single CNA….
I’d never trust my uptime to a single path, especially not for storage =)
Why CNA?
IB is lower cost, higher bandwidth, decreased latency, and a more reliable fabric that doesn’t require the limitations of FCoE.
I would propose a 2U box with ~24 2.5″ SAS drives, dual 8-10 core processors, 256GB of RAM, and 2 HCAs.
Back this with and IB fabric and you have a top tier virtualization platform. This is somewhat how Exadata goes except the storage nodes are seperated. But it does ride and IB backbone.
The day is coming when you simply acquire nodes for scale and have a virtual grid.
On the end of this chain will be minimal hops – core switch, security devices (IPS, FW, etc.), router. The entire datacenter will be virtualized – compute, storage, and network.
Thanks for the tip on opencompute, looks sweet!
Actually, I think most of these comments are behind the curve.
The next target will be virtualization grids. In these grids there is nothing centralized. Let me paint the picture.
Firt you standardize on something like Dell’s C6145 with 3+TB of storage 2-4 sockets and 512GB of RAM. You purchase a datacenter fabric utilizing Infiniband (like Xsigo). Now, you scale your compute and storage one node at a time. Every node contains a piece of your storage array.
With this grid model, all costs are incremental. No central storage device sucking up capital. Tons of commodity hardware to provide distrubed workloads.
Now because we have backed these nodes with Infiniband, we can live migrate VMs between nodes (both storage and compute) – full vMotion if you will. Or, we can replicate bits between nodes (ala Lefthand).
Now this model works for any size workload. You can start with 2 nodes and scale to whatever limit.
for customers who haven’t already invested heavily in shared storage that will soon become reality indeed.
do you know something we don’t? 🙂 or you just see the industry moving in this direction?
I’m part of Tech Marketing mainly working on Storage and interacting with the engineers directly… you bet I know more 😀
Dell C5125 is a good start, but Dell is stupid w/ their marketing/sales of it. They won’t sell them to us unless we buy them by the rack(in huge quantities)…At least that’s what our sales rep told me :/
Interestingly, a couple of [cough] overpriced blocks have been recently put on hold…
How about totally de-centralised virtualization, no dedicated “servers” at all 🙂
http://vinf.net/2009/02/24/vmware-client-hypervisor-cvp-grid-application-thoughts/
and a bit of this..
http://vinf.net/2009/08/07/vmware-esx-5/
It’s the shared storage component that is biting me. I’d like to be able to offer HA for our smaller remote offices – if a host fails or we want to patch it, we can continue running. We don’t need, and can’t afford, to duplicate the storage which could be on the order of 8TB or so. I haven’t found a small “cluster in a box” type of solution yet. THe HP DL2000 looks interesting but it looks like the storage is dedicated to each node in the box rather than shared.
What’s everybody else doing for the smaller sites to make the manageable yet affordable?