Yellow Bricks

by Duncan Epping

cloud

What’s new in 5.1?

· ·

As you might know the team I work for, VMware Technical Marketing, is responsible for a lot of the release collateral. Of course it was no different with the vSphere / vCloud 5.1 launch. Here are the links to all the excellent papers my colleagues wrote up.

Introducing SimpliVity, a new storage / compute platform

· ·

Around VMworld many new companies are “born”… well most of them have been around for a while, but fact is that they go public around VMworld. With SimpliVity it is no different. SimpliVity is a new storage company, although “storage” might be understating what they actually do.

I had an intro to SimpliVity last week, and yesterday SimpliVity publicly announced their product the OmniCube. The OmniCube is a 2U unit which combines Compute and Storage resources in a single node combining SATA and SSD drives.

Yes I can hear you thinking aahhhh another Nutanix-alike solution… well not really and I will explain why in the upcoming paragraphs. I guess the biggest difference from a physical perspective is that this is not a multi-node 2U unit like Nutanix is. This is a single node 2U unit, it can hold a max of 768GB of memory but comes with 128GB by default. It holds two 6 core 2.5Ghz intel procs and 2 x 10Gbe and 2 x 1Gbe. From a storage perspective it comes with 4 x 200GB SSD and 8 x 3TB SATA drives. With deduplication and compression, usable capacity is around 20TB. This was calculated using conservative estimates for deduplication (1.5:1) and compression (1.5:1). Depending on the use case this is more than likely a lot higher. Yes the OmniCube is a beast.

But SimpliVity / OmniCube is not about hardware, SimpliVity in my opinion is really about the solution. SimpliVity took a VM centric approach, or should we say VM aware (Not unlike Tintri). All operations / policies are on a per VM basis. So if you want to enable replication, this will be on a per VM basis. Not just replication but they have added a whole bunch of other cool features like:

  • Global management
  • Snapshots
  • Deduplication / Compression (global!)
  • Cloud Integration

The cool thing of course that these features can be combined. Having your snapshots deduplicated will have an impact on the amount of data stored. Only replicating deduplicated and compressed blocks will lower your bandwidth requirements… and that especially comes in handy when you are replicating / storing data in a cloud environment as the dedupe / compression is on a global basis.

Combine all of that with a tight VMware vSphere integration and I believe we have a very interesting solution on our hands. Now I know some of you are skeptical about these new companies popping up, and I can also be skeptical but a quick search on linkedin reveals where these guys are coming from… and with most of their engineers having a strong storage and virtualization background you know they will be on top of their game.

Simplivity has a booth at VMworld, for anyone interested in new compute/storage architectures definitely recommended to stop by, or register for their session:

SPO3287 – Everybody Wants to Rule the World. 5 Steps to Successfully Building and Ruling a Virtual Infrastructure Empire

Cormac also posted a nice article on this topic, make sure to read it!

CloudPhysics #VMworld challenge, win a Mac Pro/Air or Google Nexus

· ·

CloudPhysics just launched their website and with it a beta version of their product. As a great incentive to get people started with their product they came up with a contest where you can earn points by describing your virtualization problems. You can win some nice prices (retina Mac Pro, Google Nexus 7, Mac Air), so make sure to get started soon.

Your score is a calculation of the number of Cards you propose, plus a sum of the amount of activity you spend in the CloudPhysics portal. To achieve the best possible score, be sure to install a CloudPhysics Observer vApp, propose several new cards, vote on cards, and try out all the features of the portal. The greater the amount of activity and engagement, the higher your score can grow.

If you want to get started, go over to app.cloudphysics.com/login and login. Make sure to download the appliance! Now when you have downloaded it and got it up and running you should see data coming in soon. But where you really start collecting points is when you  start suggesting Cards. Look at the Card below, this is where you can start making suggestions, just hover over the lower right of the card to get in to the system…

Each of your suggestions will result in points. On top of that you also get points awarded for using their appliance  and commenting on suggestions from other users. A simple, but very cool challenge. The cool thing about this challenge is that these suggestions could  make it in to the product. In other words, if there is a problem you have faced many times and would like to prevent others from hitting the same thing… Report it and challenge the CloudPhysics people to make it part of their offering!

Current top-3 (after the first day):

  1. Jake Robinson
  2. William Lam
  3. Alan Renouf

Surely you can beat these 3 guys and collect your price at #VMworld. Go for it 🙂

Deploying Hadoop with Serengeti…

· ·

I read Richard McDougall’s blog post about Project Serengeti. Richard describes how you can deploy a Hadoop cluster in literally 10 minutes using Serengeti. I am not a Hadoop expert so I am probably the best qualified to test the 10 minute claim.

First of all, download the OVA. I would also suggest downloading the user guide, I needed it for the password / username to login to the Serengeti VM. (Which is: serengeti / password) So what do you need to deploy a Hadoop cluster in your vSphere environment? This is what I did:

  • Import the OVA
    • I decided to Provide a static IP instead of DHCP, I don’t have DHCP in my datacenter
  • Upgrade VMware Tools
    • Just right click the VM and upgrade the tools automatically, works as a charm. note that this is not a requirement!
  • Login to the console
    • ssh serengeti@10.27.51.21
    • username/password: serengeti/password
  • Go to the Serengeti CLI by typing
    • serengeti
  • I don’t run DHCP so in order for the Hadoop nodes to get an IP-address I will need to tell Hadoop which IPs to use. I had to remove the default network first and create a new one
    • network list
    • network delete –name defaultNetwork
    • network add –name defaultNetwork –portGroup “VM Network” –ip 10.27.51.165-200 –dns 10.27.51.122 –gateway 10.27.51.254 –mask 255.255.255.0
  • Create the Hadoop cluster by running the following commands
    • cluster create –name myHadoop
  • Now you will see a whole set of new virtual machines being created and your Hadoop cluster is created

How long did that take me? Indeed ~10 minutes… I tested if the cluster worked as follows, first SSH in to the Hadoop client node and then do the following:

  • cd /usr/local/share/pig-0.9.2/test/e2e/pig/lib/
  • hadoop jar hadoop-examples.jar teragen 1000000000 tera-data

Now you should see the worker nodes in vSphere ramping up to 100% CPU and memory utilization. It works for me… So why did I deploy it? Really to see if Richard was right and it would only take me 10 minutes? That wasn’t the reason of course. I wanted to see how it deployed and how it leveraged vSphere components. Note that this is not a GA release yet. It is version 0.5 and is probably still under heavy development. It is exciting to see though which direction we are heading in to and I am looking forward to the integration points with vSphere and vCloud Director (eventually).

One of the integration points that had my interest was the HA part that is mentioned in Richard’s post. Hortonworks was responsible for that and apparently they plug in to the vSphere HA VM and Application Monitoring. I haven’t been able to test that yet, but when I do I will update you on this. If you are at the point of testing this yourself, please note (and this is not well documented) you will have to enable “VM and Application Monitoring” explicitly as this is not enabled by default on vSphere HA cluster.

  • Right click your cluster
  • Click “Edit Settings”
  • Click the “VM Monitoring” tab and make sure “VM and Application Monitoring” is turned on.
  • Also check the settings for the individual VMs they will need to be set to “include” where applicable

That is it for now… again I am by no means a Hadoop expert and am not going to try to pretend, just exploring and broadening my horizon.

CloudPhysics, not so much stealth mode start-up anymore…

· ·

<disclaimer: I am a technical advisor for CloudPhysics>

cloud physics logoToday at the New England VMUG CloudPhysics has their first official “public appearance”. Yes some of you have heard the name a couple of times before and some of you might even know who the brains are behind this new start-up… for those who don’t let me give a brief introduction.

CloudPhysics was recently founded by John Blumenthal and Irfan Ahmad. Some of you might recognize their names as they used to work at VMware, John was a Product Manager for storage and Irfan was the person who was responsible for awesome features like Storage DRS and Storage IO Control. Together with several other brilliant people, including no one less than Carl “TPS / DRS” Waldspurger acting as an advisor and consultant, they founded a new company.

So what is CloudPhysics about? CloudPhysics is about big data, about centralized data, about analytics, about modeling data. CloudPhysics is essentially about helping you! How? Well let me try to explain that without revealing too much.

We’ve all monitored and managed environments, some of you are responsible for 3 hosts and some might be responsible for 80 hosts in different sites and in different companies. We all face several challenges and in many cases these are similar… How do you find common themes? How do you validate best practices are applied on all levels in your environment? How do you validate if your practices are actually used by others, and do you benefit from them? How do you know if you sized correctly? How do I solve specific problems? Would I benefit from a different storage platform or SSD? All of these are questions or problems you probably face daily and that is where CloudPhysics aims to come in to play.

CloudPhysics will enable you to find common best practices and problems in your environment. CloudPhysics will provide you guidance, this could be custom but also generic through for instance a link to a VMware KB article. They will enable you to compare and explore performance results. Find patterns in your environment… See trends and provide you with meaningful statistics about your environment. Sounds amazing right and probably something you wouldn’t mind testing today… The CloudPhysics product will come as a virtual appliance. The data gathered will go up to the cloud and all of the analysis will happen outside of your environment, of course with various degrees of anonymity.

CloudPhysics is constructing an analytics platform for vSphere for the application of collective intelligence to individual, local vSphere environments and users.  At the same time the platform is intended to service the needs of consulting companies, customers and the blogging community by providing APIs to enable unique exploration and discovery within the dynamic, changing dataset CloudPhysics continuously generates. Access to this dataset enables them to transform qualitative discussions into quantitative views of vSphere design and operation. CloudPhysics is not seeking to build a community; rather, it exists to empower the engineer and architect in all of us, particularly the commentators and critics essential to the industry.

For those who can’t wait, sign up at www.cloudphysics.com now for announcements and news on the beta. I am excited about CloudPhysics and I hope you all are as well.