Yellow Bricks

by Duncan Epping

Storage

Can I still provision VMs when a vSAN Stretched Cluster site has failed? Part II

· · Leave a Comment

3 years ago I wrote the following post: Can I still provision VMs when a VSAN Stretched Cluster site has failed? Last week I received a question on this subject, and although officially I am not supposed to work on vSAN in the upcoming three months I figured I could test this in the evening easily within 30 minutes. The question was simple, in my blog I described the failure of the Witness Host, what if a single host fails in one of the two “data” fault domains? What if I want to create a snapshot for instance, will this still work?

So here’s what I tested:

  • vSAN Stretched Cluster
  • 4+4+1 configuration
    • Meaning, 4 hosts in each “data site” and a witness host, for a total of 8 hosts in my vSAN cluster
  • Create a VM with cross-site protection and RAID-5 within the location

So I first failed a host in one of the two data sites. When I fail the host, the following is what happens when I create a VM with RAID-1 across sites and RAID-5 within a site:

  • Without “Force Provisioning” enabled the creation of the VM fails
  • When “Force Provisioning” is enabled the creation of the VM succeeds, the VM is created with a RAID-0 within 1 location

Okay, so this sounds similar to the originally described scenario, in my 2016 blog post, where I failed the witness. vSAN will create a RAID-0 configuration for the VM. When the host returns for duty the RAID-1 across locations and RAID-5 within each location is then automatically created. On top of that, you can snapshot VMs in this scenario, the snapshots will also be created as RAID-0. One thing to mind is that I would recommend removing “force provisioning” from the policy after the failure has been resolved! Below is a screenshot of the component layout of the scenario by the way.

I also retried the witness host down scenario, and in that case, you do not need to use the “force provisioning” option. One more thing to note. The above will only happen when you create a RAID configuration which is impossible to create as a result of the failure. If 1 host fails in a 4+4+1 stretched cluster you would like to create a RAID-1 across sites and a RAID-1 within sites then the VM would be created with the requested RAID configuration, which is demonstrated in the screenshot below.

Part 2: Is VSA the future of Software Defined Storage? (Customer use case)

· · Leave a Comment

About 6.5 years ago I wrote this blog post around the future of Software-Defined Storage and if the VSA (virtual storage appliance) is the future for it. Last week at VMworld a customer reminded me of this article. Not because they read the article and pointed me back at it, but because they implemented what I described in this post, almost to the letter.

This customer had an interesting implementation, which kind of resembles the diagram I added to the blog post, note I added a part to the diagram which I originally left out but had mentioned in the blog (yes that is why the diagram looks like it is ancient… it is):

I want to share with you what the customer is doing because there are still plenty of customers that do not realize that this is supported. Note that this is supported by both vSAN as well as VMware Cloud Foundation, providing you a future proof, scalable, and flexible full-stack HCI architecture which does not need to be implemented in a rip and replace approach!

This customer basically leverages almost all functionality of our Software-Defined Storage offering. They have vSAN with locally attached storage devices (all NVMe) for certain workloads. They have storage arrays with vVols enabled for particular workloads. They have a VAIO Filter Driver which they use for replication. They also heavily rely on our APIs for monitoring and reporting, and as you can imagine they are a big believer in Policy-Based Management, as that is what helps them with placing workloads on a particular type of storage.

Now you may ask yourself, why on earth would they have vSAN and vVols sitting next to each other? Well, they had a significant investment in storage already, the storage solution was fully vVols capable and when they started using vSAN for certain projects they simply fell in love with Storage Policy-Based Management and decided to get it enabled for their storage systems as well. Even though the plan is to go all-in on vSAN over time, the interesting part here, in my opinion, is the “openness” of the platform. Want to go all-in on vSAN? Go ahead! Want to have traditional storage next to HCI? Go ahead! Want to use software-based data services? Go ahead! You can mix and match, and it is fully supported.

Anyway, just wanted to share that bit, and figured it would also be fun to bring up this 6.5 years old article again. One more thing, I think it is also good to realize how long these transitions tend to take. If you would have asked me in 2013 when we would see customers using this approach my guess would have been 2-3 years. Almost 6.5 years later we are starting to see this being seriously looked at. Of course, platforms have to mature, but also customers have to get comfortable with the idea. Change simply takes a lot of time.

Project nanoEDGE aka tiny vSphere/vSAN supported configurations!

· · 10 Comments

A few weeks ago VMware announced Project nanoEDGE on their blog virtual blocks. I had a whole bunch of questions the following days from customers and partners interested in understanding what it is and what it does. I personally prefer to call project nanoEDGE “a recipe”. In the recipe, it states which configuration would be supported for both vSAN as well as vSphere. Lets be clear, this is not a tiny version of VxRail or VMware Cloud Foundation, this is a hardware recipe that should help customers to deploy tiny supported configurations to thousands of locations around the world.

Project nanoEDGE is a project by VMware principal system engineer Simon Richardson. The funny thing is that right around the time Simon started discussing this with customers to see if there would be interest in something like this, I had similar discussions within the vSAN organization. When Simon mentioned he was going to work on this project with support from the VMware OCTO organization I was thrilled. I personally believe there’s a huge market for this. I have had dozens of conversations over the years with customers who have 1000s of locations and are currently running single-node solutions. Many of those customers need to deliver new IT services to these locations and the requirements for those services have changed as well in terms of availability, which makes it a perfect play for vSAN and vSphere (with HA).

So first of all, what would nanoEDGE look like?

As you can see, these are tiny “desktop alike” boxes. These boxes are the Supermicro E300-9D and they come in various flavors. The recipe currently explains the solution as 2 full vSAN servers and 1 host which is used for the vSAN Witness for the 2 node configuration. Of course, you could also run the witness remotely, or even throw in a switch and go with a 3 node configuration. The important part here is that all used components are on both the vSphere as well as the vSAN compatibility guide! The benefit of using the 2-node approach is the fact that you can use cross-over cables between the vSAN hosts and avoid the cost of a 10GbE Switch as a result! So what is in the box? The bill of materials is currently as follows:

  • 3x Supermicro E300-9D-8CN8TP
    • The box comes with 4x 1GbE NIC Port and 2x 10GbE NIC Port
    • 10GbE can be used for direct connect
    • It has an Intel® Xeon® processor D-2146NT – 8 cores
  • 6 x 64GB RAM
  • 3 x PCIe Riser Card (RSC-RR1U-E8)
  • 3 x PCIe M.2 NVMe Add on Card (AOC-SLG3-2M2)
  • 3x Capacity Tier – Intel M.2 NVMe P4511 1TB
  • 3x Cache Tier – Intel M.2 NVMe P4801 375GB
  • 3x Supermicro SATADOM 64GB
  • 1 x Managed 1GbE Switch

From a software point of view the paper lists they tested with 6.7 U2, but of course, if the hardware is on the VCG for 6.7 U3 than it will also be supported to run that configuration. Of course, the team also did some performance tests, and they showed some pretty compelling numbers (40.000+ read IOPS and close to 20.000 write IOPS), especially when you consider that these types of configurations would usually run 15-20 VMs in total. One thing I do want to add, the bill of materials lists M.2 form factor flash devices, this allows nanoEdge to avoid the use of the internal unsupported AHCI disk controller, this is key in the hardware configuration! Do note, that in order to fit two M.2 devices in this tiny box, you will need to also order the listed PCIe Riser Card and the M.2 NVMe add on card, William Lam has a nice article on this subject by the way.

There are many other options on the vSAN HCL for both caching as well as capacity, so if you prefer to use a different device, make sure it is listed here.

I would recommend reading the paper, and if you have an interest in this solution please reach out to your local VMware representative for more detail/help.

2 node direct connect vSAN and error “vSphere HA agent on this host could not reach isolation address”

· · Leave a Comment

I’ve had this question over a dozen times now, so I figured I would add a quick pointer to my blog. What is causing the error “vSphere HA agent on this host could not reach isolation address” to pop up on a 2-node direct connect vSAN cluster? The answer is simple, when you have vSAN enabled HA uses the vSAN network for communication. When you have a 2-node Direct Connect the vSAN network is not connected to a switch and there are no other reachable IP addresses other than the IP addresses of the vSAN VMkernel interfaces.

When HA tries to test if the isolation address is reachable (the default gateway of the management interface) the ping will fail as a result. How you can solve this is simply by disabling the isolation response as described in this post here.

My recommended sessions for VMworld Europe 2019!

· · 2 Comments

I created a list of recommended sessions for VMworld US, so I figured I would do the same for VMworld Europe. I am limiting it to 15 sessions for Europe and removed some of the sessions I had listed for the US and added some others for EMEA. I personally have 2 sessions scheduled at the moment, and they filled up rather fast in the US, so make sure to register early: HCI1870BE and HBI2186BE.

Okay, here’s my top 15 list, please note that this is pretty much in random order:

  • 60 Minutes of Non-Uniform Memory Architecture [HBI2278BE] by Frank Denneman
    This session got raving reviews in the US, I attended it personally and I can highly recommend it. That is if you think your brain can handle it… it is deep!
  • PowerCLI Deep Dive [HBI1729BE] by Luc Dekens and Kyle Ruddy
    What can I say? These guys speak PowerCLI. They know it inside out, just make sure you are ready to go deep!
  • VMware Cloud Foundation Deep Dive [HBI2044BE] by Jason Shaw
    I am a big fan of full-stack HCI solutions, and in this session, Jason talks about what VMware Cloud Foundation brings to the table.
  • HCI Management: Current and Future [HCI1207BE] by JunChi Zhang and Christian Dickmann
    Every year these guys bring some really cool demos and they expect you to provide feedback at the end of the session. So if you want to have a chance to influence vSAN/vSphere, attend this one!
  • Project Pacific Technical Overview: Unifying vSphere and Kubernetes [HBI4500BE] by Michael West and Jared Rosoff
    There was a lot of buzz surrounding the announcement of Project Pacific. I managed to sit in one or two sessions in the US, and knowing Jared was the lead engineering on this project I suspect that this session may even go deeper than the ones I attended in the US.
  • The Virtually Speaking Podcast Live: The Future of Storage [HCI1894PE] by Pete Flecha, John Nicholson, and guests
    I was a guest on the US in this session, it was a lot of fun. A lively discussion, and great questions from the audience on top of that. Plus, it is amazing to see Pete Flecha do the intro and outro live, I always thought that was pre-recorded!
  • The Cloud Backbone Network: A Paradigm Shift in Corporate WAN [OCTO1911BE] by Israel Cidon
    I have seen a session on this topic by Israel at an internal conference and it was nothing short of mind-blowing. I would highly recommend attending this if you want to learn more about how we could potentially improve corporate WANs in the future by leveraging SD-WAN technology. Mind, this session is foreward looking!
  • Showcase Keynote: HCI – The Foundation for your Future-proof Infrastructure [HCI3551KE] by Cormac Hogan and John Gilmartin
    I presented this session in the US with John, Cormac will be presenting it in Europe with John. I am sure it will be filled with great demos once again, and I will definitely try to attend this one in person
  • Leveraging the Latest Server Technologies in vSphere [HBI2362BE] by Niels Hagoort
    This session wasn’t on my list for the US, but after watching the recording I feel it is worth attending in person! A good overview of what we are doing with the latest HW technologist in vSphere.
  • vSphere Virtual Volumes: Technical Deep Dive [HBI2853BE] by Jason Massae and Thiruvengada Govinda Thirumal
    The adoption of vVols is growing fast, and it is for a good reason. Jason and Thiruvengada will explain in-depth what vVols is and how it works, and of course what the benefits are of adopting it.
  • Tech Preview of Site Recovery Manager with Virtual Volumes [HCI2894BE] by Stefan Tsonev and Velina Krasteva
    The tech preview of SRM for vVols has been discussed various times, but in this session by Stefan and Velina it is also demonstrated. So if you are interested in disaster recovery solutions in combination with vVols then this is one to attend!
  • Meet the [ML] Driver of the Self-Driving Datacenter [MLA1904BE] by Jad El-Zein and Arun Annavarapu
    At VMworld US Project Magna was unveiled, VMware’s effort to deliver the self-driving datacenter. A very interesting concept, which is explained in this session by Jad and Arun. I’ve seen some of the work, and it has a lot of potential, highly recommend attending this session!
  • Technical Deep Dive on Cloud Native Storage 1.0 [HCI2763BE] by Cormac Hogan and Myles Gray
    Cormac and Myles are the Cloud Native experts in our business unit, and I can guarantee that this will be a great session filled with cool demos.
  • Optimizing vSAN for Performance [HCI1757BE] by Paudie ORiordan
    If anyone can explain how to optimize vSAN, how to do benchmarking, and what to avoid… it is Paudie. On top of that, he is a great speaker and you can basically ask him anything on the topic of storage. Must see!
  • Innovations in vMotion: Features, Performance, and Best Practices [HBI1421BE] by Sreekanth Setty and Arunachalam Ramanathan
    I watched this session in the US, it provides an excellent overview of the enhancements introduced for vMotion to improve vMotion (and switch over) times. They also talk about some enhancements coming in the near future. Very interesting stuff.

Scheduling opens up tomorrow (24th of September), so make sure to hit the portal early. I guarantee that many of the sessions above will be full in a matter of days. So don’t say I did not warn you in advance!