• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Yellow Bricks

by Duncan Epping

  • Home
  • Unexplored Territory Podcast
  • HA Deepdive
  • ESXTOP
  • Stickers/Shirts
  • Privacy Policy
  • About
  • Show Search
Hide Search

Looking for the VROps VSAN Content Pack?

Duncan Epping · Jun 7, 2016 ·

I just noticed the link for the VROps VSAN Content Pack had changed and it isn’t easy to find through google either. Figured I would post it quickly so at least it is indexed and easier to find. Also including the LogInsight VSAN content pack link for your convenience:

  • LogInsight VSAN content pack: https://solutionexchange.vmware.com/store/products/vmware-vsan
  • VROps VSAN content pack: https://solutionexchange.vmware.com/store/products/vrealize-operations-management-pack-for-storage-devices

Kindle version of Essential Virtual SAN (6.2) available now!

Duncan Epping · Jun 3, 2016 ·

Just noticed the Kindle version of Essential Virtual SAN (6.2), second edition, is available now! We decided to go for an “e-book” first model to get it to market as quick as possible. I hope you will enjoy the book as much as Cormac and I did writing it. Pick it up!

Fully updated for the newest versions of VMware Virtual SAN, this guide show how to scale VMware’s fully distributed storage architecture to meet any enterprise storage requirement. World-class Virtual SAN experts Cormac Hogan and Duncan Epping thoroughly explain how Virtual SAN integrates into vSphere 6.x and enables the Software Defined Data Center (SDDC). You’ll learn how to take full advantage of Virtual SAN, and get up-to-the-minute insider guidance for architecture, implementation, and management.

If you want to order the paper version at a local book store, here are the ISBN details, or just go to Amazon and pre-order it of course.

  • ISBN-13: 978-0134511665
  • ISBN-10: 0134511662

I have memory pages swapped, can vSphere unswap them?

Duncan Epping · Jun 2, 2016 ·

“I have memory pages swapped out to disk, can vSphere swap them back in to memory again” is one of those questions that comes up occasionally. A while back I asked the engineering team why we don’t “swap in” pages when memory contention is lifted. There was no real good answer for it other than it was difficult to predict from a behavioural point of view. So I asked what about doing it manually? Unfortunately the answer was: well we will look in to it but it has no real priority it this point.

I was very surprised to receive an email this week from one of our support engineers, Valentin Bondzio, that you can actually do this in vSphere 6.0. Although not widely exposed, the feature is actually in there and typically (as it stands today) is used by VMware support when requested by a customer. Valentin was kind enough to provide me with this excellent write-up. Before you read it, do note that this feature was intended for VMware Support. While it is internally supported, you’d be using it at your own risk, and consider this write-up to be purely for educational purposes. Support for this feature, and exposure through the UI, may or may not change in the future.

By Valentin Bondzio

Did you ever receive an alarm due to a hanging or simply underperforming application or VM? If yes, was it ever due to prolonged hypervisor swap wait? That might be somewhat expected in an acute overcommit or limited VM / RP scenario but very often the actual contention happened days, weeks or even month ago. In those scenarios, you were just unlucky enough that the guest or application decided to touch a lot of the memory that happened to be swapped out around the same time. Which until this exact time you either didn’t notice or if you did, didn’t pose any visible threat. It just happened to be idle data that resided on disk instead of in memory.

The notable distinction being that it is on disk with every expectation of it being in memory, meaning a (hard) page fault will suspend the execution of the VM until that very page is read from disk and back in memory. If that happens to be a fairly large and contiguous range, even with gracious pre-fetching from the ESXi, you’ll might experience some sort of service unavailability.

How to prevent this from happening in scenarios where you actually have ample free memory and the cause of contention is long resolved? Up until today that answer would be to power cycle your VM or using vMotion with local swap store to asynchronously page in the swapped out data. For everyone that is running on ESXi 6.0 that answer just got a lot simpler.

Introducing unswap

As the name implies, it will page in memory that has been swapped out by the hypervisor, whether it was actual contention during an outage or just an ill-placed Virtual Machine or Resource Pool Limit. Let’s play through an example:

A VM experienced a non-specified event (hint, it was a 2GB limit) and now about 14GB of its 16GB of allocated memory are swapped out to the default swap location.

# memstats -r vm-stats -u mb -s name:memSize:max:consumed:swapped | sed -n '/  \+name/,/ \+Total/p'
           name    memSize        max   consumed    swapped
-----------------------------------------------------------
      vm.449922      16384       2000       2000      14146

[Read more…] about I have memory pages swapped, can vSphere unswap them?

VSAN everywhere with Computacenter

Duncan Epping · Jun 1, 2016 ·

This week I had the pleasure to have a chat with Marc Huppert (VCDX181). Marc works for Computacenter in Germany as a Senior Consultant and Category Leader VMware Solutions. He primarily focuses on datacenter technology. I noticed a tweet from Marc that he was working on a project where they will be implementing a 60 site ROBO deployment. It is one of those use cases that I don’t get to see too often in Europe so I figured I would drop him a note. We had a conversation of about an hour and below is the story around this project, and some of the other projects Marc has worked on.

Marc mentioned that he has been involved with VSAN for about 1.5 years now. They at first did intensive testing internally to see what VSAN could do for their customers and looked at the various use cases for their customer base. Quickly they discovered that when combining VSAN with Fusion-IO they ended up with a very powerful combination. Not only extremely reliable (Marc mentioned he has never seen a Fusion-IO card fail), but also an extremely well performing solution. They did comparisons between Fusion-IO and regular SATA connected SSDs and performance literally doubled, not just for reads but also writes was a big difference. One of the other reasons for considering PCIe based flash is to have the maximum number of disk slots available for the capacity tier. It all makes sense to me. Right now for current projects, NVMe based flash by Intel is being explored, and I am very curious to see what Marc’s experience is going to be like in terms of performance, reliability and the operational aspects compared to Fusion-IO.

Which brings us to the ROBO project, as this is the project where NVMe will be used surprisingly enough. Marc mentioned that this customer, a large company, has over 60 locations all connected to a central (main) datacenter. Each location will be equipped with 2 hosts. Depending on the size of the location and the number of VMs needed a particular VSAN configuration can be selected:

  • Small – 1 NVMe device + 5 disks, 128GB RAM
  • Medium – 2 NVMe devices + 10 disks, 256GB RAM
  • Large – 3 NVMe devices + 15 disks, 384GB RAM

Yes, that also leaves room to grow when desired, as every disk group can go up to 7 disks. From a Compute point of view the configurations do not differ too much besides memory config and disk capacity, actually the CPU is the same, to keep operations simple. In terms of licensing, the vSphere ROBO and VSAN ROBO edition are being leveraged, which provides a great scalable and affordable ROBO infrastructure, especially when coming from a two node configuration with a traditional storage system per location. Not just the price point, but primarily the day 2 management.

When demonstrating VSAN to their customer this is what impressed the customer the most. They have two people managing the entire virtual and physical estate, that is 60 locations (120 nodes) and the main datacenter which houses ~ 5000VMs and many physical machines as a result. You can imagine that they spend a lot of time in vCenter and they prefer to manage things end to end from that same spot, definitely don’t want be switching between different management interfaces. Today they manage many small storage systems for their ROBO locations, and they immediately realised that VSAN in a ROBO configuration would reduce the time they spend managing those locations significantly.

And that is just the first step, next up would be the DMZ. They have a separate compute cluster as it stands right now, but it unfortunately connects back to the same shared storage system as where there production is running. They do fully understand the risk, but never wanted to incur the large cost associated with a storage system dedicated for their DMZ, not just capex but also from an opex point of view. With VSAN the economics change, making a fully isolated and self-contained DMZ compute and storage cluster dead simple to justify, especially when combining it with NSX.

One awesome customer if you ask me, and I am hoping they will become a public VSAN reference at some point in the future as it is a great testimony to what VSAN can do. We briefly discussed other use cases Marc had seen out in the field and Horizon View, Management Clusters and production came up. Which is very similar to what I see. Marc also mentioned that there is a growing interest in all-flash, which is not surprising considering the dollar per GB cost of SAS is very close to flash these days.

Before we wrapped it up, I asked Marc if had any challenges with VSAN itself, what he felt was most complex. Marc mentioned that sizing was a critical aspect and that they have spend a lot of time in the past figuring out which configurations to offer to customers. Today the process they use is fairly straight forward: select Ready Node configuration, change SATA SSD with PCIe based Flash or NVMe, increase or decrease number of disks. Fully supported, yet still flexible enough to meet all the demands of his customers.

Thanks Marc for the great conversation, and looking forward to meeting up with you at VMworld. (PS: Marc has visited all VMworld events so far in both the US and EMEA, a proper old-timer you could say :-))

You can find him on Twitter or on his blog: http://www.vcdx181.com

How do I get to the next level – Part 2

Duncan Epping · May 26, 2016 ·

I was going over some of the posts of the last year which have had a large number of views and/or comments, just to figure out what interests you folks the most. One of the articles that stood out in terms of number of comments definitely was “How do I get to the next level“. Not just the comments, but I also recall the large number of tweets and feedback I received in person. I just read the article again, and there are a few things which I think I should have probably emphasized more. I figured I would write a follow up… Lets start with a quote from that article.

There is also that thing called “being at the right place, at the right time” and of course seeing opportunities, grabbing opportunities and taking risks.

This phrase was in the first paragraph of the article, and I think it captures a lot more of the essence of success than may seem obvious. A month or so ago I was having a conversation with someone and that person mentioned that in terms of my role within VMware I was lucky to be in the position I am in today. Although I agree that I am very fortunate to have found such a great role, and that there is always luck involved in some shape or form, luck is usually not the reason you are rewarded with a promotion or a new opportunity.

Being at the right place at the right time is one part of it, seeing and grabbing (risky) opportunities is another part. Although you need to be somewhat lucky to have picked the right place and time, one could also argue that it is a skill / talent (like football / hockey players knowing intuitively where the ball / puck will be). Also, don’t worry about failures. Everyone makes mistakes, some pick wrong focus areas, some pick the wrong role or even wrong company to work for, be open and stay alert and assess your position or situation. How is the world (around you) evolving? Is your path leading somewhere? Is this what you expected it to be? Use the information next to make decisions. Do you keep going? Do you need to adapt or even abort and go back to the drawing board?

If I have learned one thing, than it is definitely that I learn the most when I am out of my comfort zone. And if you want to set yourself up for success, you will need to make sure you are capable of adapting fast and learn quick. Learning quick typically happens outside of your comfort zone. Now don’t mistake being out of your comfort zone with being in your panic zone. It is okay to be anxious, to be nervous, to have that jittery feeling… but be aware that this does not turn in to anxiety. Most of you have been in that place at some point in time, you need to do something far far out of your comfort zone, for me that was designing and configuring a network environment (including routing) when I had been responsible for Windows/X86 servers for a couple of years. Way out of my depth for sure back then, which resulted in anxiety and ultimately a failed project. Which leads me to the next skill that will help you grow: Knowing what you know.

But maybe more importantly, knowing what you don’t know and admitting you don’t know. Many people know what they are good/great at, and many will also advertise that broadly. But it is more importantly to understand what your limits and weaknesses are. Knowing that will allow you to understand where you need to grow, and come up with a plan to improve skills / behaviour etc. Also, it will allow you to say “no” when you are out of depth, this is key if you want to set your self up for success, and if you want to grow.

For me personally the one thing that stood out as a weakness (or even lacking skill), when I looked at where I wanted to be and what it would take, was public speaking. I tried it, I failed, no need to pretend, I knew I had to get better at it to grow from a personal perspective, but also from a career standpoint. At a certain level within each organization, presenting is simply required. I was foolish enough to throw myself in at the deep end, my first public speaking experience was at one of the largest conferences (VMworld). That was definitely far outside of my comfort zone and straight in to the panic zone. I should have taken 3 steps back, in hindsight, and started with a small session as a small VMware User Group. Read my article on this, don’t make the same mistake I did… Anxiety doesn’t even come close to describing what I felt. Yes, I did take a step back, reconvened and decided I would never do this again… Well that was my first conclusion. Quickly after that I reached the conclusion that I went too fast, I needed to slow down and take smaller steps. I needed to start with something less nerve wrecking (but still right at the edge of learning and panic zone), which was presenting at a VMUG. Today, I spend most of my time talking to customers or at events, it is safe to say that I have overcome the anxiety. Also, I managed to get really close to where I wanted to be in 3-5 years as a result.

As mentioned in “part 1”, being able to share your thoughts/opinion can make a big difference for your career, whether that is in a meeting / a blog / public speaking is besides the point even. But also speaking up and fighting for something you (strongly) believe in can make a big difference (it did for me). Passion, enthusiasm, energy are all character traits I personally value high and is what I look for when I interview potential candidates. Willingness to take that extra step and go all-in when you believe in something, yet at the same time also know which battle to pick, know when to sit down and listen. That last part is probably the most important, you cannot win them all and you cannot even fight all battles, knowing when to back down is key and an important skill to master. (Is it a skill, or a character trait even?)

Last but not least, perseverance and focus. If you truly want something, focus, make a plan and get things done. Learn from your mistakes and adapt. Don’t be afraid to take a step back and take a different route, even if it will take longer to get where you want to be. As Winston Churchill once said, “Continuous effort, not strength or intelligence, is the key to unlocking our potential.”

  • « Go to Previous Page
  • Page 1
  • Interim pages omitted …
  • Page 110
  • Page 111
  • Page 112
  • Page 113
  • Page 114
  • Interim pages omitted …
  • Page 502
  • Go to Next Page »

Primary Sidebar

About the Author

Duncan Epping is a Chief Technologist and Distinguished Engineering Architect at Broadcom. Besides writing on Yellow-Bricks, Duncan is the co-author of the vSAN Deep Dive and the vSphere Clustering Deep Dive book series. Duncan is also the host of the Unexplored Territory Podcast.

Follow Us

  • X
  • Spotify
  • RSS Feed
  • LinkedIn

Recommended Book(s)

Also visit!

For the Dutch-speaking audience, make sure to visit RunNerd.nl to follow my running adventure, read shoe/gear/race reviews, and more!

Do you like Hardcore-Punk music? Follow my Spotify Playlist!

Do you like 80s music? I got you covered!

Copyright Yellow-Bricks.com © 2026 · Log in