Starting to get this question more often, should I always use vsAN ESA, or are there reasons to go for vSAN OSA? The answer is simple, whether you should use ESA or OSA can only be answered by the most commonly used phrase in consultancy: it depends. What does it depend on? Well, your requirements and your constraints.
One thing which comes up frequently is the 25Gbps requirement from a networking perspective. I’ve seen multiple people saying that they want to use ESA but their environment is not even close to saturating 10Gbps currently, so can they use ESA with 10Gbps? No, you cannot. ESA requires 25Gbps at minimum, and the bandwidth requirement fully depends on the ESA ReadyNode configuration you select. Why? Well, with ESA there’s also a certain performance expectation, which is why there’s a bandwidth requirement. The bandwidth requirement is put in place to ensure that you can use the NVMe devices to their full potential.
With both ESA and OSA you can produce impressive performance results, the big difference between ESA and OSA is the fact that ESA does this with a single type of NVMe device across the cluster, whereas OSA uses caching devices and capacity devices. ESA has also been optimized for high performance and is better at leveraging the existing host resources to achieve those higher numbers. An example of how ESA achieves that is through multi-threading for instance. What I appreciate about ESA the most is that the stack is also optimized in terms of resource usage. By moving data services to the top of the stack, data processing (compression of encryption for example) happens at the source instead of at bottom/destination. In other words, blocks are compressed by one host, and not by two or more (depending on the selected RAID level and type). Also, data is transferred over the network compressed, which saves bandwidth etc.
Back to OSA, why would you go for the Original Storage Architecture instead of ESA? Well, like I said, if you don’t have the performance requirements that dictate the use of ESA. If you want to use vSAN File Services (not supported with ESA in 8.o), HCI Mesh etc. If you want to run a hybrid configuration. If you want to use the vSAN Standard license. Plenty of reasons to still use OSA, so please don’t assume ESA is the only option. Use what you need to achieve your desired outcome, use what fits your budget, and use what will work with your constraints and requirements.