I received a bunch of questions based on my vSAN File Services posts the past couple of days. Most questions were around how the different layers talk to each other, and where vSAN comes in to play in this platform. I understand why, I haven’t discussed this aspect yet, but that is primarily as I wasn’t sure what I could/should talk about. Let’s start with a description of how communication works, top to bottom.
- The NFS Client connects to the vSAN File Services NFS Server
- The NFS Server runs within the protocol stack container, the IPs provided during the configuration are assigned to the protocol stack container
- The protocol stack container runs within FS VM, the FS VM has no IP address assigned
- The FS VM has a VMCI device (vSocket interface), which is used to communicate with the ESXi host securely
- The ESXi host has VDFS kernel modules
- VDFS communicates with vSAN layer and SPBM
- vSAN is responsible for the lifecycle management of objects
- A file share has a 1:1 relationship with a VDFS volume and is formed out of vSAN objects
- Each file share / VDFS volume has a policy assigned, and the layout of the vSAN objects are determined by this policy
- Objects are formatted with the VDFS file system and presented as a single VDFS volume
I guess a visual may help clarify things a bit, as for me it also took a while to wrap my head around this. Look at the diagram below.
So in other words, every FS VM allows for communication to the kernel using the vSockets library through the VMCI device. I am not going to explain what vSocket is as the previous link refers to a lengthy document on this topic. The VDFS layer leverages vSAN and SPBM for the lifecycle management of the objects that form a file share. So what is this VDFS layer then? Well VDFS is the layer that exposes a (distributed) file system that resides within the vSAN object(s) and allows the protocol stack container to share it as NFS v3 or v4.1. As mentioned, the objects are presented as a single VDFS volume.
So even though vSAN File Services uses a VM to ultimately allow a client to connect to a share, the important part here is that the VM is only used for the protocol stack container. All of the distributed file system logic lives within the vSphere layer. I hope that helps to explain the architecture a bit and how the layers communicate. I also recorded a quick demo, including the diagram above with the explanation of the layers, that shows how a protocol stack container is moved from one FS VM to another when a host goes into maintenance mode. This allows for NFS clients to stay connected to the same IP-address for the file shares for NFS v3, for NFS v4.1 we do provide the ability to connect to a primary IP address and load balance automatically.