Last week we have been testing with vGPUs in our lab for our Virtual Reality applications. We also had the need to do some rendering. The rendering that we needed to do required a lot of GPU power so we figured we would test the rendering within a virtual machine using a vGPU. This by itself worked fairly well with a single vGPU with a grid_rtx8000p-12q vGPU profile. The next thing we wanted to test was having multiple vGPUs associated with a single Virtual Machine. We have vSphere 6.7 U3 in our lab, so that should be fully supported. The first thing you will need to do is make sure your VMs are VM compatibility version 15. Without 15 you can’t even add a second vGPU, to begin with.
We added the two vGPUs to our environment and tried to power it on. Below screenshots show the vGPUs configured followed, unfortunately, by a screenshot of the error message we received. For completeness sake and search engine goodness here is the error message I received in text:
Could not initialize plugin '/usr/lib64/vmware/plugin/libnvidia-vgx.so'
I did some digging and figured out that the NVIDIA Documentation explained why, you can only add multiple vGPUs using a particular profile. The VMware guidance on this is a bit fuzzy unfortunately, but I requested the blog post to be edited. The NVIDIA document is very clear, you can only use the 48q or 48c profile with the RTX8000. For the various cards this is documented here. After adding two vGPUs with the 48q profile, like shown below, the VM powers on perfectly, well after you powered off any other workload using a vGPU that is, as the 48q/c profile claims the full card!
And for completeness sake, if you don’t power off the other VMs using a vGPU then you will receive the following error:
the amount of graphics resource available in the parent resource pool is insufficient for the operation.
Which is also shown in the below screenshot.
So to recap, if you want to use multiple vGPUs here are the requirements/constraints:
- vSphere 6.7 Update 3 is required
- VM Compatibility Version 15 is required
- Only “q” anc “c” profiles are supported
- You need to select the full frame buffer for each vGPU (highest), for RTX8000 that would be 48q or 48c
- This results in not being able to run any other vGPUs on those physical cards
- This also means that you need a physical card per vGPU you want to assign, two vGPUs requires two physical GPUs!
- Maximum of 4 vGPUs per VM
I hope this helps someone who is planning on configuring something similar.