I’ve been doing VMware Consolidated Backup troubleshooting for the last couple of days. A customer ran into problems that I can’t comment on at this moment. But after an upgrade of VCB 1.1 to VCB 1.5 the customer ran into a new limitation of VCB. After 30 VM’s the script stopped working, the following error was thrown at us:
‘vcbMounter’ 5648 error] Error: Cannot mount volume 1, service not accepting new devices.
After a few search actions I noticed the following in the documentation which is clearly a new limit in VCB 1.5:
NOTE Consolidated Backup supports a maximum of 60 concurrently mounted virtual machines. For example, you can concurrently mount 60 virtual machines that have a C: drive, or 30 virtual machines that have a C: and a D: each.
In other words, no more than 60 vmdk’s maybe mounted concurrently. This limit wasn’t in 1.1, well not hard coded anyway… but 1.1 still has it’s limitations!
Clearly, on the part of having more than 5 concurrent VCB dumps, I know that this isn’t a best practice but for this customer it’s what they want and need. I stronly advise against it for any environment though! Follow the best practice of a maximum of 5, and set it up in a way that it involves 5 different datastores!
We are currently investigating other options and trying to find out what the max concurrent connections should be within the environment of this specific customer. Taking all kinds of different factors in consideration like “vmfs locking”, “scsi reservations”, stress on the vmkernel and or service console, diskspace occupation combined with fast growing snapshots etc.
I’ve been looking into VMFS locking associated to snapshots. VMFS locking occurs when metadata changes, in other words it happens with one of the following actions: snapshot file growing, vm starting(cause the file is being locked for read/write), file creation etc.
VMFS Locking means that there is only 1 host able to access the VMFS until the lock is released. So you can imagine what happens when there are 5 vm’s on the same VMFS on five different ESX hosts with snapshots that are growing! It will be like a monday morning traffic jam! So please don’t over do it.
I’ve also got the feeling that VCB is probably the most underrated and misunderstood product out there. I’ll be the first to admit that “file level” backups with VCB isn’t always as convenient as it should be but this is also due to the fact that not every Backup vendor has developed a decent integration module. But for instance CommVault Galaxy has got a special agent for VCB file level backups. This agent makes it possible to do a file level backup via VCB and restore direct to the VM via the agent! Check this PDF for more info on their solution. Full Image backups on the other hand are very useful for DR purposes but can also be used to restore single files again. You can mount the VMDK and browse the folders for the file. You can also use Vizioncore’s vRanger or Veeam’s “Veeam Backup” for a third party add-on to VCB. Both products are definitely worth checking out, and are a great extension to an often overlooked product!
Talking about Full Image Level backup’s besure to read this article, it will save you disk space on your “holding tank” and Tape Library!