I’ve been doing VMware Consolidated Backup troubleshooting for the last couple of days. A customer ran into problems that I can’t comment on at this moment. But after an upgrade of VCB 1.1 to VCB 1.5 the customer ran into a new limitation of VCB. After 30 VM’s the script stopped working, the following error was thrown at us:
‘vcbMounter’ 5648 error] Error: Cannot mount volume 1, service not accepting new devices.
After a few search actions I noticed the following in the documentation which is clearly a new limit in VCB 1.5:
NOTE Consolidated Backup supports a maximum of 60 concurrently mounted virtual machines. For example, you can concurrently mount 60 virtual machines that have a C: drive, or 30 virtual machines that have a C: and a D: each.
In other words, no more than 60 vmdk’s maybe mounted concurrently. This limit wasn’t in 1.1, well not hard coded anyway… but 1.1 still has it’s limitations!
Clearly, on the part of having more than 5 concurrent VCB dumps, I know that this isn’t a best practice but for this customer it’s what they want and need. I stronly advise against it for any environment though! Follow the best practice of a maximum of 5, and set it up in a way that it involves 5 different datastores!
We are currently investigating other options and trying to find out what the max concurrent connections should be within the environment of this specific customer. Taking all kinds of different factors in consideration like “vmfs locking”, “scsi reservations”, stress on the vmkernel and or service console, diskspace occupation combined with fast growing snapshots etc.
I’ve been looking into VMFS locking associated to snapshots. VMFS locking occurs when metadata changes, in other words it happens with one of the following actions: snapshot file growing, vm starting(cause the file is being locked for read/write), file creation etc.
VMFS Locking means that there is only 1 host able to access the VMFS until the lock is released. So you can imagine what happens when there are 5 vm’s on the same VMFS on five different ESX hosts with snapshots that are growing! It will be like a monday morning traffic jam! So please don’t over do it.
I’ve also got the feeling that VCB is probably the most underrated and misunderstood product out there. I’ll be the first to admit that “file level” backups with VCB isn’t always as convenient as it should be but this is also due to the fact that not every Backup vendor has developed a decent integration module. But for instance CommVault Galaxy has got a special agent for VCB file level backups. This agent makes it possible to do a file level backup via VCB and restore direct to the VM via the agent! Check this PDF for more info on their solution. Full Image backups on the other hand are very useful for DR purposes but can also be used to restore single files again. You can mount the VMDK and browse the folders for the file. You can also use Vizioncore’s vRanger or Veeam’s “Veeam Backup” for a third party add-on to VCB. Both products are definitely worth checking out, and are a great extension to an often overlooked product!
Talking about Full Image Level backup’s besure to read this article, it will save you disk space on your “holding tank” and Tape Library!
Pierre says
Hello,
the limitation of 60 is in all VCB Versions (see vi3_vm_backup old version).
From my experience you should do only 1 backup task. 2 are at maximum concurrent supported.
My point of view is follow up all restrictions on VCB then you will be happy. But there so much things what gonna happen or go wrong: AD, Exchange, file server with large amount of files (> 1 million per partition) or capacity (>800 GB per partition)
Best regards Pierre
Duncan says
Well with commvault and datadomain storage I noticed that the performance started to decrease with more than 5 concurrent… I usually don’t hit these limits, I always follow the best practices, that’s why they are there…
Rocco Smith says
Nice post. As the owner of http://hubhits.net I enjoy searching the Internet looking for unique useful info I can use to better my traffic exchange and business. I stumbled upon you on AOL and I have enjoyed reading your blog post. I’m bookmarking it now and will return soon.