Yellow Bricks

by Duncan Epping

Delete all snapshots

· ·

Today I encountered an old misunderstood principle again. A customer had created several snapshots on a virtual machine. Several… well to be exact 15. All snapshots were larger than 20GB. When the VMFS volume, on which this VM was located, ran out of diskspace he decided to use the button “Delete All”, but within a couple of minutes the VMFS volume ran out of diskspace again. What happened?

Situation:
Snapshot 1 – 20GB
Snapshot 2 – 10GB
Snapshot 3 – 30GB

When you choose “delete all” the following will happen:

  1. Snapshot 2 will grow to 40GB at most
  2. Snapshot 1 will grow to 60GB at most
  3. Snapshot 1 will be committed to the original VMDK
  4. All snapshot files are deleted

In other words: Snapshot 3 is merged into Snapshot 2, Snapshot 2 is merged into Snapshot 1, Snapshot 1 is merged into the original flat.vmdk and afterwards all snapshot files are deleted. This means that if you want to delete all snapshots at once you will need around 130GB of free diskspace. So think twice when you press the “delete all” button.

Reader Interactions

Comments

  2. dbirnie says

    You know you IT types are so out of touch with reality.
    The reasons for searching databases like yours is to recover from program defects.
    If you don’t want to give an answer, stop posting.
    How helpful is it to say “think twice”.

  3. Edoardo Fazzini says

    By the way i remeber also that deleting snapshot process (committing changes) in VC console, viewed below in status bar, generate a timeout error when it arrives at 95%.
    This is not a problem, because on side Host the process continue its works and finish after sometimes (it depends on how many GB are going to commit.. maybe hours..). So don’t worry!!! Be patient!!!

  4. lldmka says

    The author probably assumed a certain level of intelligence in his readers.

    By ‘think twice’ he clearly means to think about whether you have enough free space on the volume to commit the snapshots.

  5. Duncan Epping says

    Correct, think twice before you start deleting… My Article wasn’t about solving a problem, it was about explaining what happens in the background. It’s the way it’s been designed.

  6. Oinkmaster says

    If you just start by deleting the oldest snapshot then you need no additional storage at all. It then just gets merged in the original vmdk-file.

  7. Steve says

    Ok, so I don’t necessarily agree with oinkmaster… My VMs tend to be quite complicated, I keep the system drive’s VMDK on one datastore, along with the config files. I then tend to have several additional VMDK files that I keep on different VMFS datastores.

    Example, if the VM is a database, then I have a database LUN that is a lot faster and has more expensive technology at the SAN level, and then the logfiles have their own partition at the VM level, which is an additional VMFS datastore on what we call “medium” LUNs, which are kind of in between the LUNs where we keep the OS vs LUNs where we keep the Databases…

    So when I snapshot, even though the VM is associated with several LUNs, the delta is always written to the LUN/VMFS datastore where the VM’s config files are located. I have never seen this documented and had to learn this on my own, and it is possible I am just incorrect/crazy 🙂

    So yes, if your VM has a single VMDK file and the config files are stored on the same VMFS datastore, then Oinkmaster is correct, the space you use on the VMFS will probably not change much (although if the delta file contains say a delete command for half of the space, then I imagine when the snapshot is applied it would actually shrink the VMDK…?)

    But, in more complex scenarios, like where I have snapshotted my database VM and the delta for the LOGS vmdk has been written to a different datastore (the datastore where the VM’s config files are) then deleting/applying the snapshot most definitely causes the VMDK file to grow. And it can grow a lot.

    So in this case, you should not only be aware of the VMDK growing when you delete a snapshot, you also need to make sure that the VMFS datastore where the configs are located for that VM has enough free space to house the delta. Leaving a snapshot for too long can not only hose the VM, but it you are overprovisioned on your datastore (and with snapshots you may end up being overprovisioned and not know it) you really need to be aware of the free space left on the datastore that houses the config files for that particular VM.

  8. Ruben says

    Oinkmaster is right, the nicer way to remove several snapshots is to remove the nearest to the Base Disk repeatedly. This doesn’t require additional space.

    If the Snapshot Manager displays no snapshots… then the problem begins.

    On http://vmutils.blogspot.com/ you can find an script that will tell you in advance how much space the VM would need to ‘delete all’.

    However, better than solve a problem is to prevent it.

  9. Arian van der Pijl says

    Hi,
    Old posting but maybe someone can explain why the ‘delete all snashots’ function is not equivalent to manually delete the snapshots one by one?
    Regards,
    Arian!

  10. Vladan says

    Arian,

    When you delete the snapshots one by one, after each merge operation, there is a delete operation to clean the delta. That delete operation frees a disk space… you see?

    If you click the delete all, you have at first all the merge operations and only at the end you have a delete operation.

    That’s when you possibly run into troubles not having enough FREE space on your datastore to be able to delete all the snapshots AT ONCE.

    • Balu says

      I am planning to delete old snapshots of few production servers.The issue is current datastore doesn’t have much free space so i need to delete snapshots one by one.There are not more than 3 snapshots.Now my doubt is which snapshot should i delete first? the latest one or the oldest one.i am searching to find any kb’s related to this.It would be great if you could provide some insight into this.The version is ESX 4.0.0 208167

  11. Dennis Bray says

    With ESX 4 Update 2, (http://www.vmware.com/support/vsphere4/doc/vsp_esx40_u2_rel_notes.html#whatsnew)VMware has changed the order that the “Delete All” operation consolidates the snapshots. The snapshots all still need to be merged with their parent disk, but now the operation will start with the first snapshot (the one closest to the parent) and progress to the most recent. This should help reduce the number of times that some of the data is copied. This should increase the speed of consolidation, but the space requirement is likely unchanged.

  13. Chad King says

    Question?

    I have an interesting issue with a snapshot.

    So this snapshot has grayed out snapshot option. I followed this KB to just commit it with the VM off – I tried to do it when it was off and same story.

    Anyways – after doing the cloning process and allocating the new VMDK file that doesnt have a snapshot I still see that I have a snapshot file in that directory *.vmsn.

    Is there a best practice on how to get snapshot manager to work correctly again? I really don’t know what to do about this and it will more than likely still alert on having a snapshot when it now no longer has one. Thanks duncan.

  16. Srikanth says

    Suppose I created 5 snapshots, i.e snapshot 1………..snapshot5. I want delete snapshot3 , what happens next? i mean will 1,2,3 gets deleted??