Recently I blogged about a quite convenient way to release read-only locks on vmdk files that prevent snapshots from being committed. Unfortunately a quite common problem with 3rd party backup tools that leverage the vStorage API. Using StorageVMotion can help in releasing these locks but sometimes even SVmotion won't work.
One more side note here: one can ask why there can be a mismatch about the VM still running in snapshot mode and the vCenter server not knowing anything about it, or better said, without the snapshot manager showing any existing snapshots on that VM. That's because the backup programs send a "remove snapshot" command to the vCenter server. The VC sends the snapshot command to the ESX host that is responsible for the snapshot handling and at the same time it deletes the information about the existing snapshot from the VC database without waiting for the ESX to confirm snapshot removal. This is by design, don't ask me why. So the new "feature" "Consolidation" in vSphere 5.x is nothing more than a simple database query against all listed snapshots and this list is compared to the settings of each VM. When VC finds a mismatch it activates the "Consolidation needed" button. Curious....
Because I have troubleshooted several of these problems in the past I decided to give a short step-by-step guide to identify which system holds the lock and how to release it. I can't cover all situations where this problem occurs but in ~95% of all cases it was one of the two solutions provided here.
Lock [type 10c00001 offset 37965824 v 5443, hb offset 3821568
gen 28763, mode 2, owner 00000000-00000000-0000-000000000000 mtime 5628465
num 1 gblnum 0 gblgen 0 gblbrk 0]
RO Owner HB Offset 3821568 546b498e-875a9360-10e8-b499baa6cd9e
Addr <4, 31, 2>, gen 4, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 42949672960, nb 5120 tbz 3277, cow 0, newSinceEpoch 0, zla 3, bs 8388608
[...]The red marked string shows one of the MAC addresses of the host that holds the lock.
World ID: 11696695
Process ID: 0
VMX Cartel ID: 11696694
This problem is mainly caused by virtual backup proxies that use hot-add mode for data transport. If they crash or have other problems so the disk deattachment can't be completed the base VM disks remain connected and thus locked for other processes like snapshot removal tasks. That's the reason why a simple reboot of a virtual backup proxy won't help in this situation as the locked disks remain attached to the VM and are still locked for other processes.
To give a final advise: if you encounter locked files first reboot the physical backup server. Next check for the lock holder as described in this article. Next try a SVMotion. Next step is to call VMware support as you probably have a rare problem where standard procedures won't help. Good luck.