Virtual machine disk consolidation needed.

This blog post applies to the vCenter Server version 5.1.0.880146 and ESXi version 5.1.0.799733.
Today i ran into an issue with the Snapshot Consolidate feature. The vSphere Web Client informed me that a “Virtual machine disk consolidation needed” was required for my VM. The virtual machine is backed up by vSphere Data Protection 5.1.0.56.179 and we were not allowed to power off the running VM during the initial troubleshooting process.

I verified the relationship between the vSphere Web Client Snapshot Manager and the number of virtual machine disk files in the virtual machine home directory.
No existing snapshots visible through the Snapshot manager.

8 existing snapshots visible while examine the virtual machine disk files:
 
Information:Unless the default configuration is changed, all virtual machine snapshot files will be placed in the virtual machine home directory and not in the directory where the virtual machine disk file is placed. 

Next thing, i scanned through the virtual machine log file, vmware.log, and found:
Failed to open ‘/vmfs/volumes/4de74f78-1b00b4d0-92b5-e41f13b6baf2/VM01/VM01-000008.vmdk’ with flags 0x20a Failed to lock the file (16392).

I tried to use the Consolidate option but it didn’t solve my problem, received the following error message:

The below section was added to the virtual machine vmware.log file:
 vcpu-0| I120: DISK: Failed to open disk for consolidate ‘/vmfs/volumes/4de74f78-1b00b4d0-92b5-e41f13b6baf2/VM01/VM01-000008.vmdk’ : Failed to lock the file (16392) 6873

I decided to try the trick “create virtual machine snapshot” and then “Delete All” snapshots.
Unfortunately, this procedure only created one additional set of vmdk files for both virtual machine disk files meaning i now had 9 snapshots on disk and non in the vSphere Web Client.
The vSphere Web Client reported status “Completed” for the “Remove all snapshots” action but the following error was presented in the virtual machine vmware.log file:
I120: SnapshotVMXConsolidateOnlineCB: nextState = 1 uid 423
| vcpu-0| I120: SNAPSHOT:Failed to open disk /vmfs/volumes/4de74f78-1b00b4d0-92b5-e41f13b6baf2/VM01/VM01-000008.vmdk : Failed to lock the file (16392)

I used the command “vmkfstools” command to verify the owner of the file VM01-000008.vmdk, complete command:
“vmkfstoold -D /vmfs/volumes/VMFS-datastore/VM01/VM01-000008.vmdk”
The MAC address (green marked in the below vmkfstools command output) pointed to the ESXi host where the virtual machine was running.

Lock [type 10c00001 offset 58699776 v 1403, hb offset 3981312
gen 115, mode 1, owner 509cfa01-0d1ebf09-50f6-001a5335ac75 mtime 1485900

I looped through all the other files in the virtual machine home directory looking for a different (another ESXi host) file owner but all virtual machine files were owned by the ESXi host where the virtual machine was currently running.
I used the command “esxcli network nic list” to verify the MAC addresses of the ESXi host physical NICs and the command “esxcli network  ip interface list” to verify the MAC address for the ESXi hosts VMK virtual adapters.

Since the virtual machine file owner couldn’t lock the file i decided to vMotion the VM to another ESXi host in the same cluster and then vMotion the VM back again.

When the two vMotion was finished i could successfully perform the Consolidate action.