Cannot power on virtual machine, CID mismatch

Last week i was contacted by two customers facing the same problem. Both customers running vSphere 5.1 U1 and ESXi 5.1 U1 and they couldn’t start one or more virtual machines and received the following error in the vSphere Web Client:

The useful option to submit an error report in the vSphere Web Client adds additional information.

The affected virtual machine shows to following error in the vmware.log filel located in the virtual machine home directory. The most informative part of the log has been marked with red text.

2013-05-24T10:52:25.550Z| Worker#0| I120: DISKLIB-LIB   : Failed to open ‘/vmfs/volumes/519bbc2f-4a21613e-d2fa-0022649f18c0/xyz01/xyz01-000001.vmdk’ with flags 0xa The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child (18).
2013-05-24T10:52:25.550Z| Worker#0| I120: DISK: Cannot open disk “/vmfs/volumes/519bbc2f-4a21613e-d2fa-0022649f18c0/xyz01/xyz01-000001.vmdk”: The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child (18).
2013-05-24T10:52:25.550Z| Worker#0| I120: Msg_Post: Error
2013-05-24T10:52:25.550Z| Worker#0| I120: [msg.disklib.CID_MISMATCH] The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child
2013-05-27T10:52:25.550Z| Worker#0| I120: [msg.disk.noBackEnd] Cannot open the disk ‘/vmfs/volumes/519bbc2f-4a21613e-d2fa-0022649f18c0/xyz01/xyz01-000001.vmdk’ or one of the snapshot disks it depends on.

The following information was displayed in the vSphere Web Client for one of my customers.

VMware has released an excellent KB, found here, describing the problem and how to solve it. One of my customers problem was related to the backup software used which takes snapshots of the virtual machines.

I will outline the exact steps i took when solving the problem and hopefully bring some additional value (command syntax) not present in the VMware KB.

• Identify what virtual machine files were affected by the problem. The virtual machine had only one snapshot meaning the disk descriptor file mentioned in the error log, xyz01-000001.vmdk, will be the one to correct. 
• Enable and log on to the ESXi Technical Support Mode (TSM) using the Directo Console User Interface (DCUI) accessibel from the ESXi HP iLO connection.
  SSH could have been used but this was not possible based on the firewall rules protecting the ESXi management network, not the ESXi firewall.
• Change working directory to the virtual machine home directory using the command syntax “cd /vmfs/volumes/VMFS-datastore/Virtual_machine_home_directory”. The exact command i used was:
  • cd /vmfs/volumes/ds-001/xyz01
• Took a copy of the desk descriptor file, xyz01-000001.vmdk, in case something goes wrong using the command syntax “cp virtual_machine-000001.vmdk /tmp/”. The exact command i used was:
  • cp xyz-000001.vmdk /tmp/
• Verifying the existing parentCID information in the snapshot disk descriptor file xyz01-000001.vmdk using the grep command:
  • grep -i parentCID xyz01-000001.vmdk
• The output shows:
  • parentCID=d354df56
• Displaying the correct parentCID value by listing the CID information in the original disk descriptor file xyz01.vmdk using the grep command:
  • grep -i CID xyz01.vmdk
• The output shows:
  • CID=db435dcc
  • parentCID=ffffffff
• The value we need is the one next to “CID”, db435dcc. The value next to parentCID in the original disk descriptor file, xyz01.vmdk, tells us that it does not depend on any  file.
• Change the parentCID value in the snapshot disk descriptor file xyz01-000001.vmdk using the “vi” command and the following sequence:
  • vi xyz01.000001.vmdk
  • use the arrow keys to navigate to the beginning of the parentCID value according to the belog figure:
    
  • type “cw” which is the “vi” command for changing a word
  • type the original disk descriptor file, xyz01.vmdk, CID value. In my case db435dcc
  • Save the file by using the following command sequence:
    • esc
    • :wq!
• Consolidate the virtual machine (only required for one of my customers) by clicking the “Consolidate” action in the vSphere Web Client found via the affected “Virtual machine -> Actions -> All vCenter Actions -> Snapshots -> Consolidate”
• Start the virtual machine.