ESXi host disconnected from vCenter Server & the ESXi localcli command


Today i experienced an ESXi host, version 6.0 U1a, being disconnected from vCenter Server. I did try to connect direct to the ESXI host using the ESXi host embedded client, the vSphere Client and SSH without any success. The last attempt was to use the IPMI connection and that screen showed me the following:

Screen Shot 2016-02-16 at 21.45.02

That is usually not a good sign but i was lucky as the ESXi DCUI worked so i could enable ESSXi shell and access it via the F1 screen option. Based on the following log investigation i decided to disable to local ESXi firewall and when trying to disable the ESXi host local firewall i encountered another issue.

The commands i was trying to run were the following:

  • esxcli network firewall unload
  • esxcli network firewall set –enabled false

They resulted in different messages indication out of memory condition:

  • 2016-0x-xxT20:59:02.335Z cpu3:3582147)WARNING: UserMem: 7019: Failed to allocate pagetables for mmInfo: 0x43194e1ce180, startAddr: ff98 f000, length: 536576, pagePool: 18446744073709551615, status: Out of memory
  • 2016-0x-xxT20:59:57.977Z cpu32:3582449)WARNING: UserParam: 1301: could not chang e group to <host/vim/vimuser/terminal/ssh>: Admission check failed for memory resource
  • 2016-0x-xxT20:59:59.838Z cpu38:3582469)WARNING: User: 5366: Error in exec’d cart el setup: Failed to map section: Admission check failed for memory resource

As mentioned in the vSphere 6 documentation found here there is a way to bypass hostd when it is not responding and that is to use the command localcli instead of esxcli. You should use locally when instructed by support and this warning is also included in the VMware documentation:

If you use a localcli command, an inconsistent system state and potential failure can result. 

However didn’t have any other option so i ran the following commands:

  • localcli network firewall unload
  • localcli network firewall set –enabled false

When done i saw the following in the vobd.log:

  • [netCorrelator] 12406134193us: [vob.net.firewall.config.changed] Firewall configuration has changed.

About 60 seconds after disabling the ESXi host local firewall the ESXi host was joined to the vCenter Server. When you have used localcli you are instructed to restart hostd, in my case i put the ESXi host in maintenance mode and restarted it.

A case has been created with VMware and i’ll update the blog post when we find the root cause.


6 pings

Comments have been disabled.