ESXi hosts disconnected in vCenter Server

Pluralsight Free Trial
While upgrading a VMware vCenter Server to the latest release, 5.5 Update 1b which can be downloaded here, i ran into two issues while trying to connect the ESXi hosts when the upgrade was completed. The ESXi hosts were all disconnected after the vCenter Server upgrade.

Screen Shot 2014-07-08 at 16.48.05

The solution/s to my problem/s are documented in different VMware knowledge base articles so it was just a matter of trying one or more of them to solve my problem.

First problem – Cannot contact the specified host

When trying to connect my ESXi hosts by righ-click and select connect i received the following error:

Cannot contact the specified host. The host may not be available on the network, a network configuration problem may exist, or the management service on this host is not responding.

While searching the VMware knowledge base i found article 1027672 that somehow relates to my problem although the ESXi hosts were already added to the vCenter Server inventory once.

I my case i could however connect to the ESXi hosts directly by using the vSphere client, there were no network problems in terms of blocked firewall ports, DNS pointers (A and PTR) missing and the ESXi hosts did not use lockdown mode.

This problem was solved by restarting the ESXi host management services and i did this via a SSH connection to the ESXi host, and not via the remote connection (IPMI, iLO, DRAC ), following the below procedure:

  1. Login to the ESXi host via SSH
  2. type “dcui” to enter the direct console user interface
    Screen Shot 2014-07-08 at 17.16.27
    This will actually bring up the menu as you can see via the ESXi host console / primary monitor
  3. Select “Troubleshooting Options”
  4. Select “Restart Management Agents”

While finished i could connect one of the ESXi hosts successfully to the vCenter Server and to the vSphere cluster.

Second problem – ESXi host is being disconnect after connected to vCenter Server

When my first problem was solved by restarting the management agents i ran into another problem for the remaining three ESXi hosts. The ESXi hosts were disconnected during the vSphere HA configuration which is included in the connection process, a separate task but the connection process will not end before the vSphere HA configuration is finished. This means i could not manage the ESXi hosts via vCenter Server at any point in time.

I didn’t want to reconfigure the cluster and disable vSphere HA since i basically don’t know what will happen when you add ESXi hosts with local configuration that says vSphere HA is enabled to a vSphere cluster where vSphere HA is not enabled.

VMware knowledge base article 2040630 describes the scenario where you add an ESXi hosts and it becomes disconnected from vCenter Server 30 to 90 seconds after the connection. This is often caused by a firewall misconfiguration blocking port 902 over UDP which is used for heartbeat between the vCenter Server and ESXi in both directions.

Error message displayed for the ESXi host contained:

Agent is out of date and needs a manual upgrade

The resolution to the problem described in VMware knowledge base article 2016287 didn’t solve my problem and the KB is actually related to vCenter Server 5.1 and not vCenter Server 5.5 Update 1b but i gave it a try anyway.

So i decided to follow VMware knowledge base article 1003714 which describes how to reinstalling the management and HA agents. Via the SSH connection already established to my ESXi hosts i ran the following commands:

  • cp /opt/vmware/uninstallers/VMware-fdm-uninstall.sh /tmp
  • chmod +x /tmp/VMware-fdm-uninstall.sh
  • /tmp/VMware-fdm-uninstall.sh

Now i could connect two of the remaining three ESXi hosts.

So for the last ESXi host i deleted the vpxuser account, which will be added back automatically by vCenter Server, and with the vpxuser removed i could successfully connect the last ESXi hosts to my vSphere cluster

6 pings

  1. Newsletter: July 20, 2014 | Notes from MWhite

    […] hosts disconnected in vCenter This was a scary upgrade.  After upgrading to vSphere 5.5 U1b there were some problems!  I am so happy my upgrade was much […]

Comments have been disabled.