by lunarg on January 10th 2017, at 14:42

If your ESX host has lost connection to the vCenter server, but you can still directly access the host's management, and can find nothing else wrong with the vCenter server or its network connection to the host, you may want to the free space on your ESX host.

Logs and other temporary files may have filled up one or more ramdisk partitions (e.g. when you don't have a persistent scratch partition, it gets mounted on /tmp). This can cause any of the following issues:

  • Connection loss between the ESX host and vCenter server;
  • Errors when attempting to reconnect or add an ESX host to a vCenter server;
  • Errors while attempting to activate or configure HA;
  • Other unexplainable errors...

You can check free space for all ramdisks through the ESXi Shell or remotely through SSH:

vdf -h

You can ignore output about the tardisks, and look directly at the ramdisk partitions. You should get an output similar to this:

Ramdisk                   Size      Used Available Use% Mounted on
root                       32M      520K       31M   1% --
etc                        28M      184K       27M   0% --
tmp                       192M        4K      191M   0% --
hostdstats                303M        3M      299M   1% --

If you see that one of these has filled up (i.e. no or almost no available space left), you need to make some room.

Clearing log files from /tmp

Usually the tmp ramdisk gets filled up by logs from certain VIBs. Look in the /tmp folder, and check whether any logs are taking up a lot of space. The usual culprits are those generated by SANs or faulty/buggy drives, which don't log to the default location, but log to the /tmp partition instead.

You can easily clear logs from the shell by emptying them, like so:

> /tmp/log/logfile.log

If you require the contents of the log files, you can copy them off the server first, either to a datastore, or through SCP to another host or SSH-enabled file server.

About logging to the "scratch partition"

Normally, it is good and common practice to set up logging to a persistent datastore. This resolves the problem about your scratch partition filling up with logs. However, not all VIBs play fair. Some ignore the default location for logging, and create log files on the /tmp partition. As this partition is also used for installation and configuration of VIBs (among other things), a full /tmp can cause issues like those stated earlier in this article.

Known VIBs that log to /tmp

  • Certain (older) versions of the Q-Logic adapter driver create large logs at /tmp/log/ql_ima.log (solution here).
  • The Dell EqualLogic multipathing VIB creates logs at /tmp/log/equallogic/.
« August 2017»
« Most people tend to avoid true conflict. Ironically this breeds more conflict. »