Recovering from “no space left on device” or “RAMDISK is Full” for ESXi without Reboot!
ESXi might, on occasion, show the following symptoms
- Host might get disconnected from Virtual Center
- Root (/) might come across as 100% full (can be checked by running vdf -h)
- Logs might show “no space left on device” or RAM Disk is full or something similar
- Manual search of the root folder (/) might show no obvious files eating up space
The following steps might help recovering the situation – especially if the ESXi installation is not running on an SD Card but on local storage that has free space available
- Rename or delete the existing /tmp folder. So either do a “mv /tmp /tmp.old” or do a “rm -rf /tmp”
- Create a new /tmp folder. So do a “mkdir /tmp”
- Restart the management services. So go “services.sh restart”
- Check the /tmp and check if there are new files being written to the /tmp which ideally it should have, as essentially what have done is recreated the /tmp which should be created on the local disk instead of the in memory default location for ESXi
Now check the free space using “vdf -h” if all has gone according to plan, the / should show less than 1% usage
Wait for the hostd to start (/var/log/vmware/hostd.log or /var/log/hostd.log should show “Begin Services”)
Connect host back to VC!
Note: Ideally, the above is just so you can get your workloads off the affected ESXi hosts without downtime. Ideally, next steps should be to collect logs, reboot the host, and search for the root cause for the / getting full.