Systems and Operations

The Complete Linux Low-Disk-Space and Inode Exhaustion Troubleshooting Guide: Safe Cleanup, Log Rotation, and Releasing Ghost File Handles

At a glance: Nearly every cloud operations engineer has encountered the urgent “No space left on device” alert. In lightweight production environments, such as BandwagonHost's common entry-level KVM plans with typically 20GB~40GB SSD storage, or DMIT's entry-level cloud servers such as PVM plans with 10GB~40GB high-speed NVMe storage, disk capacity is often tightly budgeted. Expensive premium BGP/CN2 GIA bandwidth leaves less room for spare storage. If logs grow unchecked, caches accumulate, or

VPSMap Editorial · Updated 2026-09-26 · 24 min read · Plain-text version

Almost every engineer administering cloud servers has encountered No space left on device(No space left on device) emergency alert. In lightweight production environments, for example, BandwagonHost( BandwagonHost ) Common entry-level KVM plans (typically with a quota of 20GB40GB SSD) or DMIT entry-level cloud servers (such as 10GB models in the PVM series40GB of high-speed NVMe storage), disk space is often tightly budgeted. The high cost of premium BGP/CN2 GIA network bandwidth leaves less room for spare storage capacity. If logs grow unchecked, caches accumulate, or tiny files proliferate, available disk space can easily be exhausted within hours.

When storage actually reaches 100% full, the problem goes beyond applications being unable to write: the system may not even be able to create PAM authentication session files,/tmp temporary files or /run/user sockets cannot be allocated, causing ordinary remote SSH connections to be rejected outright.

This article explores the underlying mechanisms of Linux storage and the VFS (Virtual File System), focusing on data-block exhaustion and inode depletion. It provides a production-tested systematic troubleshooting process, a method for truncating ghost files held by process handles, and automated log-protection strategies.


One. Emergency Access: Recovery When a Full Disk Prevents SSH Login#

When disk usage is completely stuck at 100%, the OpenSSH daemon may be unable to write system audit logs while processing public-key or password logins, or unable to provide for sshd allocating a PTY terminal for the session and returns Connection closed by remote host. Do not blindly force a power cycle at this point: memory buffers that have not been flushed to disk and damaged metadata may cause the filesystem check (fsck) to get stuck in read-only mode.

Use the Provider's Out-of-Band Management Console:

1. BandwagonHost: Bypass the Network Layer with KiwiVM Interactive Console#

BandwagonHost's cloud servers are managed by its proprietary KiwiVM Control Panel under management. In extreme cases where standard SSH access is unavailable:

  • Log into KiwiVM and find in the left navigation Interactive Console or Root shell - interactive.
  • Note: the KiwiVM control panel administration password is independent of both your client area account password and the operating system's root password.
  • Through a browser-emulated hardware-level terminal, you can operate without relying on the system's /run a temporary session network, directly obtain a shell and perform emergency space-recovery operations.

2. DMIT: Open the Web Console Through the Console's Access Panel#

During initial deployment, DMIT system templates generally disable remote root password login by default and use SSH key pairs. If a full disk prevents public-key authentication from writing temporary files:

  • Log into the DMIT client area, open the relevant instance's management interface, and switch to Access label.
  • Click Console open the interactive web terminal.
  • The console provides a basic TTY login environment, allowing direct local-terminal-style troubleshooting when network services malfunction or system storage becomes locked up.

Two. Core Mechanisms: Separate Accounting for Disk Blocks and Inodes#

Linux filesystems, such as the widely used Ext4 and XFS, divide physical storage into two independent structures:

Command / configuration
                           Linux 存储空间计量体系
                 ┌───────────────────────────────────────┐
                 │          文件系统 (Ext4 / XFS)        │
                 └──────────────────┬────────────────────┘
                                    │
           ┌────────────────────────┴────────────────────────┐
           ▼                                                 ▼
┌──────────────────────┐                         ┌──────────────────────┐
│  数据块 (Data Block) │                         │   索引节点 (Inode)   │
├──────────────────────┤                         ├──────────────────────┤
│ · 存储文件真实内容   │                         │ · 存储文件元数据     │
│ · 尺寸通常为 4KB/块  │                         │ · 大小、权限、时间戳 │
│ · 指标查看:df -h    │                         │ · 指向数据块的指针   │
│ · 症状:大文件撑爆   │                         │ · 指标查看:df -i    │
│                      │                         │ · 症状:海量小文件   │
└──────────────────────┘                         └──────────────────────┘
  1. Block Exhaustion: This appears as df -h shows mount-point usage reaching 100%. This is usually caused by uncontrolled growth of one or a few huge files, such as unrotated database logs, Docker image layers, or core dump crash files.
  2. Inode Exhaustion: This appears as df -h shows several GB or even tens of GB of free space, but any touch or mkdir operations all return No space left on device. At this point, run df -i, you will find IUse% has reached 100%.
    • Cause: filesystems such as Ext4 fix the total number of inodes at formatting time, typically allocating 1 inode per 16KB of data by default. If an application accidentally generates millions of tiny files only tens of bytes long, such as expired PHP/Node.js Session files, an abnormally accumulated Postfix mail queue, or deeply nested frontend build dependencies, it may use only a few hundred megabytes of disk space while exhausting the inode pool and bringing the filesystem to a halt.

Three. Troubleshooting Stage One: Inspect the Entire Disk and Locate the Problem#

Whatever the alert, the first step after entering the terminal is to determine the nature of the problem and identify the target directory.

1. Identify the Type of Alert#

Run the following two commands to compare the data:

bash
# 检查物理块容量占用
df -hT -x tmpfs -x devtmpfs

# 检查 Inode 节点消耗量
df -ihT -x tmpfs -x devtmpfs

Parameter Descriptions:-x tmpfs -x devtmpfs filters out memory-resident virtual mount points to keep the investigation focused.

2. Identify Space Hogs: Safely Scan Large Directories#

Avoid Running Indiscriminate Commands Directly from the Root Directory: du -sh /*, because unrestricted recursion on systems with many small files or remote mounts can cause severe I/O blocking or even exhaust memory. Use commands limited to a single filesystem:

bash
# 仅在当前物理文件系统内查找(-x 避免遍历 /proc, /sys 等伪文件系统),输出最大的前 15 个目录/文件
du -ahx / 2>/dev/null | sort -rh | head -n 15

If temporary package installation is allowed, use the modern tool ncdu can greatly improve analysis efficiency:

bash
# Debian / Ubuntu 环境
apt-get install -y ncdu

# 扫描根目录并进入交互式图形界面(-x 限制在当前文件系统内)
ncdu -x /

3. Locate Inode Consumers: Efficiently Measure File Density#

Finding which directory contains huge numbers of tiny files is a difficult part of traditional troubleshooting. Conventional ls may become completely unresponsive on directories containing millions of files. Use find streaming output combined with sorting can locate it with lower overhead:

bash
# 统计各目录下的文件总数并按降序排名前 15
find / -xdev -type f -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -n 15

Four. Hands-On Stage Two: Safely Clean Up Common System-Level Space Consumers#

On small VPS instances, most disk-full problems occur in a few predictable areas: system logs, package-manager leftovers, old kernels, and container data. The following procedures prioritize production safety and avoid blindly using rm -rf /var/log/* causing the service to crash.

1. Reduce Systemd Journal Disk Usage#

Modern Linux distributions (Debian 11/12, Ubuntu 20.04/22.04/24.04) use the following by default: systemd-journald collects stdout/stderr from various systems and services. Without quota limits, logs can easily consume 2GB~4GB of space.

bash
# 检查当前 journal 日志实际磁盘消耗
journalctl --disk-usage

# 生产级清理:强制截断日志,仅保留最近 3 天或上限 150MB
journalctl --vacuum-time=3d
journalctl --vacuum-size=150M

Long-Term Hardening Configuration: Modify /etc/systemd/journald.conf, set hard limits to prevent the problem from recurring:

ini
[Journal]
SystemMaxUse=200M
SystemMaxFileSize=50M

After saving, run systemctl restart systemd-journald for it to take effect.

2. Clean Up Package Manager Caches and Orphaned Dependencies#

On systems maintained with APT, the files downloaded during each upgrade .deb packages are all stored in full in /var/cache/apt/archives/ in:

bash
# 查看 apt 缓存占用的容量
du -sh /var/cache/apt/archives/

# 安全清空全部 deb 包缓存(不影响任何已安装软件)
apt-get clean

# 自动卸载已被淘汰、不再被任何现行软件依赖的库文件与孤立包
apt-get autoremove --purge -y

3. Remove Old Kernel Versions and Clean Up /boot#

On a basic VPS configuration, if the system is /boot has a small separate partition, common in certain system templates that allocate only 500MB1GB), the system repeatedly apt upgrade 3 left behind afterward5 old kernel images (vmlinuz, initrd.img) will directly cause /boot fill it completely, interrupting any subsequent updates or patch installations.

bash
# 1. 确认当前正在运行的内核版本(务必确认,绝对不可删除运行中的内核!)
uname -r

# 2. 列出系统内所有已注册的内核相关包
dpkg -l | grep -E 'linux-image|linux-headers|linux-modules'

# 3. 使用 APT 安全移除未使用的旧内核(现代发行版首选方式)
apt-get --purge autoremove -y

If the system, because of /boot has completely run out of free space, causing apt itself fails and becomes stuck, you can use lower-level tools to manually remove the oldest kernels and break the deadlock:

bash
# 假设 5.15.0-80 为陈旧未使用内核,5.15.0-105 为当前内核
dpkg --purge linux-image-5.15.0-80-generic

4. Thorough Cleanup of the Docker Container Environment#

Docker dangling image layers (Dangling images), obsolete build caches, and container logs continuously written to standard output are the biggest space consumers in containerized deployments.

bash
# 全局查看 Docker 各子组件的占用细则
docker system df

# 一键清除所有已停止的容器、未使用的网络和虚悬镜像
docker system prune -f

# 进阶清除:如果需要清理所有未被现存容器使用的镜像(包括长期不用的基础镜像)
docker system prune -a -f

# 清理无名卷(Dangling Volumes,注意确认其中没有包含重要未挂载数据)
docker volume prune -f

Truncation Method for Uncontrolled Docker Container Logs: Many long-running containers lack log-driver size limits, so a single *-json.log may grow beyond 10GB:

bash
# 定位最大的容器日志文件
find /var/lib/docker/containers/ -name "*-json.log" -exec du -h {} + | sort -rh | head -n 10

# 在容器运行中进行清空(必须使用截断方式,不能直接 rm,原因见第五章)
truncate -s 0 /var/lib/docker/containers/<container_id>/<container_id>-json.log

Five. Tackling the Hard Part: Releasing “Ghost Deleted Files” Held by Open Handles#

Here is a classic scenario that puzzles many administrators: You discover /var/log/nginx/ under access.log occupied a full 8GB, so you confidently ran rm -f /var/log/nginx/access.log. After execution ls no longer shows the file, but then running df -h,find that used space has not changed at all and the disk still shows a critical 100% usage level.

1. The Operating System Mechanism Behind the Behavior#

In Linux VFS, a file's existence is maintained by two components:

  1. Directory Entries (dentry) and Hard Link Count (nlink):rm The System Call Essentially unlink, it merely removes the file's directory entry and changes the Inode reference counter nlink decreases by 1.
  2. Process Open File Descriptor Count (Open File Descriptors): As long as an active process, such as Nginx Master/Worker, Java, or Python, retains a file descriptor (fd) for the file, the Inode's reference count remains greater than 0.

Conclusion: only when nlink == 0 and All Processes Holding That fd Have Closed Their Handles or Exited only then does the filesystem actually release the corresponding data Blocks back to the free storage pool.

Command / configuration
执行 rm 操作后的物理状态:
[用户执行 rm] ──> 目录项断开 (ls 无法查见)
                        │
                        ▼
      [持有该文件句柄的进程 (如 Nginx PID: 12480)]
                        │
                        ▼
    Inode 引用仍存在 ──> 磁盘扇区锁定 ──> 容量绝不归还系统!

2. Identify Processes Holding Ghost Files Open#

Use lsof(List Open Files) command accurately identifies all files marked deleted whose handles have not yet been released:

bash
# 安装 lsof (若系统缺失)
apt-get install -y lsof 2>/dev/null || yum install -y lsof 2>/dev/null

# 查找处于 (deleted) 状态但依然被进程持有的文件
lsof +L1
# 或使用通用过滤:
lsof | grep -iE 'delete|deleted'

Typical output is as follows:

text
COMMAND   PID USER   FD   TYPE DEVICE    SIZE/OFF NODE NAME
nginx   12480 root    7w   REG  252,1  8589934592 131075 /var/log/nginx/access.log (deleted)

The output clearly shows: the PID is 12480 of nginx process is still using the file descriptor 7w holds a deleted log file that still occupies 8.5GB of physical space.

3. Two Ways to Reclaim Space Safely#

Method A: Gracefully Reload or Restart the Associated Service (Recommended Standard Approach)#

If the service can tolerate a brief restart, reset the process lifecycle directly:

bash
# 优雅重载服务(如果服务支持重新打开日志文件,如 Nginx 的 reopen)
nginx -s reopen || systemctl reload nginx

# 或直接重启服务以彻底关闭历史文件描述符
systemctl restart nginx

Method B: Truncate Through the Process's Open File Descriptor While Running (Seamless Zero-Downtime Approach)#

In environments where critical services, such as high-throughput gateways, payment cores, or uninterrupted long-lived connections, cannot be restarted, you can directly use the Linux kernel's /proc pseudo-filesystem to locate the process's file descriptor and truncate it in place:

bash
# 格式规范:: > /proc/<PID>/fd/<FD_NUM>
# 或使用 truncate 命令:truncate -s 0 /proc/<PID>/fd/<FD_NUM>

# 针对前文示例中的 PID 12480 和 FD 7:
: > /proc/12480/fd/7

How It Works: The redirection operation : > sends the active handle a O_TRUNC a truncation signal, the system immediately returns the physical data blocks to the free pool and reduces the file size to 0 bytes, whileThe Process Itself Does Not Throw Any Invalid-Handle Errors. After completion, run again df -h, and disk space will immediately return to normal.


Six. Prevent Problems Before They Start: Build Automated Logrotate Rotation and Self-Recovery#

The fundamental way to avoid being awakened at night by a full disk is to make log rotation and quota-based circuit breakers part of the system's routine background management.

1. Configure Standard Logrotate Rules#

Directly deleting active log files is a serious operational mistake. The proper approach is to configure logrotate Periodically rotate, compress, and move archived logs.

For custom applications, such as those running in /data/app/logs workloads), write standardized rules /etc/logrotate.d/my-app:

ini
/data/app/logs/*.log {
    daily                   # 每日检查切分
    rotate 7                # 仅保留最近 7 份归档
    missingok               # 若日志文件丢失不抛出报警
    notifempty              # 文件为空时不执行切分
    compress                # 使用 gzip 压缩归档文件,削减 90% 存储
    delaycompress           # 延迟到下一次轮转周期再压缩,保障写入安全
    sharedscripts
    copytruncate            # 关键指令:将源日志截断为 0,而非直接重命名移走,避免产生幽灵句柄
}

Important Note copytruncate: For those that do not support HUP signal to reopen log files without restarting, such services must declare copytruncate. It copies the current log contents to a backup, then immediately truncates the original file in place, ensuring that the process retains the same valid file descriptor and eliminating ghost handles.

2. Prevent Inode Exhaustion: Regularly Clean Up Sessions and Temporary Caches#

For systems that generate huge numbers of short-lived small files, configure scheduled cleanup of old files rather than waiting for an inode alert.

Add lifecycle management logic to crontab (for example, cleaning up /data/cache files under it that have not been modified for more than 7 days):

bash
# 编辑 root 定时任务
crontab -e

# 每天凌晨 3:30 执行一次清理,并直接在文件系统层删除,避免一次性载入内存
30 3 * * * find /data/cache/ -type f -mtime +7 -delete

Performance Advantages:-delete the operation is performed by find execute directly at the kernel level unlink, far more efficient than find ... | xargs rm, and will not produce the following error from an excessively long argument list: Argument list too long error.


Seven. Completing the Operations Cycle: Quick Reference for Common Inspection Commands#

Add the following checks to your routine inspection process:

Troubleshooting TargetKey Troubleshooting CommandsCore Value / Use Cases
Total Capacitydf -hT -x tmpfs -x devtmpfsQuickly Identify Which Physical Partition Is Full
Inodesdf -ihT -x tmpfs -x devtmpfsDiagnose an Inode Shortage When “Cannot Create File” Appears Despite Free Disk Space
Locate Large Filesdu -ahx / 2>/dev/null | sort -rh | head -n 15Find Large Directories and Individual Files Within a Single Filesystem, Working from Deeper Levels Upward
Small-File Accumulationfind / -xdev -type f -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -n 15Locate Directories Where Huge Numbers of Tiny Files Cause Inode Exhaustion
Ghost File Handleslsof +L1 or lsof | grep -i deletedFind those that have been rm Deleted Zombie Files Whose Physical Space Is Still Held by a Service Process
Truncate While Online: > /proc/<PID>/fd/<FD>Instantly Reclaim Physical Storage in Production Without Restarting Services or Causing Downtime
Log Truncationjournalctl --vacuum-size=100MQuickly Reclaim Several GB of System Logs Consumed by systemd

Mastering this approach, from out-of-band panel access and dual-track filesystem diagnosis to truncating ghost file handles and robust Logrotate management, ensures long-term system stability even on small VPS instances with limited hardware quotas, eliminating the risk of service interruptions caused by uncontrolled storage growth.