Linux Disk Space Monitoring and Inode Exhaustion Prevention

A Linux filesystem can run out of space in two ways: it can run out of blocks, which is what df -h shows, or it can run out of inodes, which is shown by df -i. When inodes are exhausted, you cannot create new files even though plenty of space remains. Monitor both, find the directories consuming them, and prevent growth with log rotation, cleanup jobs, and alerts that fire early.

What Is the Difference Between Blocks and Inodes?

A filesystem tracks data in blocks and metadata in inodes. Every file and directory uses one inode, which stores attributes such as ownership, permissions, and pointers to data blocks. The number of inodes is usually fixed when the filesystem is created, depending on the filesystem type and options.

A few very large files consume many blocks but few inodes. Millions of tiny files do the opposite: they use little space but consume inodes quickly. Session files, cache entries, mail queues, and build artifacts are common sources of inode exhaustion.

df -h        # block usage
df -i        # inode usage
du -xh --max-depth=1 /var | sort -h | tail
sudo find /var -xdev -type f | cut -d/ -f1-4 | sort | uniq -c | sort -n | tail

The symptom of inode exhaustion is confusing: errors such as no space left on device while df -h shows free space. Always check df -i when you see that message.

How Do You Find What Is Using Space?

Work from the top down. Check which filesystem is full, then drill into directories to find the largest ones. Use options that stay within one filesystem, so mounted volumes do not distort the numbers.

• Use du with a limited depth to locate large directories, and sort the output by size.
• Count files per directory when inodes are the problem, since size does not reveal them.
• Look at log directories, package caches, container storage, old kernels, and temporary directories first.
• Check for deleted files still held open by running processes. Space is not freed until the process closes the file, and tools such as lsof can list them.
• Remember that container runtimes keep images, layers, and volumes in their own directories, which can grow unnoticed.

The deleted but open file case explains a classic mystery: you remove a huge log, but free space does not increase. Restarting or signaling the process to reopen its logs releases the space.

What Monitoring Should You Set Up?

Do not wait for a full disk. Monitor usage and trends, and alert at thresholds that leave time to react. A disk that fills slowly over weeks needs a different response than one that fills in minutes.

• Alert on both block and inode usage percentages for every important filesystem.
• Use two levels, such as a warning at a moderate threshold and a critical alert at a higher one.
• Alert on the rate of growth, which can predict exhaustion before thresholds are hit.
• Monitor separately the filesystems where databases and logs live, since their failure modes are severe.
• Include the free space reserved for privileged use in your understanding of the numbers, since regular users may hit the limit sooner.

Most monitoring agents and exporters expose filesystem metrics, including inodes, so you rarely need custom scripts. Verify that inode metrics are actually enabled in your setup.

How Do You Prevent the Problem?

Prevention is mostly about controlling growth and choosing sensible filesystem settings.

• Configure log rotation with size and age limits, and compress old logs.
• Schedule cleanup of temporary files, caches, and expired sessions, and make applications delete what they create.
• Prune unused container images and volumes regularly, using the runtime's own cleanup commands carefully.
• Put high-volume data such as logs and uploads on separate filesystems, so one cannot starve the system partition.
• When creating a filesystem for millions of small files, choose inode density settings accordingly, since changing it later usually requires recreating the filesystem.
• Avoid storing huge numbers of files in one directory. Spread them across subdirectories.

Some filesystems allocate inodes dynamically, so inode exhaustion may behave differently. Check the documentation for the filesystem you use.

Frequently Asked Questions

Why do I get no space left on device when df shows free space?

You may be out of inodes. Check with df -i. Another cause is a quota or a reserved block limit.

Why did deleting a big file not free space?

A process may still hold it open. Find it with lsof and restart or signal the process.

Can I add inodes to an existing filesystem?

Generally not for fixed-inode filesystems. You usually must recreate it or move data to a new one.

How much free space should I keep?

Enough to absorb growth and spikes. Many teams alert at 80 percent and treat 90 percent as critical, adjusting by workload.

Summary

Watch both blocks and inodes, find culprits with du and file counts, check for deleted but open files, and prevent growth with rotation and cleanup. Consult your filesystem documentation for limits and options.