How to Fix Linux "No Space Left on Device" Error Caused by Unlinked Deleted Open Files

Linux reports the disk is full, but manually adding up the size of visible files doesn't come close to matching how much space is actually reported as used. This classic mismatch usually means a file was deleted while a running process still had it open — Linux keeps the underlying disk space allocated for as long as any process holds a file descriptor to it, even though the file no longer appears anywhere in the filesystem.

The Problem

df reports the disk as nearly full, but visible files don't account for anywhere near that much space:

$ df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   48G  1.2G  97% /var/log

$ du -sh /var/log/*
1.2G    /var/log/nginx
890M    /var/log/syslog

The visible files only account for a couple gigabytes, nowhere close to the 48GB df reports as actually used — a large, unexplained gap between what df and du report is the signature of this exact issue.

Why It Happens

Linux filesystems separate a file's directory entry (its name and location) from its actual data blocks on disk. Deleting a file removes the directory entry, but if a running process still has that file open, the kernel keeps the underlying data blocks allocated — the space isn't actually released until every process holding it open closes that file descriptor. This is a deliberate, sensible design choice (it lets a process keep writing to a file it's actively using even after it's deleted), but it's also a very common, non-obvious source of "disk full" surprises. Typical scenarios:

• A large log file was deleted directly (via rm) to free space, but the application still writing to it kept the file descriptor open, so the space was never actually freed.
• A log rotation tool deleted an old log file without properly signaling the application to reopen a fresh file handle, leaving the application still writing into the now-invisible, still-allocated old file.
• A crashed or hung process left a large temporary file open that was deleted by some cleanup process but never actually released because the crashed process is still technically running (perhaps as a zombie or hung state) and holding the handle.

The Fix

Find exactly which process is holding open, deleted files that are still consuming space — lsof can specifically filter for this:

sudo lsof +L1

COMMAND   PID  USER   FD   TYPE DEVICE SIZE/OFF NLINK  NODE NAME
nginx    8821  root   12w  REG    8,1  47GB      0  912841 /var/log/nginx/access.log (deleted)

+L1 specifically lists open files with a link count of less than 1 — exactly the "deleted but still open" state you're looking for. This confirms nginx is holding a 47GB deleted log file open, accounting for almost all the missing space.

The cleanest fix is signaling the process to reopen its log files, which releases the old (now genuinely inaccessible) file handle and starts fresh — most services support this via a reload signal rather than requiring a full restart:

sudo systemctl reload nginx

For applications without a graceful reload mechanism, a full restart accomplishes the same thing, at the cost of brief downtime:

sudo systemctl restart myapp

Verify the space was actually reclaimed:

df -h /var/log

If restarting the service isn't immediately practical and you need the space back right away, you can truncate the open file descriptor directly through /proc without touching the process at all — this frees the disk space while leaving the process running, though the process's internal file offset tracking may end up in an inconsistent state depending on how it writes to the file:

sudo truncate -s 0 /proc/8821/fd/12

Use this specific approach cautiously — it's a reasonable emergency measure to immediately free critical disk space, but a proper reload or restart afterward is still the cleaner long-term fix.

Still Not Working?

To prevent this from recurring, make sure your log rotation setup uses proper rotation semantics (like logrotate's copytruncate option, or a postrotate script that signals the application to reopen its log file) rather than simply deleting log files directly while the application is still running and writing to them:

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

The postrotate script ensures that whenever logrotate rotates the file, the application is explicitly signaled to release its old file handle and open a fresh one, preventing exactly this class of silent disk space leak from recurring on future rotations.