Linux Log Rotation Strategies for High Volume Application Logs

For high volume application logs, rotate by size as well as by time, compress old files, limit retention, and prefer telling the application to reopen its log file over copytruncate. Better still, write logs to standard output and let the platform collect them, which removes most rotation concerns from the application. The right approach depends on how much you log and how reliable you need those logs to be.

Why Is Rotation Harder at High Volume?

Log files grow without limit unless something manages them, and busy services can produce gigabytes per day. A daily rotation schedule may be too coarse: a single day's log can fill the disk before the next rotation runs. At the same time, rotating too aggressively creates many small files that are tedious to search.

There is also the question of what happens at the moment of rotation. The application holds an open file handle to the log. If you rename the file, the process keeps writing to the renamed file until it is told to open a new one. If you delete it, the space is not freed until the process closes it. Rotation must coordinate with the writer.

Disk usage is not the only concern. Large log files slow down searching and shipping, and a runaway error loop can generate enormous volumes in minutes.

What Triggers and Settings Should You Use?

The standard tool on many Linux systems is logrotate, driven by configuration files that describe how each set of logs is handled. For busy logs, combine time and size limits, so rotation happens when either threshold is crossed.

/var/log/myapp/*.log {
    daily
    maxsize 500M
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    dateext
    sharedscripts
    postrotate
        systemctl kill -s HUP myapp.service
    endscript
}

• maxsize: rotate when the file exceeds a size even before the schedule says so. Note that logrotate usually runs from a daily timer, so size checks happen only when it runs. Run it more often for very busy logs.
• rotate: how many old files to keep. Combine with age-based settings for predictable retention.
• compress and delaycompress: compress old logs, but leave the most recent rotated file uncompressed in case the process is still writing to it.
• dateext: use dates in file names, which makes them easier to identify.

Option names and behavior vary slightly by version, so check the logrotate manual on your system.

Should You Use copytruncate or a Signal?

The copytruncate option copies the log file and then truncates the original, so the application keeps writing to the same file handle without any coordination. It is convenient for programs that cannot reopen logs, but it has a flaw: lines written between the copy and the truncation are lost, and at high volume that window matters.

The cleaner approach is to rename the file and signal the application to reopen its log. Many servers support a signal for this purpose, and applications you write can implement it. Alternatively, have the application manage its own rotation through its logging library, which knows exactly when to switch files.

• Prefer reopen signals or application-managed rotation for logs you cannot afford to lose.
• Use copytruncate only when the application cannot be changed, and accept the possible loss.
• Test the postrotate script, since a failing signal can leave the application writing to a deleted file.

Is There a Better Model Than Files?

For new services, avoid log files entirely. Writing structured logs to standard output lets the platform, whether systemd, a container runtime, or an orchestrator, capture and route them. The journal and container log drivers have their own size and retention limits, which you should configure explicitly.

Ship logs to a central system quickly, so local files are only a short-term buffer. This protects you if a host fails and reduces the retention burden on each server. Even then, set local limits so a failure in the shipping pipeline does not fill the disk.

• Reduce volume at the source by using appropriate log levels and sampling noisy messages.
• Alert on abnormal growth in log volume, which often signals a bug or an attack.
• Keep retention aligned with legal and operational needs, and delete beyond that.

Frequently Asked Questions

Why is my disk still full after rotation?

The application may still hold the old file open. Signal it to reopen logs, or restart it, so the space is released.

Is copytruncate safe?

It can lose lines written during the copy. Use it only when a reopen signal is not available.

How often does logrotate run?

Typically once a day from a scheduled timer, so size-based rules apply only then unless you run it more often.

How long should I keep logs?

As long as operations and compliance require, and no longer. Many teams keep recent logs locally and older ones in central storage.

Summary

Combine size and time triggers, compress, limit retention, and prefer reopen signals to copytruncate. Better yet, log to standard output and ship centrally. Check the logrotate manual for your system.