Stop Logs Filling Your Disk: journald Limits, logrotate and Log Retention on Debian and Ubuntu

When a Linux server's root filesystem fills up, databases stop writing, package upgrades fail and SSH logins can break. Logs are one of the most common causes, because they grow quietly until the day they do not fit. This guide explains the two systems that manage logs on Debian and Ubuntu, the systemd journal and logrotate, and how to set limits so logs stay useful without taking over the disk.

Where logs live now

Since Debian 12 (bookworm), rsyslog is no longer installed by default. The bookworm release notes explain that messages are kept by the systemd journal in a binary format under /var/log/journal and read with journalctl. Servers upgraded from older releases, and many other installs, still run rsyslog as well and write text files such as /var/log/syslog. Applications such as web servers, PHP and databases usually keep their own text logs under /var/log. So there are typically two systems to manage: the journal, which manages its own size, and text files, which logrotate rotates.

First, find what is using the space

df -h /
sudo du -xh /var/log --max-depth=1 | sort -h | tail
journalctl --disk-usage

# space held by files that were deleted but are still open
sudo lsof +L1

The last command matters more than it looks. If someone deletes a large log while a process still has it open, the space is not freed until that process closes the file or restarts. lsof +L1 lists such unlinked open files (lsof(8)).

Limit the journal

By default, journald may use up to 10% of the filesystem for persistent logs and tries to leave 15% free, each capped at 4G, according to journald.conf(5). On a small VPS disk that can still be more than you want, and the default never deletes entries by age. Put your own limits in a drop-in file rather than editing the main file:

# /etc/systemd/journald.conf.d/50-limits.conf
[Journal]
SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=1month

sudo systemctl restart systemd-journald
journalctl --disk-usage

• SystemMaxUse caps the total size; SystemKeepFree keeps space free for everything else. journald respects both and uses the smaller result.
• MaxRetentionSec deletes journal files with entries older than the given time. It is off by default; set it if your retention policy says how long logs may be kept.
• Only archived journal files are deleted, so usage can briefly stay above the limit.
• Storage=auto, the default, keeps the journal on disk only if /var/log/journal exists. Do not delete that directory to save space: that makes logs disappear at every reboot.

To reclaim space immediately, rotate and vacuum by hand (journalctl(1)):

sudo journalctl --rotate --vacuum-size=500M
sudo journalctl --vacuum-time=30days

Rotate text logs with logrotate

logrotate renames, compresses and eventually deletes log files according to rules in /etc/logrotate.conf and /etc/logrotate.d/. Most packages, such as nginx and the database servers, install their own rule file there. Your applications usually do not. A rule for a PHP application's logs might look like this:

# /etc/logrotate.d/myapp
/var/www/myapp/storage/logs/*.log {
    daily
    rotate 14
    maxsize 200M
    compress
    delaycompress
    missingok
    notifempty
    su www-data www-data
    create 0640 www-data www-data
}

What each part does, per logrotate(8):

• daily with rotate 14 keeps two weeks of history.
• maxsize 200M rotates earlier when a file grows past 200 MB, but only when logrotate actually runs.
• compress with delaycompress gzips old files but leaves the most recent rotated file uncompressed, which helps programs that are still writing to it.
• su runs the rotation as the owner of the directory, which logrotate needs when the directory is writable by a non-root user.
• For a program that cannot reopen its log file, copytruncate copies and then truncates the file in place. The manual warns that a few lines may be lost in between.

Test a rule without changing anything, then force a run if needed:

sudo logrotate -d /etc/logrotate.d/myapp
sudo logrotate -f /etc/logrotate.d/myapp

The manual notes that logrotate usually runs once a day from cron or from logrotate.timer on systemd systems. Check with systemctl list-timers logrotate.timer. If a log can grow by gigabytes in a few hours, a daily run is too slow even with maxsize; fix the noisy logging, or run logrotate more often.

Container logs

Docker's default json-file logging driver keeps container output under Docker's data directory, and its max-size option defaults to unlimited (json-file driver). Set limits in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

Restart Docker afterwards; existing containers keep their old settings until they are recreated.

Keeping logs off the root disk

On busy servers, put /var/log on its own filesystem so a log storm cannot fill the root disk, or ship logs to a central log server and keep only a short history locally. Whatever you choose, write the retention down: how long application, access and security logs are kept, and where. Your security and data-protection policies may set minimums and maximums.

What you should do

• Run df -h, du on /var/log, journalctl --disk-usage and lsof +L1 on each server today.
• Add a journald drop-in with SystemMaxUse, SystemKeepFree and, if your policy needs it, MaxRetentionSec.
• Write logrotate rules for every application log that no package manages, and test them with logrotate -d.
• Cap Docker container logs with max-size and max-file.
• Alert on disk usage before it reaches 100%, and keep a documented retention period for each type of log.

Sources

• manpages.debian.org/trixie/systemd/journald.conf.5.en.html
• manpages.debian.org/trixie/systemd/journalctl.1.en.html
• manpages.debian.org/trixie/logrotate/logrotate.8.en.html
• manpages.debian.org/trixie/lsof/lsof.8.en.html
• debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html
• docs.docker.com/engine/logging/drivers/json-file