Master Docker Log Rotation

Docker log rotation isn’t enabled by default—I learned this the hard way. At 3 AM on a Tuesday, my production server ran out of disk space, bringing down every containerized service. The culprit? A single container’s logs had grown to 47 GB over three months.

Without proper log rotation, Docker appends every stdout and stderr line from your containers to JSON files in /var/lib/docker/containers/. These files grow indefinitely until your disk fills up, causing production outages.

Understand Docker Default Logging

When you run a container, Docker captures all output using the json-file logging driver. Each container gets its own log file stored at:

/var/lib/docker/containers/<container-id>/<container-id>-json.log

These files persist even after containers stop. I’ve seen production systems where stopped containers from six months ago still consumed gigabytes of disk space.

In my experience, a moderately verbose application can generate 100-200 MB of logs per day. Multiply that by dozens of containers, and you’re looking at serious storage problems within weeks.

Configure Global Log Rotation

The most reliable approach is configuring log rotation in Docker’s daemon configuration. I set this on every production host:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

This configuration limits each container to three log files of 10 MB each, with automatic compression. Total maximum disk usage per container: 30 MB.

After editing /etc/docker/daemon.json, restart the Docker daemon:

sudo systemctl restart docker

Important: This only affects new containers. Existing containers continue using their original logging configuration until you recreate them.

Optimize Per-Container Logging

Sometimes you need different log retention for specific containers. I configure this directly in Docker Compose:

version: '3.8'
services:
  web:
    image: nginx:latest
    logging:
      driver: "json-file"
      options:
        max-size: "5m"
        max-file: "5"
        compress: "true"
  
  database:
    image: postgres:14
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "10"
        compress: "true"

My web servers generate minimal logs, so 5 MB files work fine. Database containers produce more verbose output, warranting larger files and more retention.

For critical services where I need comprehensive log history, I increase max-file to 20 or even 50, depending on disk capacity and log volume.

Choose the Right Logging Driver

The json-file driver isn’t your only option. Docker supports several logging drivers, each with different tradeoffs.

Local Logging Driver

The local driver provides better performance and automatic rotation:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5",
    "compress": "true"
  }
}

I’ve measured 20-30% better write performance compared to json-file on high-throughput applications. The local driver uses a more efficient binary format and handles rotation more gracefully.

Syslog Integration

For centralized logging infrastructure, the syslog driver sends logs directly to your logging server:

services:
  app:
    image: myapp:latest
    logging:
      driver: "syslog"
      options:
        syslog-address: "tcp://logs.example.com:514"
        tag: "{{.Name}}/{{.ID}}"

This approach completely eliminates local log storage. I use this pattern when integrating with systems like Loki, Elasticsearch, or Splunk.

Monitor Docker Log Disk Usage

Configuration alone isn’t enough. I monitor log disk usage actively to catch problems before they become incidents.

Here’s a Python script I run via cron every hour to check Docker log directory size:

#!/usr/bin/env python3
import os
import json
from pathlib import Path

def get_docker_log_size():
    """Calculate total size of Docker container logs."""
    log_dir = Path("/var/lib/docker/containers")
    total_size = 0
    container_logs = []
    
    for container_dir in log_dir.iterdir():
        if container_dir.is_dir():
            log_file = container_dir / f"{container_dir.name}-json.log"
            if log_file.exists():
                size_mb = log_file.stat().st_size / (1024 * 1024)
                container_logs.append({
                    "container": container_dir.name[:12],
                    "size_mb": round(size_mb, 2)
                })
                total_size += size_mb
    
    # Sort by size, largest first
    container_logs.sort(key=lambda x: x["size_mb"], reverse=True)
    
    print(f"Total Docker logs: {round(total_size, 2)} MB")
    print(f"\nTop 5 containers by log size:")
    for log in container_logs[:5]:
        print(f"  {log['container']}: {log['size_mb']} MB")
    
    # Alert if total exceeds 5 GB
    if total_size > 5120:
        print(f"\nWARNING: Docker logs exceed 5 GB!")
        return 1
    
    return 0

if __name__ == "__main__":
    exit(get_docker_log_size())

This script alerts me when total log size exceeds 5 GB, giving me time to investigate before hitting disk limits.

Clean Up Existing Large Logs

When you first implement log rotation, you’ll likely find containers with massive existing log files. Docker won’t automatically clean these up.

I use this command to identify problematic containers:

docker ps -a --format "{{.ID}}" | while read container; do
    size=$(docker inspect --format='{{.LogPath}}' $container | xargs du -h 2>/dev/null | cut -f1)
    name=$(docker inspect --format='{{.Name}}' $container | sed 's/\///')
    echo "$size $name"
done | sort -hr | head -10

For containers with excessive logs, I truncate them manually:

truncate -s 0 $(docker inspect --format='{{.LogPath}}' <container-id>)

Better yet, recreate the container with proper log rotation configured. The recreated container automatically picks up your daemon-level or compose-level logging configuration.

Apply Production Best Practices

After managing Docker logging across hundreds of production containers, these patterns have proven most reliable:

Set global defaults conservatively. I configure daemon-level rotation with max-size: 10m and max-file: 3 on all hosts. This catches containers created outside compose files or automation.

Override for specific needs. Services with legitimate high log volume get per-container configuration in compose files. I’ve rarely needed more than 500 MB total log retention per container.

Enable compression. The compress: true option typically saves 60-80% disk space with negligible CPU overhead. I enable it everywhere.

Monitor proactively. Disk space alerts should trigger at 80% capacity, but I monitor Docker log sizes separately to catch runaway logging before it affects the filesystem.

Centralize when possible. For production systems, I prefer sending logs to external aggregation (Loki, Elasticsearch) and keeping minimal local retention. This separates log storage from application infrastructure.

Document your choices. I maintain a simple table in our infrastructure docs showing each service’s logging configuration and rationale. This prevents confusion when someone encounters a container with non-standard settings.

Manage Log Rotation During Container Lifecycle

One subtle gotcha: Docker’s log rotation only applies while a container runs. If you stop a container, its logs persist indefinitely.

I’ve seen stopped containers from failed deployments consume hundreds of gigabytes across a cluster. My practice is removing stopped containers regularly:

docker container prune -f --filter "until=168h"

This removes containers stopped for more than seven days. Adjust the timeframe based on your debugging and rollback needs.

Solve Issues Beyond Log Rotation

Sometimes the problem isn’t rotation configuration—it’s excessive logging. I’ve debugged applications logging every request parameter at INFO level, generating gigabytes daily.

The fix isn’t larger log files. It’s reducing log verbosity or filtering at the application level. Docker log rotation is a safety net, not a solution for poorly configured logging.

I also use the --log-opt mode=non-blocking option for high-throughput applications. This prevents log writes from blocking the application if the logging driver falls behind:

services:
  high-throughput-api:
    image: myapi:latest
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "5"
        mode: "non-blocking"
        max-buffer-size: "4m"

Non-blocking mode can drop logs under extreme pressure, but it prevents logging from becoming a bottleneck.

Conclusion

Docker log rotation should be configured before deploying to production, not after a disk space incident. The daemon-level configuration provides a safety net, while per-container settings handle special cases.

My standard approach: global rotation via daemon.json, per-service overrides in compose files, compression enabled everywhere, and proactive monitoring. This combination has prevented log-related incidents across diverse production environments.

The 3 AM outage taught me that defaults aren’t always safe. Implementing Docker log rotation takes fifteen minutes but prevents hours of incident response. Don’t wait for a production outage to configure your logging strategy.