Skip to content

Log Rotation Configuration

Keeping container logs and system logs tidy and storage usage under control

Overview

Both Docker logs and systemd journal logs are configured with automatic rotation to prevent unlimited disk usage. Different containers have different log limits based on their verbosity.

Current Configuration

System Logs (journald)

File: /etc/systemd/journald.conf

[Journal]
SystemMaxUse=500M
  • Max storage: 500 MB for all journal logs
  • Retention: Automatic cleanup when SystemMaxUse limit is hit (nota: SystemKeepFree NO está configurado)

Apply changes: 

sudo systemctl restart systemd-journald

Manual cleanup if needed: 

# Clean logs older than 7 days
sudo journalctl --vacuum-time=7d

# Limit to 500M
sudo journalctl --vacuum-size=500M

Docker Logs

Global Default (All Containers)

File: /etc/docker/daemon.json

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
  • Max size per file: 10 MB
  • Max files: 3 (rotates when full)
  • Total per container: 30 MB max

Noisy Containers (Reduced Limits)

The following containers have stricter limits (5MB x 2 files = 10MB total) to reduce storage:

Infra Stack: - loki - 5MB x 2 files (was generating 13k lines/hour) - promtail - 5MB x 2 files (was generating 1.1k lines/hour)

Media Stack: - plundrio - 5MB x 2 files (was generating 4.5k lines/hour)

Tools Stack: - changedetection - 5MB x 2 files (was generating 1.7k lines/hour)

Storage Estimates

Before Optimization

  • Per container default: 30 MB max
  • Noisy containers (4): 120 MB total
  • Other containers (~50): 1.5 GB max

Total potential: ~1.6 GB for Docker logs

After Optimization

  • Noisy containers (4): 40 MB total (saved 80 MB)
  • Other containers (~50): 1.5 GB max

Total potential: ~1.5 GB for Docker logs

Loki Storage

  • Current: 4.4 GB (medido 2026-07-12; retención 720h/30d domina)
  • Projected: ~1.2 GB/month
  • Retention: 30 days (in Loki config, loki-config.yaml)

Combined Docker + Loki: Loki ~4.4 GB por sí solo ya supera la estimación previa de ~2.7 GB

Why These Limits?

Noisy Container Justification

Loki: The log collector itself was the most verbose container. Since all logs are stored in /home/monxas/appdata/loki/, we don't need large Docker logs for Loki itself.

Promtail: Similarly, Promtail's Docker logs are less important since all container logs are already being forwarded to Loki.

Plundrio: Generates lots of download/transfer status logs. Important events are also sent to ntfy notifications, so we don't need extensive log history in Docker.

Changedetection: Verbose website monitoring logs. Alerts are sent via ntfy, so Docker logs are mainly for debugging.

Log Access

Via Docker

# Recent logs
sudo docker logs <container> --tail 100

# Last hour
sudo docker logs <container> --since 1h

# Follow live
sudo docker logs <container> -f

# Query specific container
curl -G -s "http://localhost:3101/loki/api/v1/query_range" \
  --data-urlencode 'query={container_name="jellyfin"}' \
  --data-urlencode 'limit=100'

# Or use Grafana Explore
http://192.168.0.208:3030

Monitoring Log Usage

Check Docker Log Sizes

# All container logs
sudo du -sh /var/lib/docker/containers/*/

# Largest logs
sudo du -h /var/lib/docker/containers/*/*.log | sort -h | tail -10

# Specific container
sudo docker inspect --format='{{.LogPath}}' <container> | xargs ls -lh

Check Loki Storage

# Loki data directory
du -sh /home/monxas/appdata/loki/

# Breakdown by subdirectory
du -sh /home/monxas/appdata/loki/*

Modifying Limits

Change Limits for a Specific Container

Edit the compose file and add/modify:

services:
  mycontainer:
    image: myimage:latest
    # ... other config ...
    logging:
      driver: "json-file"
      options:
        max-size: "5m"   # 5 MB per file
        max-file: "2"    # 2 files (10 MB total)

Then recreate the container: 

cd /home/monxas/stacks/<stack-name>
sudo docker compose up -d <container-name>

Change Global Default

  1. Edit /etc/docker/daemon.json
  2. Restart Docker daemon: 
    sudo systemctl restart docker
  3. Recreate all containers to apply new limits

⚠️ Warning: Restarting Docker daemon will briefly stop all containers!

Log Rotation Behavior

How It Works

  1. Container writes to Docker log file
  2. When file reaches max-size, Docker creates a new file
  3. Old files are renamed with .1, .2, etc.
  4. When max-file limit is reached, oldest file is deleted
  5. Process repeats

Example Timeline

For a container with max-size=5m and max-file=2:

Time 0:     container.log (0 MB)
Time 1:     container.log (5 MB) → rotates
Time 2:     container.log (0 MB), container.log.1 (5 MB)
Time 3:     container.log (5 MB), container.log.1 (5 MB) → rotates
Time 4:     container.log (0 MB), container.log.1 (5 MB) → old .1 becomes .2, gets deleted

Best Practices

  1. Use Loki for historical logs - Docker logs are for recent/debugging
  2. Set appropriate limits - Noisy containers should have smaller limits
  3. Important logs via ntfy - Critical events should trigger notifications
  4. Monitor Loki storage - Check /home/monxas/appdata/loki/ monthly
  5. Don't reduce too much - Keep at least 10 MB per container for debugging

Troubleshooting

Container logs disappeared

# Check if logs were rotated
sudo ls -lh /var/lib/docker/containers/<container-id>/

# Check Loki for historical logs
{container_name="mycontainer"}

Loki storage growing too fast

# Check which containers are noisiest
for c in $(sudo docker ps --format "{{.Names}}"); do
  echo "$c: $(sudo docker logs $c --since 1h 2>&1 | wc -l) lines/hour"
done | sort -rn -k2

# Reduce logging for noisy containers
# Add logging limits in their compose files

Out of disk space

# Check Docker log usage
sudo du -sh /var/lib/docker/containers/

# Check Loki usage
du -sh /home/monxas/appdata/loki/

# Emergency cleanup (be careful!)
sudo docker system prune -a  # Removes unused images/containers

Files Modified