Logging and Log Rotation

Proper log management is essential for production environments to prevent disk space issues and maintain system performance. This section outlines how to configure log rotation and retention policies in your Docker Compose setup.

Log Driver Configuration

Docker provides built-in logging drivers that can be configured in your docker-compose.yml file to handle log rotation and retention. The most commonly used driver is the json-file driver with specific options for rotation.

Example Logger Configuration

version: "3.9"
services:
  virtual-signer-1:
    image: iofinnet/io-vault-virtual-signer:latest
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "10m"     # Maximum size of each log file
        max-file: "5"       # Maximum number of log files
        compress: "true"    # Enable compression of rotated files

Configuration Options Explained

• max-size: Sets the maximum size of the log before it is rotated. Values can use suffixes like "k" (kilobytes), "m" (megabytes), or "g" (gigabytes).
• max-file: Specifies the maximum number of log files that can exist. After reaching this limit, the oldest file is removed as new ones are created.
• compress: When set to "true", rotated log files are compressed to save disk space.

Setting Log Rotation for All Services

To apply consistent log rotation settings across all services in your Docker Compose file, you can define these settings at the root level:

version: "3.9"
x-logging: &default-logging
  logging:
    driver: "json-file"
    options:
      max-size: "10m"
      max-file: "5"
      compress: "true"

services:
  virtual-signer-1:
    <<: *default-logging
    image: iofinnet/io-vault-virtual-signer:latest
    # other service configs...
    
  virtual-signer-2:
    <<: *default-logging
    image: iofinnet/io-vault-virtual-signer:latest
    # other service configs...

Alternative Logging Drivers

For more advanced log management, consider using alternative logging drivers:

• local: Optimized for performance and reliability.
• json-file: Stores logs locally as JSON.
• syslog: Sends logs to a syslog server.
• journald: Integrates with systemd.
• fluentd: Streams logs to Fluentd.

Using the local Driver

The local driver is optimized for performance and reliability:

services:
  virtual-signer-1:
    image: iofinnet/io-vault-virtual-signer:latest
    logging:
      driver: "local"
      options:
        max-size: "10m"
        max-file: "5"
        compress: true

Using External Log Management Systems

For production environments, consider using dedicated log aggregation systems:

services:
  virtual-signer-1:
    image: iofinnet/io-vault-virtual-signer:latest
    logging:
      driver: "fluentd" # or "syslog", "gelf", etc.
      options:
        fluentd-address: "localhost:24224"
        tag: "docker.{{.Name}}"

Consult the Docker Documentation

Docker provides further information about logging drivers and configuring them on their docs pages.

Best Practices

1. Adjust log size and count based on disk space: For services with verbose logging, consider smaller max-size values.
2. Enable compression: This significantly reduces disk space usage for logs.
3. Monitor log usage: Regularly check log disk usage to ensure your rotation settings are appropriate.
4. Consider log level configuration: Configure your application to use appropriate log levels in production.
5. Implement centralized logging: For distributed systems, use centralized logging solutions like ELK stack or Graylog.

Testing Your Log Rotation Configuration

After setting up log rotation, verify it works correctly:

# Generate logs until rotation occurs
docker-compose exec virtual-signer-1 sh -c 'for i in {1..10000}; do echo "Log message $i" >> /proc/1/fd/1; done'

# Verify log files have been rotated
docker inspect --format='{{.LogPath}}' virtual-signer-1
ls -la $(docker inspect --format='{{.LogPath}}' virtual-signer-1).*

This configuration ensures your Docker containers won't fill up disk space with logs while maintaining accessibility to recent log data for troubleshooting purposes.

Applying to Existing Containers

In the event that you already have a Virtual Signer and wish to apply the logging settings to it, follow these steps. Log rotation settings cannot be changed for running containers, so you'll have to stop the container(s) first.
Follow these steps:

1. Update your docker-compose.yml file with the log rotation configuration

2. Recreate the containers using one of these approaches:

   Bash

   # Option 1: Recreate containers while preserving volumes
   docker-compose down && docker-compose up -d
   
   # Option 2: For minimal downtime, recreate one service at a time
   docker-compose up -d --force-recreate virtual-signer-1
   

3. Verify the new logging configuration:

   Bash

   docker inspect virtual-signer-1 | grep -A 15 LogConfig
   

Note that existing log files won't be automatically rotated. After applying these settings, rotation will only affect new logs. You may want to manually clean up existing large log files:

# Find the log file path
LOG_PATH=$(docker inspect --format='{{.LogPath}}' virtual-signer-1)

# Backup existing log (optional)
sudo cp $LOG_PATH $LOG_PATH.backup

# Truncate the log file
sudo truncate -s 0 $LOG_PATH

For production environments, schedule these changes during maintenance windows to minimize service disruption.