How To Manage Docker Logfiles with Logrotate

Paul Knulst -

Introduction

Logs are important and are an essential piece of telemetry data. Logs contain all kinds of events that take place in your application and can be used to debug or troubleshoot issues. Often it is possible to identify the root cause of a problem by having good logs.

Using Docker and Docker Compose to containerize and scale applications is very easy nowadays. Unfortunately, operation complexity has increased, and frequently changing container-based environments are challenging to monitor.

Luckily Docker can display container logs that are generated through stdout and stderr with different logging drivers (local, json-file, syslog, awslog, etc). As default, Docker uses the json-file logging driver. This logging driver caches container logs and saves them to the disk.    

But before deep-diving into Docker logs, adjusting the default Docker log settings, and enabling Docker log rotation I will briefly describe what happened to me and why it could lead to big problems for you if you do not enable Logrotate for your container.

The Problem

It is 10:30 pm and I am working on my private Docker Swarm cluster. I just finished a cool new service and start to build and deploy it. I already closed every terminal, exited the IDE, and want to turn off my computer to finally relax. But, THE BUILDS ARE FAILING. Oh no...

I cannot push or pull to my self-hosted private docker registry. But the containers are still running without any problem.

I asked myself why it is not possible to read/write to my registry until I realized that the disk is full. I opened a terminal and connect to my machine where I checked how much space is left. Unfortunately, nothing...

As I did not have any clue I searched on my disk which folder is using so much space and after some time it turns out the Docker folder is big. Nearly 400GB big. Why is it so big?

The answer: logs. a LOT of logs. Really, a LOT!

Unfortunately, I was not able to simply delete the Docker folders because it could harm my services so I was forced to research Docker logs and how I am able to fix my problem.

In the end, I fixed it on the same day (But, it was really close to midnight)

A Brief On Docker Logs

In Docker are two types of log files.

  1. Docker daemon logs: Generated by the Docker daemon and saved on the host. This log file provided information about the state of the Docker platform.
  2. Docker container logs: Docker container logs contains every log entry related to any application running in a container.

Unfortunately, Docker does not have any size restriction on logging files. This means they will increase over time and consume more and more storage if left unhandled. Every service that stays online and produces a large number of log messages, will consume more disk space over time.

Where are Docker logs stored?

Normally, logs are kept on the Docker host because the containers are stateless which means they won't save data if they are rebuilt/restarted. As default, Docker uses the json-file logging driver which records all stdout/stderr output in JSON format and create a file for each container at:

/var/lib/docker/containers/[container-id]/[container-id]-JSON.log

With this information in mind, I was able to delete the biggest log files and free up 200GB of disk space. Funnily, it was only 1 container that had this amount of logs but it was running for nearly one year and I totally forgot about it.

Docker Log Rotation To The Rescue

To be able to rotate logs produced by the Docker container I first had to configure the Docker daemon to use a particular log driver. This could be done by editing the daemon.json on the Docker host.

Step 1: Switch to the Docker daemon configuration file:

  • On Linux: /etc/docker/
  • On Windows: C:\ProgramData\docker\config\daemon.json

Step 2: If no logging driver is set, use the following command to set it up:

{  "log-driver": "json-file"}

You can also use another log driver but I prefer json-file. Docker recommends local because it performs log rotation by default.

Step 3: As I choose json-file I have to add an additional configuration for log rotation:

{  
    "log-driver": "json-file",  
    "log-opts": {    
        "max-size": "15m",    
        "max-file": "5",    
    }
}

Step 4: Save the file and restart the docker daemon:

sudo systemctl restart docker

Although the daemon.json was updated and the Docker daemon was restarted it will not work for any running container because the default logging driver will only impact containers that are created after modifying it. Existing containers will still work with the initial configuration which was json-file without any log rotation!

In order to enable log rotating for existing containers, they have to be recreated. Afterward, they will use log rotation!

Configuring Log Rotation For Specific Containers

Instead of adjusting the Docker daemon's default logging driver, it is possible to configure a single Docker container to use a different logging driver and enable log rotation. This can be done while creating the container with Docker run and within a Docker Compose file.

Log Rotation With Docker Run

To run a new container with another logging driver you need to include the --log-driver flag while using docker run:

docker run -it --log-driver json-file nginx

This snippet defines json-file as the logging driver and does not configure log rotation.

To find out which logging driver is currently used by any container you can use the following command:

$ docker inspect -f '{{.HostConfig.LogConfig.Type}}' <CONTAINER>

To set up a json-file logging driver with log rotation enabled use this command:

docker run  --log-opt max-size=15m --log-opt max-file=5 nginx:latest

Simplifying the above command:

--log-opt max-size=15m: Set the maximum file size for your Docker log file.
--log-opt max-file=5: Sets the maximum amount of log files to use. If the limit is reached, Docker will delete the oldest one and starts from the beginning.
nginx:latest: The container image name

Log rotation in Docker Compose

If you are using Docker Compose to run your Docker containers you can open up any Compose file and add a new section containing the logging driver and log rotate settings:

  webserver:
    image: nginx:latest
    logging:
      driver: "json-file"
      options:
        max-file: 5
        max-size: 15m

Simplifying the Compose section:

image: nginx:latest: The container image name
driver: Sets the logging driver to use
max-size: 15m: Set the maximum file size for your Docker log file.
max-file: 5: Sets the maximum amount of log files to use. If the limit is reached, Docker will delete the oldest one and starts from the beginning.

Closing Notes

In this guide, I showed how a small incident in my Docker environment forced me to learn about Docker log rotation.

Luckily, it was very easy to set up log rotation for my Docker environment and have every new container use this approach. For existing containers, I manually had to adjust the Compose file and redeploy them as I use a Docker Swarm environment.

I hope this article gave you a quick and neat overview of how to enable log rotation in your own Docker environment.

Keep in mind that Docker container environments are highly dynamic with multiple layers of abstraction and it's hard to debug such environments. Because of this, many applications create an enormous amount of logs which play a critical role in providing information to identify issues. You will always need these logs and you should not deactivate them!

How about you? Do you use log rotation for your Docker containers? Do you have any other approach? Also, do you have any questions regarding Docker logs? I would love to hear your thoughts. Please share everything in the comments.

Feel free to connect with me on Medium, LinkedIn, Twitter, and GitHub.