Nomad: Docker logs growing infinitely

Created on 5 Jun 2019 · 4Comments · Source: hashicorp/nomad

Nomad version

Nomad v0.9.1 (4b2bdbd9ab68a27b10c2ee781cceaaf62e114399)

Issue

Infinitely growing Docker logs. Docker stores the logs under the dockerd --data-root. Nomad seems to be collecting them separately and also performs the nice log rotation there. However, the actual source of the logs is never cleaned. We already have a file which grew to 116GB worth of access logs in a short time, here a du -h snippet:
117G /var/lib/docker/containers/dafbe8ab0f5aec724e49d80aee5ef77824838870460a0182e76c0924b45ae4ba/dafbe8ab0f5aec724e49d80aee5ef77824838870460a0182e76c0924b45ae4ba-json.log

Reproduction steps

Any job producing logs

themdrivedocker typbug

Source

ygersie

Most helpful comment

This seems to be a side effect of the recent changes in how Nomad collects logs from the Docker driver.
It changed from syslog to json-file, without max-size/max-file configured.

For the time being I configured Docker to rotate its files in /etc/docker/daemon.json

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "2"
  }
}

I know I can pass the logging config to the driver in the job file but it would be nice to be able to do so also in the client config, under the newly introduced plugin/docker/config stanza

gpaggi on 6 Jun 2019

👍7

All 4 comments

This seems to be a side effect of the recent changes in how Nomad collects logs from the Docker driver.
It changed from syslog to json-file, without max-size/max-file configured.

For the time being I configured Docker to rotate its files in /etc/docker/daemon.json

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "2"
  }
}

I know I can pass the logging config to the driver in the job file but it would be nice to be able to do so also in the client config, under the newly introduced plugin/docker/config stanza

gpaggi on 6 Jun 2019

👍7

How to add it to a nomad task stanza:

     task "redis" {
      driver = "docker"
      config {
        image = "redis:3.2"
        port_map {
          db = 6379
        }
        logging {
          config {
            max-size= "10m",
            max-file= "10"
          }
        }
      }
....