Docker containers logs to Influxdb

InfluxDB 2
1

Hi all,

I’m in desperate need of some help.

Architecture:

OS: Windows 10 Pro & Docker for Windows

  • 18 containers, among other things, InfluxDB, Chronograf and Grafana.
    Telegraf installed on Windows and reporting host and Docker metrics to InfluxDB and displayed in Grafana. All ok.

Now I’m trying to get all Docker logs to InfluxDB. Why InfluxDB and not ELK you might wonder. I am in no means an expert in programing stuff. So if I managed to get the TIG stack working and getting friendly I prefer to keep all things there.

Using Portainer I observed that all the containers logs are in the below location:
/var/lib/docker/containers/containerID/container ID-json.log

Here is my docker compose file (only Influxdb and telegraf parts)

influxdb-telegraf:
    image: 'influxdb:latest'
    container_name: influxdb-telegraf
    hostname: influxdb-telegraf
    volumes:
      - '.\influxdb-telegraf:/var/lib/influxdb'
      - '.\influxdb-telegraf\etc:/etc/influxdb/'
    ports:
      - '8086:8086'
    restart: unless-stopped
    labels:
      - 'traefik.frontend.rule=Host:influxdb-telegraf.localhost'
      - traefik.backend=influxdb-telegraf
    env_file:
      - .env
    environment:
      - 'TZ=${TZ}'
      - com.ouroboros.enable=true
  telegraf:
    image: 'telegraf:latest'
    container_name: telegraf
    hostname: telegraf
    volumes:
      - '.\telegraf\telegraf.conf:/etc/telegraf/telegraf.conf'
      - '/var/lib/docker/containers:/var/lib/docker/containers/'
      - '/var/run/docker.sock:/var/run/docker.sock'
    ports:
      - '8092:8092/udp'
      - '8094:8094'
      - '8125:8125/udp'
    environment:
      - TZ=Europe/Bucharest
    command: '--debug'
    restart: unless-stopped

I tried to pass the Docker containers logs with Logparser but wasn’t able to make it work. Someody over Github suggested that I should use tail input plugin.

Could anybody who already worked this out give me a hand ?

Current telegraf log:

2019-08-16T11:46:35Z I! Starting Telegraf 1.11.3,
2019-08-16T11:46:35Z I! Using config file: /etc/telegraf/telegraf.conf,
2019-08-16T11:46:35Z I! Loaded inputs: tail,
2019-08-16T11:46:35Z I! Loaded aggregators: ,
2019-08-16T11:46:35Z I! Loaded processors: ,
2019-08-16T11:46:35Z I! Loaded outputs: influxdb,
2019-08-16T11:46:35Z I! Tags enabled: host=containers,
2019-08-16T11:46:35Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"containers", Flush Interval:10s,
2019-08-16T11:46:35Z D! [agent] Connecting outputs,
2019-08-16T11:46:35Z D! [agent] Attempting connection to output: influxdb,
2019-08-16T11:46:35Z D! [agent] Successfully connected to output: influxdb,
2019-08-16T11:46:35Z D! [agent] Starting service inputs,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/26e49e977584453b34f00abf6d72d51d4e39302a4816c65cfafc67b77b710464/26e49e977584453b34f00abf6d72d51d4e39302a4816c65cfafc67b77b710464-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/2ae752f1c69d9f854625dc3ef9df3807b51f1ed8b0b7dab26f066fb8a21690a3/2ae752f1c69d9f854625dc3ef9df3807b51f1ed8b0b7dab26f066fb8a21690a3-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/408230c390a5d80794bb224818c0a6342af07cd65b82f860031dcd3bcfe3beb5/408230c390a5d80794bb224818c0a6342af07cd65b82f860031dcd3bcfe3beb5-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/462d6822c39e3da20a725b24ad3c60ed7a2a61d37a996f9bf33431aeceada7b9/462d6822c39e3da20a725b24ad3c60ed7a2a61d37a996f9bf33431aeceada7b9-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/4d4af0ca70a15ca76dd5bf85aa2717527040b23a428c7d618867d15f903ddf62/4d4af0ca70a15ca76dd5bf85aa2717527040b23a428c7d618867d15f903ddf62-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/56778f278a66ea040f7e64d9b7cba1750779c9b325d763e6f80b1390221dd02c/56778f278a66ea040f7e64d9b7cba1750779c9b325d763e6f80b1390221dd02c-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/609abde70995a98ac3db6d0b5608b044361429ef75980c85fcd934341afc3f5e/609abde70995a98ac3db6d0b5608b044361429ef75980c85fcd934341afc3f5e-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/88fc1498b014bea7f177cb2e805328ecba0e03be4720949df3e60a93432cd3be/88fc1498b014bea7f177cb2e805328ecba0e03be4720949df3e60a93432cd3be-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/8f9435272692c01b6bb18f29394bd61b67379113f5aa5231052deff6088d8caf/8f9435272692c01b6bb18f29394bd61b67379113f5aa5231052deff6088d8caf-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/900775f80e160f4c687234d55f8ed89e4732ec9962992ee75ffc669c00d47ff5/900775f80e160f4c687234d55f8ed89e4732ec9962992ee75ffc669c00d47ff5-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/97eb33a781a81b62b3d40e1a7771a445a79ce5e3a821118e07e5c2b57fe2575e/97eb33a781a81b62b3d40e1a7771a445a79ce5e3a821118e07e5c2b57fe2575e-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/a3d1933459a1bf72f6317f96530ad82f94fcf1ebc1881d71352c4b9e98182f2d/a3d1933459a1bf72f6317f96530ad82f94fcf1ebc1881d71352c4b9e98182f2d-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/c68e485a6863f53a33379e0eee6aca55483d3dc73f5ed18d6b2d243bf646b971/c68e485a6863f53a33379e0eee6aca55483d3dc73f5ed18d6b2d243bf646b971-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/c7cf89bea2d8bc5b4ffbfc25ab4513b9e5ffb714ecb18f46b793a7bdf93fd7f5/c7cf89bea2d8bc5b4ffbfc25ab4513b9e5ffb714ecb18f46b793a7bdf93fd7f5-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/c885127c8daf936bcec87b253a9aa404500bf662ff7c50f7e6fc63f898ae55c4/c885127c8daf936bcec87b253a9aa404500bf662ff7c50f7e6fc63f898ae55c4-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/c9fb1d5a5789de8d828b8fe6b053d329ec518ee3ee8cb991a3791df995eb4bed/c9fb1d5a5789de8d828b8fe6b053d329ec518ee3ee8cb991a3791df995eb4bed-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/e1c5eb131bc514d2cd33ac3595b63caddd405f2311e005915cc3f1aadd149263/e1c5eb131bc514d2cd33ac3595b63caddd405f2311e005915cc3f1aadd149263-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/fbe7d4a74fdfd5a80b49a13761d939b3ed71a68316c16909af2128a67fdb4ae8/fbe7d4a74fdfd5a80b49a13761d939b3ed71a68316c16909af2128a67fdb4ae8-json.log,
2019-08-16T11:46:50Z D! [outputs.influxdb] buffer fullness: 0 / 10000 metrics. ,
2019-08-16T11:47:00Z D! [outputs.influxdb] buffer fullness: 0 / 10000 metrics. ,

Thanks.

2

In the end I just gave up on using TIG stack for the moment and using some form of EFK (ElasticSearch, Filebeat and Kibana).