Hi all,
I’m in desperate need of some help.
Architecture:
OS: Windows 10 Pro & Docker for Windows
- 18 containers, among other things, InfluxDB, Chronograf and Grafana.
Telegraf installed on Windows and reporting host and Docker metrics to InfluxDB and displayed in Grafana. All ok.
Now I’m trying to get all Docker logs to InfluxDB. Why InfluxDB and not ELK you might wonder. I am in no means an expert in programing stuff. So if I managed to get the TIG stack working and getting friendly I prefer to keep all things there.
Using Portainer I observed that all the containers logs are in the below location:
/var/lib/docker/containers/containerID/container ID-json.log
Here is my docker compose file (only Influxdb and telegraf parts)
influxdb-telegraf:
image: 'influxdb:latest'
container_name: influxdb-telegraf
hostname: influxdb-telegraf
volumes:
- '.\influxdb-telegraf:/var/lib/influxdb'
- '.\influxdb-telegraf\etc:/etc/influxdb/'
ports:
- '8086:8086'
restart: unless-stopped
labels:
- 'traefik.frontend.rule=Host:influxdb-telegraf.localhost'
- traefik.backend=influxdb-telegraf
env_file:
- .env
environment:
- 'TZ=${TZ}'
- com.ouroboros.enable=true
telegraf:
image: 'telegraf:latest'
container_name: telegraf
hostname: telegraf
volumes:
- '.\telegraf\telegraf.conf:/etc/telegraf/telegraf.conf'
- '/var/lib/docker/containers:/var/lib/docker/containers/'
- '/var/run/docker.sock:/var/run/docker.sock'
ports:
- '8092:8092/udp'
- '8094:8094'
- '8125:8125/udp'
environment:
- TZ=Europe/Bucharest
command: '--debug'
restart: unless-stopped
I tried to pass the Docker containers logs with Logparser but wasn’t able to make it work. Someody over Github suggested that I should use tail input plugin.
Could anybody who already worked this out give me a hand ?
Current telegraf log:
2019-08-16T11:46:35Z I! Starting Telegraf 1.11.3,
2019-08-16T11:46:35Z I! Using config file: /etc/telegraf/telegraf.conf,
2019-08-16T11:46:35Z I! Loaded inputs: tail,
2019-08-16T11:46:35Z I! Loaded aggregators: ,
2019-08-16T11:46:35Z I! Loaded processors: ,
2019-08-16T11:46:35Z I! Loaded outputs: influxdb,
2019-08-16T11:46:35Z I! Tags enabled: host=containers,
2019-08-16T11:46:35Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"containers", Flush Interval:10s,
2019-08-16T11:46:35Z D! [agent] Connecting outputs,
2019-08-16T11:46:35Z D! [agent] Attempting connection to output: influxdb,
2019-08-16T11:46:35Z D! [agent] Successfully connected to output: influxdb,
2019-08-16T11:46:35Z D! [agent] Starting service inputs,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/26e49e977584453b34f00abf6d72d51d4e39302a4816c65cfafc67b77b710464/26e49e977584453b34f00abf6d72d51d4e39302a4816c65cfafc67b77b710464-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/2ae752f1c69d9f854625dc3ef9df3807b51f1ed8b0b7dab26f066fb8a21690a3/2ae752f1c69d9f854625dc3ef9df3807b51f1ed8b0b7dab26f066fb8a21690a3-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/408230c390a5d80794bb224818c0a6342af07cd65b82f860031dcd3bcfe3beb5/408230c390a5d80794bb224818c0a6342af07cd65b82f860031dcd3bcfe3beb5-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/462d6822c39e3da20a725b24ad3c60ed7a2a61d37a996f9bf33431aeceada7b9/462d6822c39e3da20a725b24ad3c60ed7a2a61d37a996f9bf33431aeceada7b9-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/4d4af0ca70a15ca76dd5bf85aa2717527040b23a428c7d618867d15f903ddf62/4d4af0ca70a15ca76dd5bf85aa2717527040b23a428c7d618867d15f903ddf62-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/56778f278a66ea040f7e64d9b7cba1750779c9b325d763e6f80b1390221dd02c/56778f278a66ea040f7e64d9b7cba1750779c9b325d763e6f80b1390221dd02c-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/609abde70995a98ac3db6d0b5608b044361429ef75980c85fcd934341afc3f5e/609abde70995a98ac3db6d0b5608b044361429ef75980c85fcd934341afc3f5e-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/88fc1498b014bea7f177cb2e805328ecba0e03be4720949df3e60a93432cd3be/88fc1498b014bea7f177cb2e805328ecba0e03be4720949df3e60a93432cd3be-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/8f9435272692c01b6bb18f29394bd61b67379113f5aa5231052deff6088d8caf/8f9435272692c01b6bb18f29394bd61b67379113f5aa5231052deff6088d8caf-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/900775f80e160f4c687234d55f8ed89e4732ec9962992ee75ffc669c00d47ff5/900775f80e160f4c687234d55f8ed89e4732ec9962992ee75ffc669c00d47ff5-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/97eb33a781a81b62b3d40e1a7771a445a79ce5e3a821118e07e5c2b57fe2575e/97eb33a781a81b62b3d40e1a7771a445a79ce5e3a821118e07e5c2b57fe2575e-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/a3d1933459a1bf72f6317f96530ad82f94fcf1ebc1881d71352c4b9e98182f2d/a3d1933459a1bf72f6317f96530ad82f94fcf1ebc1881d71352c4b9e98182f2d-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/c68e485a6863f53a33379e0eee6aca55483d3dc73f5ed18d6b2d243bf646b971/c68e485a6863f53a33379e0eee6aca55483d3dc73f5ed18d6b2d243bf646b971-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/c7cf89bea2d8bc5b4ffbfc25ab4513b9e5ffb714ecb18f46b793a7bdf93fd7f5/c7cf89bea2d8bc5b4ffbfc25ab4513b9e5ffb714ecb18f46b793a7bdf93fd7f5-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/c885127c8daf936bcec87b253a9aa404500bf662ff7c50f7e6fc63f898ae55c4/c885127c8daf936bcec87b253a9aa404500bf662ff7c50f7e6fc63f898ae55c4-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/c9fb1d5a5789de8d828b8fe6b053d329ec518ee3ee8cb991a3791df995eb4bed/c9fb1d5a5789de8d828b8fe6b053d329ec518ee3ee8cb991a3791df995eb4bed-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/e1c5eb131bc514d2cd33ac3595b63caddd405f2311e005915cc3f1aadd149263/e1c5eb131bc514d2cd33ac3595b63caddd405f2311e005915cc3f1aadd149263-json.log,
2019-08-16T11:46:35Z D! [inputs.tail] tail added for file: /var/lib/docker/containers/fbe7d4a74fdfd5a80b49a13761d939b3ed71a68316c16909af2128a67fdb4ae8/fbe7d4a74fdfd5a80b49a13761d939b3ed71a68316c16909af2128a67fdb4ae8-json.log,
2019-08-16T11:46:50Z D! [outputs.influxdb] buffer fullness: 0 / 10000 metrics. ,
2019-08-16T11:47:00Z D! [outputs.influxdb] buffer fullness: 0 / 10000 metrics. ,
Thanks.