How to config [[inputs.socket_listener]] for a Docker container?

Telegraf
1

I can pull Docker metrics successfully using [[inputs.docker]] in a test setting - the user has access to docker group. However, I’m also interested in logging specific events from a container and this plugin only outputs overall metrics and stats.

So I looked into [[inputs.socket_listener]]. I have no idea what kind of data will come out, but I know what I’m looking for from watching sudo docker logs -f <container>. A few things came up during testing:

  1. telegraf --config <file> --test doesn’t work:

[agent] skipping plugin [[inputs.socket_listener]]: service inputs not supported in --test mode

whereas [[inputs.docker]] works just fine with --test. I suggest this information should be added to telegraf/plugins/inputs/socket_listener/README.md at master · influxdata/telegraf · GitHub

  1. Then, an output plugin is mandatory when not running in --test mode. Fine, I added

    [[outputs.file]]
 files = ["stdout"]

to the test .conf and since I don’t know what kind of data format to expect, I then added

data_format = "value"
data_type = "string"

but this is also not supported

[telegraf] Error running agent: Error parsing inputs.socket.conf, Invalid data format: value

and this error went away when I switched to data_format = "influx".
Maybe this limitation of the [[outputs.file]] plugin should also be indicated in telegraf/plugins/outputs/file/README.md at master · influxdata/telegraf · GitHub

  1. Finally I got this

$ telegraf --config inputs.socket.conf
2019-08-19T12:06:38Z I! Starting Telegraf 1.11.4
2019-08-19T12:06:38Z I! Loaded inputs: socket_listener
2019-08-19T12:06:38Z I! Loaded aggregators:
2019-08-19T12:06:38Z I! Loaded processors:
2019-08-19T12:06:38Z I! Loaded outputs: file
2019-08-19T12:06:38Z I! Tags enabled: host=hostname
2019-08-19T12:06:38Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:“hostname”, Flush Interval:10s
2019-08-19T12:06:38Z E! [agent] Service for input inputs.socket_listener failed to start: listen unix /var/run/docker.sock: bind: address already in use
2019-08-19T12:06:38Z E! [telegraf] Error running agent: listen unix /var/run/docker.sock: bind: address already in use

And immediately after I ran the [[inputs.docker]] file in --test mode and it produced and output, in spite of using the same endpoint address, “unix:///var/run/docker.sock”.

So why does it work in one situation and not the other and what can I do to fix it?
The tests where done in a VPS that already has a Telegraf systemd service running, gathering general system metrics. This service has no [[inputs.socket_listener]] or [[inputs.docker]] plugins configured. The .conf files used in the testing were in the user’s home dir, not in /etc/telegraf/telegraf.d/.

2

So I dug around a bit more and found inputs.docker_log, but hasn’t yet been released?

In any case, the first 2 points stand.

3

Yes, the docker_log input is not released yet, though you can try it out using the nightly builds and we are planning to release 1.12.0-rc1 next week.

[agent] skipping plugin [[inputs.socket_listener]]: service inputs not supported in --test mode

You can think about the inputs plugins working in one of two ways:

  1. The input can poll a source and produce metrics every interval seconds.
  2. The input can listen for events and when they occur produce a metric. This type is sometimes referred to as a “service input”.

In the current version of Telegraf, you can’t use service inputs with --test because we don’t provide any time to wait for an event to occur. In the upcoming release, this has been improved and you can ask Telegraf to pause for a number of seconds with the --test-wait option. So for example you can wait 10 seconds and then exit:

telegraf --config <file> --test --test-wait 10

[telegraf] Error running agent: Error parsing inputs.socket.conf, Invalid data format: value

Both the input plugins and output plugins use the data_format option, but they are actually referring to separate items. An input plugin can use any the input data formats while an output can use any output data format. In some cases we support both directions, but for value it can only be used in inputs.

4

Your explanations are appreciated, but what about documenting this in the readme files for each plugin? That was the point I was trying to make.

On a side note, why does trying to use socket_listener with docker.sock doesn’t work?

5

On a side note, why does trying to use socket_listener with docker.sock doesn’t work?

The protocol doesn’t match, the Docker socket is an a HTTP server expecting incoming HTTP requests. The socket_listener plugin on the other hand listens on a socket for incoming line-delimited data in the format specified by data_format. The error you see, bind: address already in use, is because two programs cannot listen on the same socket at once.

6

docker_log input is great, but it’s not there same as docker events.

If you’re looking to audit the Docker event stream, I’d recommend running docker events --format '{{json .}}' > /var/log/docker_events and parsing with input.tail

:grinning:

7

Thanks for the reply. Would it work with inputs.exec plugin running the command? Or inputs.exec can’t handle an open stream? Can’t I redirect stdout from running the command docker logs so-and-so or docker events so-and-so directly for processing (grok most likely) within Telegraf (without running the command externally and tailing a file)?

And should I switch to docker_log once it’s released, or do you anticipate the alternative method you recommend will have a better performance for a few release cycles still?

8

The problem with using a script with the exec plugin is that it expects the script to run, print, and exit. It doesn’t have support for a long running child process. The docker_log plugin should be a good permanent solution, and will be comparable to performance wise to running the docker logs -f command. The only other option I would consider is the syslog input if you are using dockers syslog logging driver.