Telegraf can't use logparser for nginx access logs in Docker container

I’m trying to get telegraf to record the nginx access logs using telegraf’s log parser and send it to influxdb.

Telegraf is definitely recording the nginx stub status and sending it to influxdb, but the access logs for the log parser aren’t being recorded.

The problem is that nginx’s official Docker image redirects /var/log/nginx/access.log to stdout, so if you exec into the container and do cat /var/log/nginx/access.log, you won’t actually see anything. You have to do docker logs <id of container> to get the access logs.

Telegraf needs access to the log files, and they don’t have a Docker log parser, so what I did is created a custom /var/log/nginx/custom/access.log in my nginx container and have the logs written there.

Then I created a shared nginxlog volume so that the telegraf container can read the logs.

I can do cat /var/log/nginx/custom/access.log from within the nginx docker container and see the logs get output, but when I do cat /var/log/nginx/custom/access.log in the telegraf container, it shows an empty file, so the file isn’t being synced somehow.

Keep in mind I needed to create an empty access.log on my host to mount it into the nginx container initially.

Here is my docker-compose.yml file:

version: "3.4"
services:
  influxdb:
    container_name: influxdb
    restart: always
    image: influxdb:1.7-alpine
    ports:
      - "8087:8086" # http
      - "8084:8083" # admin
      - "8090:8089/udp" # udp
    environment:
      INFLUXDB_REPORTING_DISABLED: "true"
      INFLUXDB_DATA_QUERY_LOG_ENABLED: "true"
      INFLUXDB_HTTP_AUTH_ENABLED: "false"
      INFLUXDB_ADMIN_USER: "admin"
      INFLUXDB_ADMIN_PASSWORD: "admin"
      INFLUXDB_UDP_ENABLED: "true"
      INFLUXDB_UDP_DATABASE: "metrics"
      INFLUXDB_RETENTION_ENABLED: "false"
    volumes:
      - ./influxdb:/var/lib/influxdb
    networks:
      log:
        ipv4_address: 172.25.0.5
  nginx:
    image: nginx:1.15-alpine
    ports:
      - "8008:80"
    volumes:
      - nginxlog:/var/log/nginx
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./index.html:/var/www/index.html
      - ./access.log:/var/log/nginx/custom/access.log
      - ./error.log:/var/log/nginx/custom/error.log
    networks:
      log:
        ipv4_address: 172.25.0.2
  telegraf:
    restart: always
    image: telegraf:1.9-alpine
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
      - nginxlog:/var/log/nginx
    networks:
      log:
        ipv4_address: 172.25.0.4
volumes:
  nginxlog:
networks:
  log:
    driver: bridge
    ipam:
      config:
        - subnet: 172.25.0.0/24

Here is my telegraf.conf file:

[agent]
  hostname = "nginx_frontend"

[[outputs.influxdb]]
  urls = ["http://172.25.0.5:8086"]
  database = "nginx_frontend"

[[inputs.nginx]]
  urls = ["http://172.25.0.2:80/nginx_status"]
  response_timeout = "5s"

[[inputs.logparser]]
   files = ["/var/log/nginx/access.log"]
   from_beginning = true
   name_override = "nginx_access_log"

  [inputs.logparser.grok]
    patterns = ["%{COMBINED_LOG_FORMAT}"]

[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"

[[outputs.file]]
  files = ["stdout", "/tmp/nginx.out"]

nginx.conf:

worker_processes auto;

events {
	worker_connections 1000;
}

http {
	server_tokens off;
	server {
		listen 80 default_server;
        index   index.html;
        root /var/www/html;
        location /nginx_status {
            stub_status on;
            access_log off;
            allow 127.0.0.1;
            allow 172.25.0.4;
            deny all;
        }
    }
}

As a workaround I ended up mounting the host’s access.log into the telegraf container instead of doing the shared volume.

I don’t like the idea of one giant nginx log file, so is there a better way to log nginx access logs from a Docker container w/o having to resort to creating a physical access log?

Hi, did you ever find a solution for this?

Hi @SiGmAX666

Are you having problems parsing nginx log files or getting logs from a Docker container?

Hi @rawkode, I am having trouble parsing logs from the Nginx Docker container. I would like to get the logs from the container without directly connecting to the access.log file, as the Docker container natively outputs that to stdout. I was unable to figure out how to do that.

My workaround for the time being was to build Nginx without the two stdout/stderr mappings, mounting /var/log/nginx to a volume, and connecting that volume to a Telegraf container.

This method is working fine but feels like a hack.

I am very open to another method or tips!

If you can hold out a little longer, Telegraf 1.12 is shipping with a Docker logs plugin

If you’re super keen, you can compile from master.

If you want to use a more generic approach, Telegraf ships a syslog compatible plugin too; and you can point the Docker Daemon logs at it.

Thanks for the reply. I compiled 1.12rc1 into a Docker and it’s running nicely. It’s pushing logs to Influxdb as expected.

Reading the documents on the new plugin, I can’t use custom patterns with it like you can with the logparser plugin, right? (inputs.logparser.gork)

You should be able to set data_format = "grok" and tweak just as any other plugin

I re-read the docs today and what you say sounds like it should work, however, I am getting an error when doing that. It seems like the data_format command is not part of the plugin? I skimmed the docker_log/client.go and syslog/syslog.go files and I wonder if the docker_log module is missing a couple default imports like fmt or io. I don’t know enough to have any useful input here sadly though.

Line 28 referenced below is the data_format = "gork" row. I should mention that it logs successfully if I exclude the data_format and gork_custom_patterns rows.

Error:
[telegraf] Error running agent: Error parsing /etc/telegraf/telegraf.conf, line 28: field corresponding to 'data_format' is not defined in docker_log.DockerLogs
[telegraf] Error running agent: Error parsing /etc/telegraf/telegraf.conf, line 29: field corresponding to 'grok_custom_patterns' is not defined in docker_log.DockerLogs

Conf:
[[inputs.docker_log]]
container_name_include = ["nginx-web"]
data_format = "gork"
grok_custom_patterns = '''
<< format pattern >>
'''

We didn’t add data_format support to this plugin, but you can use it with the parser processor:

[[processors.parser]]
  namepass = ["docker_log"]
  parse_fields = ["message"]
  grok_patterns = ["%{NUMBER:value:int}"]
  merge = "override"
  data_format = "grok"