Hard time getting telegraf logparser automatically detect file change

, ,
1

I’m having a problem to monitor a log file using telegraf logparser input plugin.

The file I want to monitor has the following lines format:

2019-04-01 15:20:23,510 INFO my_log_message

This is my configuration for telegraf

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  debug = true
  quiet = false
  logfile = "/var/log/telegraf/telegraf.log"
  hostname = ""
  omit_hostname = false

[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]
  database = "telegraf"
  username = ""
  password = ""
  retention_policy = ""
  write_consistency = "any"
  timeout = "5s"

[[inputs.logparser]]
    files = ["/tmp/mylog.log"]
    from_beginning = true
    [inputs.logparser.grok]
        measurement = "user_log"
        patterns = ["%{LOG_PATTERN}"]
        custom_patterns = '''
LOG_PATTERN %{TIMESTAMP_ISO8601:timestamp:ts-rfc3339} %{LOGLEVEL:loglevel:tag} %{GREEDYDATA:message}
'''

This loads well the log file into the influxdb when I launch my TICK Stack in docker containers.

However, I can’t make it work to insert new entries into the influxdb whenever the log file (/tmp/mylog.log) is updated.

I can see in /var/log/telegraf/telegraf.conf that the plugin is well pushing data every 10s.

2019-04-01T13:49:19Z D! [outputs.influxdb] wrote batch of 10 metrics in 3.012963ms
2019-04-01T13:49:19Z D! [outputs.influxdb] buffer fullness: 8 / 10000 metrics.

However, if I investigate my “user_log” measurement, no new entry is added into influxdb.

I tried to monitor the telegraf.log to see if this one is noticing the updates. I added the following lines to the telegraf configuration.

[[inputs.logparser]]
  files = ["/var/log/telegraf/telegraf.log"]

  [inputs.logparser.grok]
    measurement = "telegraf_log"
    patterns = ['^%{TIMESTAMP_ISO8601:timestamp:ts-rfc3339} %{TELEGRAF_LOG_LEVEL:level:tag}! %{GREEDYDATA:msg}']
    custom_patterns = '''
TELEGRAF_LOG_LEVEL (?:[DIWE]+)
'''

This time, I can clearly see the new entries to appear every 10 seconds in the “telegraf_log” measurement.

I can’t get where is the problem in my case.

If I stop the containers, delete my “user_log” measurement in influx db and restart all the containers, the “user_log” measurement is well recreated with all the lines of the log file “/tmp/mylog.log”, so it doesn’t seem to be an issue with the parsing of the new lines.

Any idea how to progress on this issue?

Thanks in advance for your help.

1 Like
2

After further investigation, I found that if the file is modified from outside the docker container, telegraf doesn’t notify the file change. However if I update the file from inside the container, for example by pushing a new line as follow, it works. I can well see the new line in influxdb

echo '2019-04-02 11:16:42,776 INFO My dummy message ' >> mylog.log

Any idea why?

3

OK. Adding the watch_method option to “poll” in the Telegraf logparser plugin solved the issue:

# telegraf.conf
watch_method = "poll"