Telegraf tagdrop does not work

1

Hi all,

I am using telegraf version 1.14.3 and I am trying to filter out records with certain IP addresses from our apache logfile and write the records into a test-file. The relevant part of the telegraf.conf looks as follows:

[agent]
  debug = false
  logtarget = "file"
  logfile = "/var/log/telegraf/telegraf.log"

[[inputs.logparser]]
  files = ["/var/log/apache2/access.log"]
  from_beginning = false
  name_override = "downloads"
  [inputs.logparser.grok]
    patterns = ["%{CUSTOM_LOG}"]
    custom_patterns = '''
      CUSTOM_LOG %{IPORHOST:ip} %{USER} %{USER} \[%{HTTPDATE:time:ts-"02/Jan/2006:15:04:05 -0700"}\] "(%{WORD:verb} (HTTP/%{DATA}))" %{NUMBER:response:int} %{DATA:connstatus} (%{NUMBER:bytes:int}|-) %{QS} %{QS:agent} %{NUMBER:timeinms:int} %{NUMBER:totalbytes:int}'''
  [inputs.logparser.tagdrop]
    ip = ["1.2.3.4"]

[[outputs.file]]
  files = ["/tmp/telegraf/output.log"]

The problem is that tagdrop seems not to work as expected. When a record with IP address 1.2.3.4 is added to apache log file, this record will also be included in the output file.

Is this a bug in telegraf or an error in my config?

Many thanks for your help.

2

I believe this is because ip is being saved as a string field, not as a tag. To collect it as a tag you need to add the pattern with :tag suffix: %{IPORHOST:ip:tag}

3

Thank you for your reply.
It looks like adding the :tag suffix solved the problem. I was not conscious about the fact that the :tag suffix has to be added.
From the documentation about configuring telegraf (Configuring Telegraf | Telegraf 1.14 Documentation) it is still not clear for me that my configuration was wrong/incomplete.
Anyway, as it took me many hours to investigate the problem I will not forget that anymore.

4

Sorry about the confusion, there is definitely room for improvements throughout the Telegraf documentation and this is one of the trickiest plugins to configure. For setting up these patterns you will want to reference the grok parser.

P.S. In the upcoming 1.15 release of Telegraf we are also going to deprecate the logparser plugin, don’t worry it is an easy switch to using the tail plugin with data_format=grok, just a few options need renamed:

[[inputs.tail]]
  files = ["/var/log/apache2/access.log"]
  from_beginning = false
  name_override = "downloads"
  data_format = "grok"
  grok_patterns = ["%{CUSTOM_LOG}"]
  grok_custom_patterns = '''
      CUSTOM_LOG %{IPORHOST:ip} %{USER} %{USER} \[%{HTTPDATE:time:ts-"02/Jan/2006:15:04:05 -0700"}\] "(%{WORD:verb} (HTTP/%{DATA}))" %{NUMBER:response:int} %{DATA:connstatus} (%{NUMBER:bytes:int}|-) %{QS} %{QS:agent} %{NUMBER:timeinms:int} %{NUMBER:totalbytes:int}'''
  [inputs.tail.tagdrop]
    ip = ["1.2.3.4"]