I want to read the Telegraf logs and store them in influxdb

Telegraf
1

Hello Everyone,

I want to read the Telegraf logs and store them in influxdb to monitor the metrics write errors and other errors. (context deadline, flush interval warning, etc). I am currently using tail and grok to achieve this but it’s still not working. Kindly help me with this.

logs format:-

2022-10-06T13:20:20Z D! [inputs.disk] [SystemPS] => kept…
2022-10-06T13:20:20Z D! [inputs.disk] [SystemPS] → using mountpoint “F:”…
2022-10-06T13:20:20Z D! [inputs.disk] [SystemPS] => kept…
2022-10-06T13:20:25Z D! [outputs.influxdb_v2] Wrote batch of 467 metrics in 190.4682ms
2022-10-06T13:20:25Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

tail configuration:-

[[inputs.tail]]
files = [“C:/Users/acean/Desktop/abc.txt”]
from_beginning = false
pipe = false
data_format = “grok”
grok_patterns = [“%{DATA}”]

2

I’m using the following:

[[inputs.tail]]
  files = ['__LogFilePath__']
  character_encoding = 'utf-8'
  data_format = 'grok'
  path_tag = ''
  grok_patterns = ["%{TIMESTAMP_ISO8601:time:ts-rfc3339} %{WORD:level:tag}! %{GREEDYDATA:message}"]
  grok_timezone = "UTC"
  name_override = '__MeasurementName__'
  watch_method = 'poll'
1 Like
3

Thanks, @Giovanni_Luisotto.It’s working now.

4

@Giovanni_Luisotto just a small question, is it possible to tail only the errors and warnings? we can use the quiet option in conf file, but is it possible without that?

5

yeah, just adding


[[inputs.tail]]
  files = ['__LogFilePath__']
  character_encoding = 'utf-8'
  data_format = 'grok'
  path_tag = ''
  grok_patterns = ["%{TIMESTAMP_ISO8601:time:ts-rfc3339} %{WORD:level:tag}! %{GREEDYDATA:message}"]
  grok_timezone = "UTC"
  name_override = '__MeasurementName__'
  watch_method = 'poll'
  [inputs.tail.tagpass]
    level = [ "_ValueToKeep1_", "_ValueToKeep2_" ]

Note: I don’t remember the actual values to be filter ERR/WARN/whatever… also I’m not sure if the values are case-seisitive or not

1 Like
6

Thanks a lot @Giovanni_Luisotto

final conf-

[[inputs.tail]]
files = [‘C:/Users/acean/Desktop/logs/test.log’]
character_encoding = ‘utf-8’
data_format = ‘grok’
path_tag = ‘’
grok_patterns = [“%{TIMESTAMP_ISO8601:time:ts-rfc3339} %{WORD:level:tag}! %{GREEDYDATA:message}”]
#grok_timezone = “UTC”
name_override = ‘telegraf_logs’
watch_method = ‘poll’

[inputs.tail.tagpass]
level = [ “W”, “E” ]

1 Like