Input tail and file problem and test

Telegraf
, ,
1

Hello

Im configuring a input.file with telegraf

im reading for a file.log

the problem is the text have lines since 2012 to 2024. only 1 line per day

i send the data to influx using grok

is there some way to see for example lines from 2024 only? or the last 10 lines for example…

if i use input.file it sends all the lines to influx but only i want the last so i think is better input.tail but is difficult to test if is work with a “telegraf --test --debug” because as i say file is updated each 24h aprox … i would like test with last lines to test if input.line it works ok as input.file does


[[inputs.file]]
  files = ["C:\\items\\logs\\events.log"]
  data_format = "grok"
  grok_patterns = [
    '^<%{NUMBER}>%{NUMBER} %{TIMESTAMP_ISO8601:timestamp} %{WORD:server_name:tag}  \[threshold@%{NUMBER:threshold_value} name="%{DATA:metric_name:tag}" value="%{NUMBER:metric_value}(?:K)?" threshold="%{DATA:metric_threshold}"\] %{GREEDYDATA:rest_of_line}'

]
  grok_timezone = "UTC"
  name_override = 'events_log'

[[processors.defaults]]
  [processors.defaults.fields]
    metric_threshold = "N/A"

with this conf seems to work good, telegraf --test shows me info

event_log,metric_name=yuth.sr,metric_threshold=“50”,metric_value=“58.0”,rest_of_line=more data info",threshold_value=“11453”,timestamp=“2024-03-06T03:06:14Z” 1709722675000000000

if use this conf

[[inputs.tail]]
  files = ["C:\\items\\logs\\events.log"]
  from_beginning = false
  watch_method = "poll"
  data_format = "grok"
  grok_patterns = [
    '^<%{NUMBER}>%{NUMBER} %{TIMESTAMP_ISO8601:timestamp} %{WORD:server_name:tag}  \[threshold@%{NUMBER:threshold_value} name="%{DATA:metric_name:tag}" value="%{NUMBER:metric_value}(?:K)?" threshold="%{DATA:metric_threshold}"\] %{GREEDYDATA:rest_of_line}'

]
  grok_timezone = "UTC"
  name_override = 'events_log'

[[processors.defaults]]
  [processors.defaults.fields]
    metric_threshold = "N/A"

telegraf --test show me that

C:\Telegraf>telegraf --test --debug --config telegraf_agent.d\event_log.conf

2024-03-06T11:52:52Z I! Starting Telegraf 1.21.4

2024-03-06T11:52:52Z I! Loaded inputs: tail

2024-03-06T11:52:52Z I! Loaded aggregators:

2024-03-06T11:52:52Z I! Loaded processors: defaults

2024-03-06T11:52:52Z W! e[31mOutputs are not used in testing mode!e[0m

2024-03-06T11:52:52Z I! Tags enabled: host=ri3343

2024-03-06T11:52:52Z D! [agent] Initializing plugins

2024-03-06T11:52:52Z D! [agent] Starting service inputs

2024-03-06T11:52:52Z D! [inputs.tail] Tail added for “C:\items\logs\events.log”

2024-03-06T11:52:52Z D! [agent] Stopping service inputs

2024-03-06T11:52:52Z D! [inputs.tail] Recording offset 15615917 for “C:\items\logs\events.log”

2024-03-06T11:52:52Z D! [inputs.tail] Tail removed for “C:\items\logs\events.log”

2024-03-06T11:52:52Z D! [agent] Input channel closed

2024-03-06T11:52:52Z D! [agent] Processor channel closed

2024-03-06T11:52:52Z D! [agent] Stopped Successfully

2

@Artax1442 well you do have from_beginning = false so tail will start at the end of the file. Furthermore, tail is a “service plugin” and those don’t work well with --test as the event needs to happen in the tiny fragment of time the plugin actually runs. You should use --test-wait 120 for example to wait 2 minutes for a new line to appear in the file.