Capture only certain lines of my log file

Telegraf
,
1

I want to capture only certain lines of my log file, only the ones that match a regexp. For example, I want to count the “Exception in route” lines in my application file, or I want to capture the lines related to a unique python method. Is it possible?
And even more, can I look for several expressions or pattern in the same file? And, can I configure telegraf to read different log files, each one with its pattern or set of patterns?

2

I want to capture only certain lines of my log file, only the ones that match a regexp. For example, I want to count the “Exception in route” lines in my application file, or I want to capture the lines related to a unique python method. Is it possible?

This is possible with the logparser input although you cannot yet add static fields that are not contained in the source log. This can make it a difficult to simply count lines.

can I look for several expressions or pattern in the same file?

Yes, your pattern can contain multiple patterns:

    patterns = ["%{FIRST_PATTERN}|%{SECOND_PATTERN}"]

And, can I configure telegraf to read different log files, each one with its pattern or set of patterns?

To do this you can define multiple logparser plugins:

[[inputs.logparser]]
  ... snip ...

[[inputs.logparser]]
  ... snip ...
3

Thanks Daniel for the information. Can you please check below config if it works?. And I don’t want to maintain any count on how many times it occurred. So, I don’t think I’ll use static fields. What will be use-case of static fields ?

My use-case is to send data to influx db when ever I see below pattern and alert through grafana. I’m not sure on does the output looks like for below config and how can I translate it to alert.

# Stream and parse log file(s).

[[inputs.logparser]]

## Log files to parse.

## These accept standard unix glob matching rules, but with the addition of

## ** as a “super asterisk”. ie:

## /var/log/**.log → recursively find all .log files in /var/log

## /var/log//.log → find all .log files with a parent dir in /var/log

## /var/log/apache.log → only tail the apache log file

## Read files that currently exist from the beginning. Files that are created

## while telegraf is running (and that match the “files” globs) will always

## be read from the beginning.

files = ["/etc/sv/api-wms_distribution_order_loader-v1/log/main/current"]
from_beginning = false

## Parse logstash-style “grok” patterns:

## Telegraf built-in parsing patterns: https://goo.gl/dkay10

[inputs.logparser.grok]
patterns = [“%{Exception in route}”]
measurement = “logmonitor”

4

I think your pattern will need to be changed, take a look through the docs for the grok parser. Also, at least one pattern must have a “semantic name” in order to create a metric field.

5

Hi @pavanbajaj88

Did u find a solution for this scenario, Even i am finding it difficult