Parse a custom log using telegraf logparser input

Telegraf
1

Hi,

I have a custom log file with entries as given below
How to write a grok pattern to match this and output it to influxdb. I have been trying for a long time with no success.

[Tue, 18 Jul 2017 02:39:45 GMT]  GET  200  /lectures/questions/150  2032  47.8.10.6 [17.427 ms] phone
[Tue, 18 Jul 2017 02:39:45 GMT]  GET  200  /forums/fetch/lectures/31  2032  47.8.10.6 [8.398 ms] phone
[Tue, 18 Jul 2017 02:39:45 GMT]  GET  304  /users/phone/2032  2032  47.8.10.6 [6.613 ms] phone
[Tue, 18 Jul 2017 02:40:06 GMT]  GET  200  /lectures/summary  2032  47.8.10.6 [12.817 ms] phone
2

I wrote a few tips in the logparser readme, have you seen these? I highly suggest working on your pattern one token at a time.

3

Thanks for the reply. I followed it and was able to find the pattern of all except timestamp.

%{WORD:method} %{NUMBER:resp} %{URIPATHPARAM:request} %{NUMBER:userid} %{IPORHOST:clientip} [%{NUMBER:resptime}%{SPACE}ms] %{WORD:device}

For the time I tried %{DATESTAMP_RFC2822: timestamp} but it says not matched.
Can I get some help here

4

Ok, I got the pattern. Thanks for the help.
If any body needs it the pattern to match time stamp is

^[(?%{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} GMT)]

5

Hi,

Here is an example of my log:

test.co:443 80.81.174.142 www.test.co [07/Nov/2017:09:53:34 +0200] 10524941 “GET /api/v1/company/jobs HTTP/1.1” 200 34653 “https://www.test.co/v/hunt” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36” WgFmfn8AAAEAAF7ZVwUAAAAB ad3doh5o6zpyk5zubtrck8ta2p1tz187

I added this to my telegraf.conf:

[[inputs.logparser]]

files to tail.

files = [“/var/log/apache2/other_vhosts_access.log”]

Read file from beginning.

from_beginning = false

Override the default measurement name, which would be “logparser_grok”

name_override = “apache_access_log”

For parsing logstash-style “grok” patterns:

[inputs.logparser.grok]
patterns = [“%{CUSTOM_LOG}”]
custom_patterns = ‘’’
CUSTOM_LOG %{DATA:host} %{IP:client} %{DATA} [%{HTTPDATE:ts:ts-httpd}] %{NUMBER:resp_time:tag} "(?:%{WORD:verb:tag} %{NOTSPACE:r$
‘’’

I tried using http://grokdebug.herokuapp.com/ to make sure my custom pattern works, and it looks like it does, but no entries were created in influx DB :frowning:

Tested with other patterns and log examples and it worked, so I guess I’m missing something in the [[inputs.logparser]].

Thanks a lot!

6

The examples in this post is now slightly outdated. For the new " grok_custom_patterns" format, see this update I wrote after not getting above 2017 examples to work in 2020 Telegraf: Custom log parsing with latest Tail Plugin, GROK and InfluxDB configuration

7

Hi I want to filter all the logs where the status code is above 500

Please suggest me either pattern or regex to achieve this