# Custom log parsing with latest Tail Plugin, GROK and InfluxDB (Grafana ready)
**URL:** https://community.influxdata.com/t/custom-log-parsing-with-latest-tail-plugin-grok-and-influxdb-grafana-ready/14986
**Category:** Telegraf
**Tags:** influxdb, telegraf
**Created:** [June 25, 2020, 8:07pm UTC](https://community.influxdata.com/t/custom-log-parsing-with-latest-tail-plugin-grok-and-influxdb-grafana-ready/14986 "2020-06-25T20:07:13Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@eclements](https://community.influxdata.com/u/eclements)
#### Post date: [June 25, 2020, 8:07pm UTC](https://community.influxdata.com/t/custom-log-parsing-with-latest-tail-plugin-grok-and-influxdb-grafana-ready/14986/1 "2020-06-25T20:07:13Z")
Influx now uses **inputs.tail** instead of **inputs.logparser**
This post shares an examples of the new **grok\_custom\_patterns** parameter.
I found mostly examples with old examples from 2017, I hope this catches the eyes of people looking for examples in 2020. I struggled to get **Telegraf inputs.tail** to work due to the changes in config style and lack of clarity in the **InfluxDB** / **Telegraf** / **Tail** / **Logparser** / **GROK** docs.
The docs specifically references the following example but without giving a full example. It also doesn’t explaining how **grok\_custom\_patterns** relates to **grok\_patterns**.
**Text book example**
> [[inputs.tail]]
> files = [“/var/log/apache/access.log”]
> from\_beginning = false
> grok\_patterns = [“%{COMBINED\_LOG\_FORMAT}”]
> name\_override = “apache\_access\_log”
> **grok\_custom\_pattern\_files = []**
> **grok\_custom\_patterns = ‘’’**
> **‘’’**
> grok\_timezone = “Canada/Eastern”
> data\_format = “grok”
Here is my explanation of latest config format:
**Example Log Line**
> RESULT,2020-06-25 15:34:34,UNXPRD01,PROD\_INSTANCE01,Running
**telegraf.conf**
```
[[inputs.tail]]
## file(s) to tail:
files = ["c:\\magicparser\\customlogs\\host01.log"]
from_beginning = false
#name of the "Metric" (which I want to see in Grafana eventually)
name_override = "magicparser"
grok_patterns = ["%{CUSTOM_LOG}"]
grok_custom_patterns = '''
MAGICDATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}
CUSTOM_LOG %{MAGICDATE:date},%{WORD:log_entry_hostname:tag},%{WORD:log_entry_service:tag},%{WORD:log_entry_state}
'''
data_format = "grok"
```
**Explanations**
1. “ **files** ” - should be clear, the file you want to monitor. Only one file per input agent.
2. “ **from\_beginning** ” - only tick if you are sure you want to process your whole log file. I typically only used this when I was testing.
3. “ **name\_override** ” - Will set the metric name. For my use case, this is the name of the Metric I will use in Grafana. Grafana has Metrics, Tags and Fields. Metric is the primary name. Tags are indexed for high performance. Fields is for often changing data.
4. “ **grok\_patterns** ” - is the required field (according to my tests) that input.tail can not work without. You have to define your main pattern here. In this case I indicate I will use a CUSTOM\_LOG pattern
5. “ **grok\_custom\_patterns** ” - This is the tricky part that the docs did not explain well. This parameter allows me to use the triple quote approach to define a series of lines of custom patterns. This value defines the details of pattern that I had listed above in “grok\_patterns”. The two work together. What is nice about GROK patterns is you can use one pattern in another. This way you break the problem up in manageable pieces. See how I defined “MAGICDATE” as my own custom pattern, and then used that in my “CUSTOM\_LOG” pattern.
6. **data\_format = “grok”** - tells the tail plugin that we are using the GROK data format.
**Tips**
1. **A note on data types**. - You will see that I didn’t just leave GROK to decide data formats for me. I overwrite the string default by appending the “tag” keyway to my element definition. This forces InfluxDB to store this field as a tag. This means the field will be indexed. Also in my use case this means I can now see the field pop up in the Grafana query builder as a WHERE parameter. Note that you should not just make all fields tags, see this post about [performance impact of tags.](https://blog.zhaw.ch/icclab/influxdb-design-guidelines-to-avoid-performance-issues/#:~:text=InfluxDB%20lets%20you%20specify%20fields,value%20of%20the%20field%20column.)
> {WORD:log\_entry\_service:tag}
1. **Use timestamp to force InfluxDB to use your logs time** - I didn’t override the Telegraf timestamp with my date. I just store my date as a string field. If I used a [timestamp pattern](https://docs.influxdata.com/telegraf/v1.14/data_formats/input/grok/), like follows, I could have told Telegraf to use my log entry’s time as its timestamp and ignore the time of reading the log file:
> %{TIMESTAMP\_ISO8601:timestamp:ts-“2006-01-02 15:04:05”}
1. **Use command line output for testing** - Don’t waste time trying to send data to InfluxDB immediately, first try out your work locally to the standard output/command line using the file output. Remember you can only have one output at a time (as far as I know), so comment out your existing output plugin before adding this:
> [[outputs.file]]
> files = [“stdout”]
> data\_format = “influx”
1. **Build and test one field at a time** - Pattern matcing is an incredibly tricking and painful exercise. It’s recommended to start with just one parameter in a test log file and then add one by one your other fields. Trust me. I read that advise another blog I can’t find now, and it was a great tip.
For example, with above log file just start with a test.log file simply with:
> RESULT
And get that to match and make sure everything is working before tackling the rest.
Good luck for your pattern matching.
---
### Author:  [@mrAbhishek](https://community.influxdata.com/u/mrAbhishek)
#### Post date: [July 1, 2020, 2:27pm UTC](https://community.influxdata.com/t/custom-log-parsing-with-latest-tail-plugin-grok-and-influxdb-grafana-ready/14986/2 "2020-07-01T14:27:51Z")
Hey @eclements,
I am using the following input conf to transfer data from log file to influxdb but it is not working:
[[inputs.tail]]
```
files = ["/var/log/testlog.log"]
from_beginning = true
grok_patterns = ["%{ERROR_LOG}", "%{CUSTOM_LOG}"]
grok_custom_patterns = '''
ERROR_LOG level=%{LOGLEVEL:severity:tag} msg="%{GREEDYDATA:value:tag}"
CUSTOM_LOG %{GREEDYDATA:value:tag}
```
‘’’
data\_format = “grok”
but if i remove the tag modifiers, it works and i can see the data in my influxdb, something like this:
[[inputs.tail]]
```
files = ["/var/log/testlog.log"]
from_beginning = true
grok_patterns = ["%{ERROR_LOG}", "%{CUSTOM_LOG}"]
grok_custom_patterns = '''
ERROR_LOG level=%{LOGLEVEL:severity} msg="%{GREEDYDATA:value}"
CUSTOM_LOG %{GREEDYDATA:value}
```
‘’’
data\_format = “grok”
Do you have any idea as to why this is happening?
I need those tags in my influxdb
---
### Author:  [@daniel](https://community.influxdata.com/u/daniel)
#### Post date: [July 1, 2020, 8:20pm UTC](https://community.influxdata.com/t/custom-log-parsing-with-latest-tail-plugin-grok-and-influxdb-grafana-ready/14986/3 "2020-07-01T20:20:13Z")
At least one value must be captured as a field.
---
### Author:  [@prashanthjbabu](https://community.influxdata.com/u/prashanthjbabu)
#### Post date: [July 2, 2020, 10:17am UTC](https://community.influxdata.com/t/custom-log-parsing-with-latest-tail-plugin-grok-and-influxdb-grafana-ready/14986/4 "2020-07-02T10:17:33Z")
@daniel would it be possible to store the original log content as well as the grok fields . Like in the above case , I’d like to store level and msg and also store “value” ( which is the original log content thrown out by tail )
---
### Author:  [@daniel](https://community.influxdata.com/u/daniel)
#### Post date: [July 2, 2020, 11:52pm UTC](https://community.influxdata.com/t/custom-log-parsing-with-latest-tail-plugin-grok-and-influxdb-grafana-ready/14986/5 "2020-07-02T23:52:38Z")
@prashanthjbabu You can, but I know about a bug you will probably run into. Let’s say you have:
```auto
grok_patterns = ["%{A:a}"]
grok_custom_patterns = '''
A %{NUMBER:x} %{NUMBER:y}"
'''
```
And a document:
```auto
1 2
3 4
```
The results would be:
```auto
> tail a="1 2",x="1",y="2"
> tail a="3 4",x="3",y="4"
```
The issue you might run into is that nested patterns with named captures at multiple levels like this don’t save the child “modifiers”, you won’t be able specify if `x` or `y` are tags or integers unless you remove the `a` capture. You can work around this for now using the `converter` processor.
---
### Author:  [@prashanthjbabu](https://community.influxdata.com/u/prashanthjbabu)
#### Post date: [July 3, 2020, 4:02am UTC](https://community.influxdata.com/t/custom-log-parsing-with-latest-tail-plugin-grok-and-influxdb-grafana-ready/14986/6 "2020-07-03T04:02:19Z")
@daniel Thanks for your reply… I also found out this method which seems to be working as well
```auto
[[processors.parser]]
parse_fields=["value"]
merge="override"
grok_patterns = ["%{ERROR_LOG}"]
grok_custom_patterns = '''
ERROR_LOG %{SYSLOGTIMESTAMP:syslog_timestamp:drop} %{SYSLOGHOST:syslog_hostname:drop} %{DATA:syslog_program:string}(?:\[%{POSINT:syslog_pid:drop}\])?: level=%{LOGLEVEL:severity:tag} msg="%{GREEDYDATA:value:drop}"
'''
data_format = "grok"
```
The processor.parser takes the “value” field generated by tail and runs grok on it and appends its fields retaining the original “value” . This should work as well right?
---
### Author:  [@eclements](https://community.influxdata.com/u/eclements)
#### Post date: [July 3, 2020, 1:13pm UTC](https://community.influxdata.com/t/custom-log-parsing-with-latest-tail-plugin-grok-and-influxdb-grafana-ready/14986/7 "2020-07-03T13:13:18Z")
Hi @mrAbhishek I found the same as what @daniel answered. If I made “all” my values tags, I would not see the data in InfluxDB. I would have to make at least one field not a tag. In other words, have to have at least one field value.
---
### Author:  [@system](https://community.influxdata.com/u/system)
#### Post date: [July 3, 2020, 2:13pm UTC](https://community.influxdata.com/t/custom-log-parsing-with-latest-tail-plugin-grok-and-influxdb-grafana-ready/14986/8 "2020-07-03T14:13:20Z")
This topic was automatically closed 60 minutes after the last reply. New replies are no longer allowed.