Telegraf - inputs.logparser problems - Grok no match

Telegraf
,
1

I’m trying to send log file to influxdb through telegraf.

I’ve found on the web tutorial with an example on site: https://www.dev-eth0.de/blog/2016/12/07/grafana_fhem_influxdb.html

I want added new lines in log file, so now it looks like this:

2018-04-16 13:19:37,225 INFO [com.micro.mapper.EJBExceptionMapper] (default task-45) Not Found! Message: Contract version on date does not exist.
2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 started.
2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 finished.
2018-04-16 13:19:39,105 WARNING [com.micro.contract.service.VersionService] (default task-53) Version for contract number: 100134 and valid on date: 2018-03-13T11:22:55.370 cannot be found. No entity found for query
2018-04-16 13:19:39,105 ERROR [org.jboss.as.ejb3.invocation] (default task-53) WFLYEJB0034: EJB Invocation failed on component VersionService for method public com.odateam.micro.contract.entity.VersionTrail com.odateam.micro.contract.service.VersionService.getVersion(java.lang.Long,java.time.LocalDateTime): javax.ejb.EJBTransactionRolledbackException: 1 constraint violation(s) occurred during method validation.
2016-12-07_21:16:38 Balkon_TX35DTH temperature: 146

Then I have added new pattern named MY_TEST, so my config file telegraf.conf looks like this

[[inputs.logparser]]

file(s) to tail:

files = [“C:\Program Files\telegraf - Copy\TestLog\test.log”]

Read file from beginning.

from_beginning = false

Override the default measurement name, which would be “logparser_grok”

name_override = “test_calculator”

For parsing logstash-style “grok” patterns:

[inputs.logparser.grok]
patterns = [“%{TEMP}”, “%{HUM}”, “%{MY_TEST}”]
measurement = “fhem”

custom_patterns = ‘’’
TEMP %{DATA:timestamp:ts-“2006-01-02_15:04:05”} %{DATA:room:tag} temperature: %{NUMBER:temperature:float}
HUM %{DATA:timestamp:ts-“2006-01-02_15:04:05”} %{DATA:room:tag} humidity: %{NUMBER:humidity:float}
MY_TEST %{DATA:timestamp:ts-“2006-01-02 15:04:05,000”} %{DATA:level:tag} [%{DATA:service:tag}] (%{DATA:task:tag}) %{DATA:message:tag}
‘’’

[[outputs.influxdb]]

The full HTTP or UDP endpoint URL for your InfluxDB instance.

urls = [“http://127.0.0.1:8086”] # required

The target database for metrics (telegraf will create it if not exists).

database = “telegraf” # required

Write timeout (for the InfluxDB client), formatted as a string.

timeout = “5s”

Now it imports only last line (the one with “2016-12-07_21:16:38 Balkon_TX35DTH temperature: 146”) that is also used in tutorial described in the upper link.

And I get following telegraf.log output:
2018-04-19T11:05:50Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:00Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:10Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:19Z D! Grok no match found for: "2018-04-16 13:19:37,225 INFO [com.micro.mapper.EJBExceptionMapper] (default task-45) Not Found! Message: Contract version on date does not exist. "
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 started.”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 finished.”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:39,105 WARNING [com.micro.contract.service.VersionService] (default task-53) Version for contract number: 100134 and valid on date: 2018-03-13T11:22:55.370 cannot be found. No entity found for query”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:39,105 ERROR [org.jboss.as.ejb3.invocation] (default task-53) WFLYEJB0034: EJB Invocation failed on component VersionService for method public com.odateam.micro.contract.entity.VersionTrail com.odateam.micro.contract.service.VersionService.getVersion(java.lang.Long,java.time.LocalDateTime): javax.ejb.EJBTransactionRolledbackException: 1 constraint violation(s) occurred during method validation.”
2018-04-19T11:06:20Z D! Output [influxdb] buffer fullness: 1 / 10000 metrics.
2018-04-19T11:06:20Z D! Output [influxdb] wrote batch of 1 metrics in 2.9999ms

I’m trying to send log file to influxdb through telegraf.

I’ve found on the web tutorial with an example on site: https://www.dev-eth0.de/blog/2016/12/07/grafana_fhem_influxdb.html

I want added new lines in log file, so now it looks like this:

2018-04-16 13:19:37,225 INFO [com.micro.mapper.EJBExceptionMapper] (default task-45) Not Found! Message: Contract version on date does not exist.
2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 started.
2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 finished.
2018-04-16 13:19:39,105 WARNING [com.micro.contract.service.VersionService] (default task-53) Version for contract number: 100134 and valid on date: 2018-03-13T11:22:55.370 cannot be found. No entity found for query
2018-04-16 13:19:39,105 ERROR [org.jboss.as.ejb3.invocation] (default task-53) WFLYEJB0034: EJB Invocation failed on component VersionService for method public com.odateam.micro.contract.entity.VersionTrail com.odateam.micro.contract.service.VersionService.getVersion(java.lang.Long,java.time.LocalDateTime): javax.ejb.EJBTransactionRolledbackException: 1 constraint violation(s) occurred during method validation.
2016-12-07_21:16:38 Balkon_TX35DTH temperature: 146
Then I have added new pattern named MY_TEST, so my config file telegraf.conf looks like this

[[inputs.logparser]]

file(s) to tail:

files = [“C:\Program Files\telegraf - Copy\TestLog\test.log”]

Read file from beginning.

from_beginning = false

Override the default measurement name, which would be “logparser_grok”

name_override = “test_calculator”

For parsing logstash-style “grok” patterns:

[inputs.logparser.grok]
patterns = [“%{TEMP}”, “%{HUM}”, “%{MY_TEST}”]
measurement = “fhem”

custom_patterns = ‘’’
TEMP %{DATA:timestamp:ts-“2006-01-02_15:04:05”} %{DATA:room:tag} temperature: %{NUMBER:temperature:float}
HUM %{DATA:timestamp:ts-“2006-01-02_15:04:05”} %{DATA:room:tag} humidity: %{NUMBER:humidity:float}
MY_TEST %{DATA:timestamp:ts-“2006-01-02 15:04:05,000”} %{DATA:level:tag} [%{DATA:service:tag}] (%{DATA:task:tag}) %{DATA:message:tag}
‘’’

[[outputs.influxdb]]

The full HTTP or UDP endpoint URL for your InfluxDB instance.

urls = [“http://127.0.0.1:8086”] # required

The target database for metrics (telegraf will create it if not exists).

database = “telegraf” # required

Write timeout (for the InfluxDB client), formatted as a string.

timeout = “5s”
Now it imports only last line (the one with “2016-12-07_21:16:38 Balkon_TX35DTH temperature: 146”) that is also used in tutorial described in the upper link.

And I get following telegraf.log output:

2018-04-19T11:05:50Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:00Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:10Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:19Z D! Grok no match found for: "2018-04-16 13:19:37,225 INFO [com.micro.mapper.EJBExceptionMapper] (default task-45) Not Found! Message: Contract version on date does not exist. "
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 started.”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 finished.”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:39,105 WARNING [com.micro.contract.service.VersionService] (default task-53) Version for contract number: 100134 and valid on date: 2018-03-13T11:22:55.370 cannot be found. No entity found for query”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:39,105 ERROR [org.jboss.as.ejb3.invocation] (default task-53) WFLYEJB0034: EJB Invocation failed on component VersionService for method public com.odateam.micro.contract.entity.VersionTrail com.odateam.micro.contract.service.VersionService.getVersion(java.lang.Long,java.time.LocalDateTime): javax.ejb.EJBTransactionRolledbackException: 1 constraint violation(s) occurred during method validation.”
2018-04-19T11:06:20Z D! Output [influxdb] buffer fullness: 1 / 10000 metrics.
2018-04-19T11:06:20Z D! Output [influxdb] wrote batch of 1 metrics in 2.9999ms
For all other strings in my log file I get an error “Grok no match found for:”. Did I not set correctly my pattern or am I doing something else wrong.

Can anyone give me a hint please?

Thank you Damjan

2

One issue I think you might be running into is with the comma decimal mark in the timestamp, unfortunately there isn’t a way to specify this in the timestamp format.

These is an open feature request here with more details.

3

Hi Daniel,

do you know, how can i parse my log and save everything as string?

Thank you
Damjan

4

I don’t think it will be possible until that feature request is implemented.

5

Hi,

I’ve now managed to set the correct pattern for timestamp. But I still don’t know how to

  • [string1.string2.string2] into table value string1.string2.string3 I have tried with [%{WORD:myStrings}], but then I have problems with dot character “.”
  • string between ()
  • string that defines everything after the sign “)”

Can you help me with this?

Thank you and best regards
Damjan

6

Try using the DATA pattern here.

7

Hi Kai,

do you know where can I find documentation, so I will know hot to use Data, word or any other.

for example I still don’t know how to use Data in [string1.string2.string3].

Best regards
Damjan

8

Most of the patterns are defined here, I think you can just use [%{DATA:myStrings}].