Telegraf - inputs.logparser problems - Grok no match

I’m trying to send log file to influxdb through telegraf.

I’ve found on the web tutorial with an example on site: https://www.dev-eth0.de/blog/2016/12/07/grafana_fhem_influxdb.html

I want added new lines in log file, so now it looks like this:

2018-04-16 13:19:37,225 INFO [com.micro.mapper.EJBExceptionMapper] (default task-45) Not Found! Message: Contract version on date does not exist.
2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 started.
2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 finished.
2018-04-16 13:19:39,105 WARNING [com.micro.contract.service.VersionService] (default task-53) Version for contract number: 100134 and valid on date: 2018-03-13T11:22:55.370 cannot be found. No entity found for query
2018-04-16 13:19:39,105 ERROR [org.jboss.as.ejb3.invocation] (default task-53) WFLYEJB0034: EJB Invocation failed on component VersionService for method public com.odateam.micro.contract.entity.VersionTrail com.odateam.micro.contract.service.VersionService.getVersion(java.lang.Long,java.time.LocalDateTime): javax.ejb.EJBTransactionRolledbackException: 1 constraint violation(s) occurred during method validation.
2016-12-07_21:16:38 Balkon_TX35DTH temperature: 146

Then I have added new pattern named MY_TEST, so my config file telegraf.conf looks like this

[[inputs.logparser]]

file(s) to tail:

files = [“C:\Program Files\telegraf - Copy\TestLog\test.log”]

Read file from beginning.

from_beginning = false

Override the default measurement name, which would be “logparser_grok”

name_override = “test_calculator”

For parsing logstash-style “grok” patterns:

[inputs.logparser.grok]
patterns = ["%{TEMP}", “%{HUM}”, “%{MY_TEST}”]
measurement = “fhem”

custom_patterns = ‘’’
TEMP %{DATA:timestamp:ts-“2006-01-02_15:04:05”} %{DATA:room:tag} temperature: %{NUMBER:temperature:float}
HUM %{DATA:timestamp:ts-“2006-01-02_15:04:05”} %{DATA:room:tag} humidity: %{NUMBER:humidity:float}
MY_TEST %{DATA:timestamp:ts-“2006-01-02 15:04:05,000”} %{DATA:level:tag} [%{DATA:service:tag}] (%{DATA:task:tag}) %{DATA:message:tag}
‘’’

[[outputs.influxdb]]

The full HTTP or UDP endpoint URL for your InfluxDB instance.

urls = [“http://127.0.0.1:8086”] # required

The target database for metrics (telegraf will create it if not exists).

database = “telegraf” # required

Write timeout (for the InfluxDB client), formatted as a string.

timeout = “5s”

Now it imports only last line (the one with “2016-12-07_21:16:38 Balkon_TX35DTH temperature: 146”) that is also used in tutorial described in the upper link.

And I get following telegraf.log output:
2018-04-19T11:05:50Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:00Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:10Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:19Z D! Grok no match found for: "2018-04-16 13:19:37,225 INFO [com.micro.mapper.EJBExceptionMapper] (default task-45) Not Found! Message: Contract version on date does not exist. "
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 started.”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 finished.”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:39,105 WARNING [com.micro.contract.service.VersionService] (default task-53) Version for contract number: 100134 and valid on date: 2018-03-13T11:22:55.370 cannot be found. No entity found for query”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:39,105 ERROR [org.jboss.as.ejb3.invocation] (default task-53) WFLYEJB0034: EJB Invocation failed on component VersionService for method public com.odateam.micro.contract.entity.VersionTrail com.odateam.micro.contract.service.VersionService.getVersion(java.lang.Long,java.time.LocalDateTime): javax.ejb.EJBTransactionRolledbackException: 1 constraint violation(s) occurred during method validation.”
2018-04-19T11:06:20Z D! Output [influxdb] buffer fullness: 1 / 10000 metrics.
2018-04-19T11:06:20Z D! Output [influxdb] wrote batch of 1 metrics in 2.9999ms

I’m trying to send log file to influxdb through telegraf.

I’ve found on the web tutorial with an example on site: https://www.dev-eth0.de/blog/2016/12/07/grafana_fhem_influxdb.html

I want added new lines in log file, so now it looks like this:

2018-04-16 13:19:37,225 INFO [com.micro.mapper.EJBExceptionMapper] (default task-45) Not Found! Message: Contract version on date does not exist.
2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 started.
2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 finished.
2018-04-16 13:19:39,105 WARNING [com.micro.contract.service.VersionService] (default task-53) Version for contract number: 100134 and valid on date: 2018-03-13T11:22:55.370 cannot be found. No entity found for query
2018-04-16 13:19:39,105 ERROR [org.jboss.as.ejb3.invocation] (default task-53) WFLYEJB0034: EJB Invocation failed on component VersionService for method public com.odateam.micro.contract.entity.VersionTrail com.odateam.micro.contract.service.VersionService.getVersion(java.lang.Long,java.time.LocalDateTime): javax.ejb.EJBTransactionRolledbackException: 1 constraint violation(s) occurred during method validation.
2016-12-07_21:16:38 Balkon_TX35DTH temperature: 146
Then I have added new pattern named MY_TEST, so my config file telegraf.conf looks like this

[[inputs.logparser]]

file(s) to tail:

files = [“C:\Program Files\telegraf - Copy\TestLog\test.log”]

Read file from beginning.

from_beginning = false

Override the default measurement name, which would be “logparser_grok”

name_override = “test_calculator”

For parsing logstash-style “grok” patterns:

[inputs.logparser.grok]
patterns = ["%{TEMP}", “%{HUM}”, “%{MY_TEST}”]
measurement = “fhem”

custom_patterns = ‘’’
TEMP %{DATA:timestamp:ts-“2006-01-02_15:04:05”} %{DATA:room:tag} temperature: %{NUMBER:temperature:float}
HUM %{DATA:timestamp:ts-“2006-01-02_15:04:05”} %{DATA:room:tag} humidity: %{NUMBER:humidity:float}
MY_TEST %{DATA:timestamp:ts-“2006-01-02 15:04:05,000”} %{DATA:level:tag} [%{DATA:service:tag}] (%{DATA:task:tag}) %{DATA:message:tag}
‘’’

[[outputs.influxdb]]

The full HTTP or UDP endpoint URL for your InfluxDB instance.

urls = [“http://127.0.0.1:8086”] # required

The target database for metrics (telegraf will create it if not exists).

database = “telegraf” # required

Write timeout (for the InfluxDB client), formatted as a string.

timeout = “5s”
Now it imports only last line (the one with “2016-12-07_21:16:38 Balkon_TX35DTH temperature: 146”) that is also used in tutorial described in the upper link.

And I get following telegraf.log output:

2018-04-19T11:05:50Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:00Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:10Z D! Output [influxdb] buffer fullness: 0 / 10000 metrics.
2018-04-19T11:06:19Z D! Grok no match found for: "2018-04-16 13:19:37,225 INFO [com.micro.mapper.EJBExceptionMapper] (default task-45) Not Found! Message: Contract version on date does not exist. "
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 started.”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:37,255 INFO [com.micro.contract.service.PremiumEvolutionService] (default task-57) Premium evolution for contract 100134 finished.”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:39,105 WARNING [com.micro.contract.service.VersionService] (default task-53) Version for contract number: 100134 and valid on date: 2018-03-13T11:22:55.370 cannot be found. No entity found for query”
2018-04-19T11:06:19Z D! Grok no match found for: “2018-04-16 13:19:39,105 ERROR [org.jboss.as.ejb3.invocation] (default task-53) WFLYEJB0034: EJB Invocation failed on component VersionService for method public com.odateam.micro.contract.entity.VersionTrail com.odateam.micro.contract.service.VersionService.getVersion(java.lang.Long,java.time.LocalDateTime): javax.ejb.EJBTransactionRolledbackException: 1 constraint violation(s) occurred during method validation.”
2018-04-19T11:06:20Z D! Output [influxdb] buffer fullness: 1 / 10000 metrics.
2018-04-19T11:06:20Z D! Output [influxdb] wrote batch of 1 metrics in 2.9999ms
For all other strings in my log file I get an error “Grok no match found for:”. Did I not set correctly my pattern or am I doing something else wrong.

Can anyone give me a hint please?

Thank you Damjan

One issue I think you might be running into is with the comma decimal mark in the timestamp, unfortunately there isn’t a way to specify this in the timestamp format.

These is an open feature request here with more details.

Hi Daniel,

do you know, how can i parse my log and save everything as string?

Thank you
Damjan

I don’t think it will be possible until that feature request is implemented.

Hi,

I’ve now managed to set the correct pattern for timestamp. But I still don’t know how to

  • [string1.string2.string2] into table value string1.string2.string3 I have tried with [%{WORD:myStrings}], but then I have problems with dot character “.”
  • string between ()
  • string that defines everything after the sign “)”

Can you help me with this?

Thank you and best regards
Damjan

Try using the DATA pattern here.

Hi Kai,

do you know where can I find documentation, so I will know hot to use Data, word or any other.

for example I still don’t know how to use Data in [string1.string2.string3].

Best regards
Damjan

Most of the patterns are defined here, I think you can just use [%{DATA:myStrings}].