Telegraf tail input parsing using GROK - Syntax help

Telegraf
, ,
1

Hi,
I am trying to get Telegraf (1.22.3 on Windows) to tail a game server log and extract the timestamp and on the same line entry parse out the tick rate (performance of the game server), example of log below:

[2022.05.14-00.37.13:544][397]LogSquad: USQGameState: Server Tick Rate: 42.67

So having used the heroku grok debugger, I built a custom pattern that parses the information I need:

\[(?<time_stamp>[\d\.\-\s\.\:]+).*Rate\:\s(?<TPS>\d\d\.\d\d)

which gives me the following:

{
  "time_stamp": [
    [
      "2022.05.14-00.37.13:544"
    ]
  ],
  "TPS": [
    [
      "42.67"
    ]
  ]
}

However I have no idea how to convert the custom pattern to a GROK pattern that telegraf can understand, and having wasted many hours reviewing various examples, it is no clearer to me.

Is someone able to point me in the right direction?

Thanks, a extremely frustrated person. :slight_smile:

2

Phew, this is a tough one… :grimacing:

This telegraf config below works, however i wasn’t able to parse the milliseconds, whatever i tried, the parser failed, therefore i excluded the milliseconds:

[[inputs.tail]]
  files = ["Tom_Aspland.txt"]
  name_override = "USQGameState"
  tagexclude = ["path"]  # we dont need the file path as tag?
  data_format = "grok"
  grok_patterns = ['\[%{DATA:timestamp:ts-"2006.01.02-15.04.05"}:\d+\]\[\d+\].+Rate: %{NUMBER:TPS:float}']

This should work in my opinion, but it doesn’t:

grok_patterns =  ['\[%{DATA:timestamp:ts-"2006.01.02-15.04.05:000"}\]\[\d+\].+Rate: %{NUMBER:TPS:float}']

I think the Go time parser itself cannot handle it, because i tried it in Go itself, which also failed:

package main

import (
	"fmt"
	"time"
)

func main() {
	s := "2022.05.14-00.37.13:544"
	t, err := time.Parse("2006.01.02-15.04.05:000", s)

	fmt.Println(t)
	fmt.Println(err)
}

with

parsing time "2022.05.14-00.37.13:544" as "2006.01.02-15.04.05:000": cannot parse "544" as ":000"

You can play around here with my example on the Go Playground:

I have no clue, how to get the milliseconds working with the Go time parser… :man_shrugging: :man_shrugging: :man_shrugging:
Any Golang experts here?

2 Likes
3

Per golang/go#6189 it appears that a period and a comma are supported, but not a colon in Go. The bug claims a localization issue.

These would work in the playground example:

2022.05.14-00.37.13.544
2022.05.14-00.37.13,544
1 Like
4

Ok, since this seems to be a dead end, lets try a different approach with a processors.starlark plugin, this should work:

[[inputs.tail]]
  files = ["Tom_Aspland.txt"]
  name_override = "USQGameState"
  tagexclude = ["path"]  # we dont need the file path as tag
  data_format = "grok"
  grok_patterns = ['\[%{DATA:timestamp:string}\].+Rate: %{NUMBER:TPS:float}']

[[processors.starlark]]
  namepass = ["USQGameState"]
  source = '''
load('time.star', 'time')

def apply(metric):
  timestamp = metric.fields.pop("timestamp")
  seconds, milliseconds = timestamp.split(":")
  nanoseconds = int(milliseconds) * 1000000
  metric.time = time.parse_time(seconds, format="2006.01.02-15.04.05").unix_nano + nanoseconds
  return metric
'''
5

Thanks Franky1 and jpowers,

I am not worried about milliseconds being parsed, as the message only appears every 30-60 seconds in the log.

When I get home I’ll give it a run, the only things I need tagging are the time excluding sub second and the TPS value for inclusion in a graph.

I really appreciate the effort involved.

With gratitude,
Tom

2 Likes
6

Your first solution works perfectly!

Thanks again.

1 Like