Telegraf file-input custom grok pattern timestamp and line breaks

hi,
i am trying to work with grok, but i am failing.
my need:
parse frequently a file, which is written newly (so no “tail” needed).
within this file i use grok pattern to extract informations.

the file is structured like this:

TBatch Batch
{
  CreationTime = $01D1231234543210
  GUID = \7B123a45b6-c78d-90e1-2fab-c345de6fa789\7D
  Version = 3.4.5
  PageCount = 12
  DocumentCount = 3
  DeclinedPageCount = 0
  DeclinedDocumentCount = 0
  Documents = 1,2,3
  DisplayName = NAME-OF-TYPE-TIMESTAMP
  Priority = 2
  BatchClass = ID-OF-BATCH
  Position = ID-POSITION
  State = 2
  Stamp_Created = 24.09.2021|14:59:35|ID_BATCH|HOSTNAME|PROCUSER
}

i have two questions:

  1. how would it be possible to get the linebreaks removed to get all information in one? with several grok patterns i get several data which will be inserted to the database (influxdb).
  2. i have issues in getting the timestamp converted in to unix_timestamp. I tried using this custom grok pattern:
MYTS %{DATE_EU}.%{TIME}

and this grpk_pattern:

"\\sStamp_Created\\s=\\s%{MYTS:mytimestamp}.ID.*"

output:
mytimestamp=“04.10.2021|17:33:35”

when i try to convert that to another timestamp, i get an error message:

Error parsing timestamp [04.10.2021|17:33:35], could not find any suitable time layouts.

help appreciated,
kidn regards,
andre

Hello @astrakid,
Your timestamp needs to be in one of the following formats:

  • Timestamp modifiers:
  • ts (This will auto-learn the timestamp format)
  • ts-ansic (“Mon Jan _2 15:04:05 2006”)
  • ts-unix (“Mon Jan _2 15:04:05 MST 2006”)
  • ts-ruby (“Mon Jan 02 15:04:05 -0700 2006”)
  • ts-rfc822 (“02 Jan 06 15:04 MST”)
  • ts-rfc822z (“02 Jan 06 15:04 -0700”)
  • ts-rfc850 (“Monday, 02-Jan-06 15:04:05 MST”)
  • ts-rfc1123 (“Mon, 02 Jan 2006 15:04:05 MST”)
  • ts-rfc1123z (“Mon, 02 Jan 2006 15:04:05 -0700”)
  • ts-rfc3339 (“2006-01-02T15:04:05Z07:00”)
  • ts-rfc3339nano (“2006-01-02T15:04:05.999999999Z07:00”)
  • ts-httpd (“02/Jan/2006:15:04:05 -0700”)
  • ts-epoch (seconds since unix epoch, may contain decimal)
  • ts-epochnano (nanoseconds since unix epoch)
  • ts-epochmilli (milliseconds since unix epoch)
  • ts-syslog (“Jan 02 15:04:05”, parsed time is set to the current year)
  • ts-“CUSTOM”
    telegraf/plugins/parsers/grok at master · influxdata/telegraf · GitHub
    I don’t think that timestamp will work. You could use an execd processor plugin to convert. That might also help you with your new line problem too.
    Tagging a telegraf expert for advice @Mya thank you!
1 Like

thanks for involving @Mya . i am a little bit lost here.

Hi @astrakid, I am sorry to hear you are having a hard time with this. Will you please post your config file so I can run it and mess with it on my end? Have you tried taking a look at this thread?

1 Like