How to store two timestamps in InfluxDB via Telegraf?

, , ,
1

Hello everyone,

i’m new to the whole TIG stack, however already have set up a few things already to my liking - but yesterday i’ve hit a roadblock. After googling frantically for a few hours, reading up on Github Issues etc., i guess i’ll ask around here.

So, my data gets created via custom scripts, logged to files. Telegraf then ingests those, writes them to InfluxDB and Grafana presents them neat and tidy - so far, so good.

Yesterday, i tried to parse a log that has, among other data, two timestamps - a shortened entry in the logfile may look like this:

Dateend: 2020-09-06 Datestart: 2020-09-05 somemorefieldsanddata

I’m parsing all the logs via the grok parser plugin and the accompanying patterns - my most basic pattern to parse this data looks like this:

 grok_patterns = ['Dateend: %{RBDATE:abc} Datestart: %{RBDATE:timestamp:ts-"2006-01-02"} %{GREEDYDATA}']
  grok_custom_patterns = '''
    RBDATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY}
  '''

This pattern does work, however, i only can make the first (well, actuall the second) or “Datestart” timestamp work.

“Dateend” gets saved as string, but this is not what i actually want.

I noticed that i get parser errors as soon as i append :ts-"2006-01-02" to %{RBDATE:abc} - grok seemingly can’t store two similiar timestamps?

I’d much appreciate any pointer you can give me! Is this really “impossible”? I hope not! :slight_smile:

2

Hey clclcl,

Line protocol doesn’t have a specific field type for timestamps other than the unix-nanosecond-formatted timestamp for the whole metric. Sometimes what people do if they want those fields is to store them as integers, and this at least lets you do things like calculate the difference between two timestamps, or maybe convert it back at the query layer.

so yes, at the moment your only options are storing the extra date as strings or integers in a field value

1 Like