Telegraf Grok - how to handle dynamic / repeating patterns?

Telegraf
1

Considering the following output:

PROD - VALUE1: 50 VALUE2: 392 VALUE5: 60 VALUE8: 66
UAT - VALUE1: 4 VALUE2: 400 VALUE3: 40 VALUE4: 22 VALUE5: 99 VALUE6: 70

How would i go best about matching that with a Grok filter?

What i do have now is something like this

%{WORD:Environment:tag} - (VALUE1: %{INT:VALUE1} )?(VALUE2: %{INT:VALUE2} )?(VALUE3: %{INT:VALUE3} )?(VALUE4: %{INT:VALUE4} )?(VALUE5: %{INT:VALUE5} )?(VALUE6: %{INT:VALUE6} )?(VALUE7: %{INT:VALUE7} )?(VALUE8: %{INT:VALUE8})?

That more or less does work - however this is really unwieldy (real number of values in the output i have to parse is at least 2x to 3x) and, most of all, the output could be added to at any time with fields i do not know the name beforehand.

So, what i’m trying to achieve is some kind of dynamic, repeating pattern that always has the name of the value from the log output, data-type is always an integer.

I hope someone does have a suggestion on how to go about something like this, i surely am not the only one with such a request? :slight_smile:

Thanks in advance!

2

Any pointers? If someone could weigh in i’d be very grateful :slight_smile:

3

Have you looked at the custom pattern section here? You could try writing a regular expression to look for VALUEnumber. VALUE[\d+:\d+] will match VALUE followed by any number.

1 Like
4

Sorry for not explaining myself clearer: VALUE was just a placeholder - i don’t actually know beforehand what’s where - basically this could just be random key:value pairs where i don’t know beforehand what the key is named and each line could contain any number of key:value pairs.

What i now did is the following (i guess another small detail i have left out that i, luckily, can control output-format): i rewrote my outputting script to generate logfmt-compatible output.

So, what i now have is looking like

ENV=PROD ZZZ=20 YFA=11 DFGHUUEJAJJA=893

What i additionally needed was to basically select (*) where ENV=PROD to display all k:v pairs for the defined ENV.

With logfmt output you will need to use the processor.converter to switch the string in ENV to a tag:

[[processors.converter]]
  [processors.converter.fields]
    tag = ["ENV"]

Seems to work well so far :slight_smile: