Changing snmp input with regex using telegraf pulls before influxdb

sickriffs · September 18, 2020, 1:19pm

Hello, and thank you in advance for any help you may provide.

I have influxdb and telegraf running and collecting data.

I have a question about using regex to change the results of an snmp pull from an IP camera.

The results I get for some of the fields contain the word PERCENT with the data. So, instead of getting just a numerical value of “10”, I get “10 PERCENT.”

I have been using [[processors.regex]] to try and get rid of the “PERCENT” but I seem to be missing something.

Here is my current config:

[[inputs.snmp]]
  agents = [ "ipaddress" ]
  version = 2
  community = "pass"
  interval = "60s"
  timeout = "10s"
  retries = 3

[[inputs.snmp.field]]
  name="cpuPercent"
  oid = "HIK-DEVICE-MIB::cpuPercent.0"

[[processors.regex]]
  namepass = "cpuPercent"
  order = 1
  [[processors.regex.fields]]
    pattern = "[^ ]+ PERCENT"
    replacement = "${1}"

When I look in the influxdb I am still getting a value of “10 PERCENT”

name: snmp
time                agent_host cpuPercent 
----                ---------- ---------- 
1600392300000000000 IP-ADDRESS 10 PERCENT

I believe the regex itself is correct (I certainly could be wrong), but I believe I am missing something with the telegraf syntax.

Any advice? Thanks again!

Giovanni_Luisotto · September 22, 2020, 8:55am

This should work.

I’ve added the converter processor to turn the string into an actual numeric value

[[processors.regex]]
  order = 1
  namepass = "cpuPercent"
  [[processors.regex.fields]]
    pattern = "[^0-9]+"
    replacement = "${1}"

[[processors.converter]]
  order = 2
  [processors.converter.fields]
    integer = ["cpuPercent"]

sickriffs · September 21, 2020, 11:39pm

Thank you so much for the reply!

I gave it a shot, and I get an error when I run the telegraf test script.

Code:

 [[inputs.snmp.field]]
   name="cpuPercent"
   oid = "HIK-DEVICE-MIB::cpuPercent.0"

  [[processors.regex]]
    order = 1
    namepass = "cpuPercent"
  [[processors.regex.fields]]
    pattern = "[^0-9]+"
    replacement = "${1}"

   [processors.converter.fields]
    order = 2
    integer = ["cpuPercent"]

The result I get:

telegraf --test --config /etc/telegraf/telegraf.d/hiktest.conf

2020-09-21T23:36:20Z I! Starting Telegraf 1.15.2

2020-09-21T23:36:20Z E! [telegraf] Error running agent: Error loading config file /etc/telegraf/telegraf.d/hiktest.conf.test: Unsupported config format: converter

Giovanni_Luisotto · September 22, 2020, 8:57am

I’ve edited the previous answer with the correct snippet, I was missing a section.

Your config should be like this

{...}

 [[inputs.snmp.field]]
   name="cpuPercent"
   oid = "HIK-DEVICE-MIB::cpuPercent.0"

[[processors.regex]]
  order = 1
  namepass = "cpuPercent"
  [[processors.regex.fields]]
    pattern = "[^0-9]+"
    replacement = "${1}"

[[processors.converter]]
  order = 2
   [processors.converter.fields]
    integer = ["cpuPercent"]

sickriffs · September 22, 2020, 11:21am

Thanks again for the help Giovanna,

I am getting different error with this bit of code. A runtime error.

Code:


  [[inputs.snmp.field]]
    name="cpuPercent"
    oid = "HIK-DEVICE-MIB::cpuPercent.0"

  [[processors.regex]]
    order = 1
    namepass = "cpuPercent"
    [[processors.regex.fields]]
      pattern = "[^0-9]+"
      replacement = "${1}"

  [[processors.converter]]
    order = 2
    [processors.converter.fields]
      integer = ["cpuPercent"]

Error:

2020-09-22T11:20:22Z I! Starting Telegraf 1.15.2
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1f93e24]

goroutine 29 [running]:
github.com/influxdata/telegraf/plugins/processors/converter.(*Converter).convertFields(0xc0001bf470, 0x2f59ae0, 0xc00050c000)
        /go/src/github.com/influxdata/telegraf/plugins/processors/converter/converter.go:278 +0x8c4
github.com/influxdata/telegraf/plugins/processors/converter.(*Converter).Apply(0xc0001bf470, 0xc0000b4800, 0x1, 0x1, 0xc0000aeea0, 0xc00006af38, 0xc00027ed80)
        /go/src/github.com/influxdata/telegraf/plugins/processors/converter/converter.go:86 +0x9c
github.com/influxdata/telegraf/plugins/processors.(*streamingProcessor).Add(0xc00019d060, 0x2f59ae0, 0xc00050c000, 0x2f4aa80, 0xc00019d600, 0x2e52e01, 0x101)
        /go/src/github.com/influxdata/telegraf/plugins/processors/streamingprocessor.go:35 +0x81
github.com/influxdata/telegraf/models.(*RunningProcessor).Add(0xc0001bf5f0, 0x2f59ae0, 0xc00050c000, 0x2f4aa80, 0xc00019d600, 0xc00019d520, 0x0)
        /go/src/github.com/influxdata/telegraf/models/running_processor.go:95 +0x106
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc00049fa00, 0xc00019d4a0)
        /go/src/github.com/influxdata/telegraf/agent/agent.go:559 +0x156
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
        /go/src/github.com/influxdata/telegraf/agent/agent.go:554 +0x90

Giovanni_Luisotto · September 22, 2020, 2:12pm

Oh well, this is interesting, but sadly I can’t tell you why that happens, I can only guess. (The syntax itself is correct, and to me, it should work.)

A test I would do is to change the converter data type, to a different one (you are the one who knows which is the appropriate data type so pick the best one)

##Current
    [processors.converter.fields]
      integer = ["cpuPercent"]

##Test 1
    [processors.converter.fields]
      float = ["cpuPercent"]
##Test 2
    [processors.converter.fields]
      unsigned= ["cpuPercent"]

In general, any panic error is unwanted as errors are usually managed/caught, maybe @ssoroka will be interested in this.

sickriffs · September 22, 2020, 7:24pm

Thank you for all of your help!

The result of oid = "HIK-DEVICE-MIB::cpuPercent.0 is an int. 0-100.

Unfortunately, both tests you suggest result in a panic.

panic: runtime error: invalid memory address or nil pointer dereference

Perhaps I can think of a work around, or do without.

Again, I appreciate all of your time!

sickriffs · September 22, 2020, 7:38pm

Additionally, I upgraded to 1.15.3 just now, and get the same results.

Giovanni_Luisotto · September 22, 2020, 8:20pm

If you remove the converter plugin which result do you get?
it should be a number, but of string data type… so not really useful. that said, try to do so, maybe in the value there is an invalid char or something and that breaks the converter processor.

You might consider opening an issue on github for the panic error you are getting (and if you don’t I will)

sickriffs · September 22, 2020, 9:25pm

“10 PERCENT” is the result without the converter plugin.

influxdb result:

name: snmp
time                agent_host cpuPercent 
----                ---------- ---------- 
1600392300000000000 IP-ADDRESS 10 PERCENT

test script result:

> snmp,agent_host=IP-ADDRESS,host=hostname cpuPercent="10 PERCENT",deviceModel="DS-2CD2042WD-I",manufacturer="Hikvision",memUsed="80 PERCENT",staticIp="IP-ADDRESS",vidEncode="H.264" 1600809604000000000
111

I will attempt to open the issue, and if I fail, I will report back.

Giovanni_Luisotto · September 22, 2020, 10:27pm

Oh well that explains it, the regex is not working properly, or at least the replacement. And that string cannot be converted to an integer.

I’ve tested the regex in the golang playground and it actually matched only the number. I’ll have another look at it in the morning

sickriffs · September 22, 2020, 11:41pm

Thank you very much! Should I still open an issue on github for the panic error?

ssoroka · September 23, 2020, 4:16am

Yes, we’d love issues for the panics you’re getting. https://github.com/influxdata/telegraf

I think the problem you’re having with the pattern/replacement config here is that you can’t use ${1} without including the selector parenthesis ( ) around the part of the regex you’re trying to select. eg: pattern = "([^0-9]+)"

sickriffs · September 23, 2020, 11:19am

Hi ssoroka,

I receive a panic with the pattern you have suggested as well.

I will open up an issue this morning.

sickriffs · September 28, 2020, 4:13pm

In case someone stumbles on this thread and is looking for the answer.

Something to note, the plugin portion [[processors.regex]] and [processors.converter]] needs to be at the end of the file, after all of the [[inputs.snmp.field]] parts (if you have multiple)

I had the regex incorrect, which is corrected below.

The namepass portion was also incorrect, corrected below. The namepass should reference the inputs portions, and then the key, references the name of the inputs.snmp

Here was the solution:

[[inputs.snmp.field]]
    name="cpuPercent"
    oid = "HIK-DEVICE-MIB::cpuPercent.0"

[[processors.regex]]
    order = 1
    namepass = ["snmp"]
    [[processors.regex.fields]]
      key = "cpuPercent"
      pattern = "^(\\d+).*"
      replacement = "${1}"

  [[processors.converter]]
    order = 2
    [processors.converter.fields]
      integer = ["cpuPercent"]