Changing snmp input with regex using telegraf pulls before influxdb

Hello, and thank you in advance for any help you may provide.

I have influxdb and telegraf running and collecting data.

I have a question about using regex to change the results of an snmp pull from an IP camera.

The results I get for some of the fields contain the word PERCENT with the data. So, instead of getting just a numerical value of “10”, I get “10 PERCENT.”

I have been using [[processors.regex]] to try and get rid of the “PERCENT” but I seem to be missing something.

Here is my current config:

[[inputs.snmp]]
  agents = [ "ipaddress" ]
  version = 2
  community = "pass"
  interval = "60s"
  timeout = "10s"
  retries = 3

[[inputs.snmp.field]]
  name="cpuPercent"
  oid = "HIK-DEVICE-MIB::cpuPercent.0"

[[processors.regex]]
  namepass = "cpuPercent"
  order = 1
  [[processors.regex.fields]]
    pattern = "[^ ]+ PERCENT"
    replacement = "${1}"

When I look in the influxdb I am still getting a value of “10 PERCENT”

name: snmp
time                agent_host cpuPercent 
----                ---------- ---------- 
1600392300000000000 IP-ADDRESS 10 PERCENT 

I believe the regex itself is correct (I certainly could be wrong), but I believe I am missing something with the telegraf syntax.

Any advice? Thanks again!

This should work.

I’ve added the converter processor to turn the string into an actual numeric value

[[processors.regex]]
  order = 1
  namepass = "cpuPercent"
  [[processors.regex.fields]]
    pattern = "[^0-9]+"
    replacement = "${1}"

[[processors.converter]]
  order = 2
  [processors.converter.fields]
    integer = ["cpuPercent"]

Thank you so much for the reply!

I gave it a shot, and I get an error when I run the telegraf test script.

Code:

 [[inputs.snmp.field]]
   name="cpuPercent"
   oid = "HIK-DEVICE-MIB::cpuPercent.0"

  [[processors.regex]]
    order = 1
    namepass = "cpuPercent"
  [[processors.regex.fields]]
    pattern = "[^0-9]+"
    replacement = "${1}"

   [processors.converter.fields]
    order = 2
    integer = ["cpuPercent"]

The result I get:

telegraf --test --config /etc/telegraf/telegraf.d/hiktest.conf

2020-09-21T23:36:20Z I! Starting Telegraf 1.15.2

2020-09-21T23:36:20Z E! [telegraf] Error running agent: Error loading config file /etc/telegraf/telegraf.d/hiktest.conf.test: Unsupported config format: converter

I’ve edited the previous answer with the correct snippet, I was missing a section.

Your config should be like this

{...}

 [[inputs.snmp.field]]
   name="cpuPercent"
   oid = "HIK-DEVICE-MIB::cpuPercent.0"

[[processors.regex]]
  order = 1
  namepass = "cpuPercent"
  [[processors.regex.fields]]
    pattern = "[^0-9]+"
    replacement = "${1}"

[[processors.converter]]
  order = 2
   [processors.converter.fields]
    integer = ["cpuPercent"]

Thanks again for the help Giovanna,

I am getting different error with this bit of code. A runtime error.

Code:


  [[inputs.snmp.field]]
    name="cpuPercent"
    oid = "HIK-DEVICE-MIB::cpuPercent.0"

  [[processors.regex]]
    order = 1
    namepass = "cpuPercent"
    [[processors.regex.fields]]
      pattern = "[^0-9]+"
      replacement = "${1}"

  [[processors.converter]]
    order = 2
    [processors.converter.fields]
      integer = ["cpuPercent"]

Error:

2020-09-22T11:20:22Z I! Starting Telegraf 1.15.2
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1f93e24]

goroutine 29 [running]:
github.com/influxdata/telegraf/plugins/processors/converter.(*Converter).convertFields(0xc0001bf470, 0x2f59ae0, 0xc00050c000)
        /go/src/github.com/influxdata/telegraf/plugins/processors/converter/converter.go:278 +0x8c4
github.com/influxdata/telegraf/plugins/processors/converter.(*Converter).Apply(0xc0001bf470, 0xc0000b4800, 0x1, 0x1, 0xc0000aeea0, 0xc00006af38, 0xc00027ed80)
        /go/src/github.com/influxdata/telegraf/plugins/processors/converter/converter.go:86 +0x9c
github.com/influxdata/telegraf/plugins/processors.(*streamingProcessor).Add(0xc00019d060, 0x2f59ae0, 0xc00050c000, 0x2f4aa80, 0xc00019d600, 0x2e52e01, 0x101)
        /go/src/github.com/influxdata/telegraf/plugins/processors/streamingprocessor.go:35 +0x81
github.com/influxdata/telegraf/models.(*RunningProcessor).Add(0xc0001bf5f0, 0x2f59ae0, 0xc00050c000, 0x2f4aa80, 0xc00019d600, 0xc00019d520, 0x0)
        /go/src/github.com/influxdata/telegraf/models/running_processor.go:95 +0x106
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc00049fa00, 0xc00019d4a0)
        /go/src/github.com/influxdata/telegraf/agent/agent.go:559 +0x156
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
        /go/src/github.com/influxdata/telegraf/agent/agent.go:554 +0x90

Oh well, this is interesting, but sadly I can’t tell you why that happens, I can only guess. (The syntax itself is correct, and to me, it should work.)

A test I would do is to change the converter data type, to a different one (you are the one who knows which is the appropriate data type so pick the best one)

##Current
    [processors.converter.fields]
      integer = ["cpuPercent"]

##Test 1
    [processors.converter.fields]
      float = ["cpuPercent"]
##Test 2
    [processors.converter.fields]
      unsigned= ["cpuPercent"]

In general, any panic error is unwanted as errors are usually managed/caught, maybe @ssoroka will be interested in this.

Thank you for all of your help!

The result of oid = "HIK-DEVICE-MIB::cpuPercent.0 is an int. 0-100.

Unfortunately, both tests you suggest result in a panic.

panic: runtime error: invalid memory address or nil pointer dereference

Perhaps I can think of a work around, or do without.

Again, I appreciate all of your time!

Additionally, I upgraded to 1.15.3 just now, and get the same results.

If you remove the converter plugin which result do you get?
it should be a number, but of string data type… so not really useful. that said, try to do so, maybe in the value there is an invalid char or something and that breaks the converter processor.

You might consider opening an issue on github for the panic error you are getting (and if you don’t I will)

“10 PERCENT” is the result without the converter plugin.

influxdb result:

name: snmp
time                agent_host cpuPercent 
----                ---------- ---------- 
1600392300000000000 IP-ADDRESS 10 PERCENT 

test script result:

> snmp,agent_host=IP-ADDRESS,host=hostname cpuPercent="10 PERCENT",deviceModel="DS-2CD2042WD-I",manufacturer="Hikvision",memUsed="80 PERCENT",staticIp="IP-ADDRESS",vidEncode="H.264" 1600809604000000000
111

I will attempt to open the issue, and if I fail, I will report back.

Oh well that explains it, the regex is not working properly, or at least the replacement. And that string cannot be converted to an integer.

I’ve tested the regex in the golang playground and it actually matched only the number. I’ll have another look at it in the morning

1 Like

Thank you very much! Should I still open an issue on github for the panic error?

Yes, we’d love issues for the panics you’re getting. https://github.com/influxdata/telegraf

I think the problem you’re having with the pattern/replacement config here is that you can’t use ${1} without including the selector parenthesis ( ) around the part of the regex you’re trying to select. eg: pattern = "([^0-9]+)"

Hi ssoroka,

I receive a panic with the pattern you have suggested as well.

I will open up an issue this morning.

In case someone stumbles on this thread and is looking for the answer.

Something to note, the plugin portion [[processors.regex]] and [processors.converter]] needs to be at the end of the file, after all of the [[inputs.snmp.field]] parts (if you have multiple)

I had the regex incorrect, which is corrected below.

The namepass portion was also incorrect, corrected below. The namepass should reference the inputs portions, and then the key, references the name of the inputs.snmp

Here was the solution:

[[inputs.snmp.field]]
    name="cpuPercent"
    oid = "HIK-DEVICE-MIB::cpuPercent.0"

[[processors.regex]]
    order = 1
    namepass = ["snmp"]
    [[processors.regex.fields]]
      key = "cpuPercent"
      pattern = "^(\\d+).*"
      replacement = "${1}"

  [[processors.converter]]
    order = 2
    [processors.converter.fields]
      integer = ["cpuPercent"]