Changing snmp input with regex using telegraf pulls before influxdb

Collect
,
#1

Hello, and thank you in advance for any help you may provide.

I have influxdb and telegraf running and collecting data.

I have a question about using regex to change the results of an snmp pull from an IP camera.

The results I get for some of the fields contain the word PERCENT with the data. So, instead of getting just a numerical value of “10”, I get “10 PERCENT.”

I have been using [[processors.regex]] to try and get rid of the “PERCENT” but I seem to be missing something.

Here is my current config:

[[inputs.snmp]]
  agents = [ "ipaddress" ]
  version = 2
  community = "pass"
  interval = "60s"
  timeout = "10s"
  retries = 3

[[inputs.snmp.field]]
  name="cpuPercent"
  oid = "HIK-DEVICE-MIB::cpuPercent.0"

[[processors.regex]]
  namepass = "cpuPercent"
  order = 1
  [[processors.regex.fields]]
    pattern = "[^ ]+ PERCENT"
    replacement = "${1}"

When I look in the influxdb I am still getting a value of “10 PERCENT”

name: snmp
time                agent_host cpuPercent 
----                ---------- ---------- 
1600392300000000000 IP-ADDRESS 10 PERCENT

I believe the regex itself is correct (I certainly could be wrong), but I believe I am missing something with the telegraf syntax.

Any advice? Thanks again!

#2

This should work.

I’ve added the converter processor to turn the string into an actual numeric value

[[processors.regex]]
  order = 1
  namepass = "cpuPercent"
  [[processors.regex.fields]]
    pattern = "[^0-9]+"
    replacement = "${1}"

[[processors.converter]]
  order = 2
  [processors.converter.fields]
    integer = ["cpuPercent"]
#3

Thank you so much for the reply!

I gave it a shot, and I get an error when I run the telegraf test script.

Code:

 [[inputs.snmp.field]]
   name="cpuPercent"
   oid = "HIK-DEVICE-MIB::cpuPercent.0"

  [[processors.regex]]
    order = 1
    namepass = "cpuPercent"
  [[processors.regex.fields]]
    pattern = "[^0-9]+"
    replacement = "${1}"

   [processors.converter.fields]
    order = 2
    integer = ["cpuPercent"]

The result I get:

telegraf --test --config /etc/telegraf/telegraf.d/hiktest.conf

2020-09-21T23:36:20Z I! Starting Telegraf 1.15.2

2020-09-21T23:36:20Z E! [telegraf] Error running agent: Error loading config file /etc/telegraf/telegraf.d/hiktest.conf.test: Unsupported config format: converter

#4

I’ve edited the previous answer with the correct snippet, I was missing a section.

Your config should be like this

{...}

 [[inputs.snmp.field]]
   name="cpuPercent"
   oid = "HIK-DEVICE-MIB::cpuPercent.0"

[[processors.regex]]
  order = 1
  namepass = "cpuPercent"
  [[processors.regex.fields]]
    pattern = "[^0-9]+"
    replacement = "${1}"

[[processors.converter]]
  order = 2
   [processors.converter.fields]
    integer = ["cpuPercent"]
#5

Thanks again for the help Giovanna,

I am getting different error with this bit of code. A runtime error.

Code:


  [[inputs.snmp.field]]
    name="cpuPercent"
    oid = "HIK-DEVICE-MIB::cpuPercent.0"

  [[processors.regex]]
    order = 1
    namepass = "cpuPercent"
    [[processors.regex.fields]]
      pattern = "[^0-9]+"
      replacement = "${1}"

  [[processors.converter]]
    order = 2
    [processors.converter.fields]
      integer = ["cpuPercent"]

Error:

2020-09-22T11:20:22Z I! Starting Telegraf 1.15.2
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1f93e24]

goroutine 29 [running]:
github.com/influxdata/telegraf/plugins/processors/converter.(*Converter).convertFields(0xc0001bf470, 0x2f59ae0, 0xc00050c000)
        /go/src/github.com/influxdata/telegraf/plugins/processors/converter/converter.go:278 +0x8c4
github.com/influxdata/telegraf/plugins/processors/converter.(*Converter).Apply(0xc0001bf470, 0xc0000b4800, 0x1, 0x1, 0xc0000aeea0, 0xc00006af38, 0xc00027ed80)
        /go/src/github.com/influxdata/telegraf/plugins/processors/converter/converter.go:86 +0x9c
github.com/influxdata/telegraf/plugins/processors.(*streamingProcessor).Add(0xc00019d060, 0x2f59ae0, 0xc00050c000, 0x2f4aa80, 0xc00019d600, 0x2e52e01, 0x101)
        /go/src/github.com/influxdata/telegraf/plugins/processors/streamingprocessor.go:35 +0x81
github.com/influxdata/telegraf/models.(*RunningProcessor).Add(0xc0001bf5f0, 0x2f59ae0, 0xc00050c000, 0x2f4aa80, 0xc00019d600, 0xc00019d520, 0x0)
        /go/src/github.com/influxdata/telegraf/models/running_processor.go:95 +0x106
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc00049fa00, 0xc00019d4a0)
        /go/src/github.com/influxdata/telegraf/agent/agent.go:559 +0x156
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
        /go/src/github.com/influxdata/telegraf/agent/agent.go:554 +0x90
#6

Oh well, this is interesting, but sadly I can’t tell you why that happens, I can only guess. (The syntax itself is correct, and to me, it should work.)

A test I would do is to change the converter data type, to a different one (you are the one who knows which is the appropriate data type so pick the best one)

##Current
    [processors.converter.fields]
      integer = ["cpuPercent"]

##Test 1
    [processors.converter.fields]
      float = ["cpuPercent"]
##Test 2
    [processors.converter.fields]
      unsigned= ["cpuPercent"]

In general, any panic error is unwanted as errors are usually managed/caught, maybe @ssoroka will be interested in this.

#7

Thank you for all of your help!

The result of oid = "HIK-DEVICE-MIB::cpuPercent.0 is an int. 0-100.

Unfortunately, both tests you suggest result in a panic.

panic: runtime error: invalid memory address or nil pointer dereference

Perhaps I can think of a work around, or do without.

Again, I appreciate all of your time!

#8

Additionally, I upgraded to 1.15.3 just now, and get the same results.

#9

If you remove the converter plugin which result do you get?
it should be a number, but of string data type… so not really useful. that said, try to do so, maybe in the value there is an invalid char or something and that breaks the converter processor.

You might consider opening an issue on github for the panic error you are getting (and if you don’t I will)

#10

“10 PERCENT” is the result without the converter plugin.

influxdb result:

name: snmp
time                agent_host cpuPercent 
----                ---------- ---------- 
1600392300000000000 IP-ADDRESS 10 PERCENT

test script result:

> snmp,agent_host=IP-ADDRESS,host=hostname cpuPercent="10 PERCENT",deviceModel="DS-2CD2042WD-I",manufacturer="Hikvision",memUsed="80 PERCENT",staticIp="IP-ADDRESS",vidEncode="H.264" 1600809604000000000
111

I will attempt to open the issue, and if I fail, I will report back.

#11

Oh well that explains it, the regex is not working properly, or at least the replacement. And that string cannot be converted to an integer.

I’ve tested the regex in the golang playground and it actually matched only the number. I’ll have another look at it in the morning

1 Like
#12

Thank you very much! Should I still open an issue on github for the panic error?

#13

Yes, we’d love issues for the panics you’re getting. https://github.com/influxdata/telegraf

I think the problem you’re having with the pattern/replacement config here is that you can’t use ${1} without including the selector parenthesis ( ) around the part of the regex you’re trying to select. eg: pattern = "([^0-9]+)"

#14

Hi ssoroka,

I receive a panic with the pattern you have suggested as well.

I will open up an issue this morning.

#15

In case someone stumbles on this thread and is looking for the answer.

Something to note, the plugin portion [[processors.regex]] and [processors.converter]] needs to be at the end of the file, after all of the [[inputs.snmp.field]] parts (if you have multiple)

I had the regex incorrect, which is corrected below.

The namepass portion was also incorrect, corrected below. The namepass should reference the inputs portions, and then the key, references the name of the inputs.snmp

Here was the solution:

[[inputs.snmp.field]]
    name="cpuPercent"
    oid = "HIK-DEVICE-MIB::cpuPercent.0"

[[processors.regex]]
    order = 1
    namepass = ["snmp"]
    [[processors.regex.fields]]
      key = "cpuPercent"
      pattern = "^(\\d+).*"
      replacement = "${1}"

  [[processors.converter]]
    order = 2
    [processors.converter.fields]
      integer = ["cpuPercent"]
#16

Hello,
Sorry for my bad english (google translate is my friend) and i am a beginner.
I reproduced your example of conversion solution for the “cpuPercent” field like this:

> [[processors.regex]]
>   order = 1
>   namepass = ["snmp"]
>   [[processors.regex.fields]]
>     key = "cpuPercent"
>     pattern = "^(\\d+).*"
>     replacement = "${1}"
>   [[processors.regex.fields]]
>     key = "memUsed"
>     pattern = "^(\\d+).*"
>     replacement = "${1}"
> 
> [[processors.converter]]
>   order = 2
>   [processors.converter.fields]
>     integer = ["cpuPercent", "memUsed"]

but it does not work I always have a “panic” error.

2020-10-20T19:30:05Z I! Starting Telegraf 1.15.3
2020-10-20T19:30:05Z I! Using config file: /etc/telegraf/telegraf.conf
2020-10-20T19:30:05Z I! Loaded inputs: kernel system cpu disk diskio swap docker snmp snmp mem processes
2020-10-20T19:30:05Z I! Loaded aggregators:
2020-10-20T19:30:05Z I! Loaded processors: regex converter
2020-10-20T19:30:05Z I! Loaded outputs: influxdb
2020-10-20T19:30:05Z I! Tags enabled: host=telegraf
2020-10-20T19:30:05Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"telegraf", Flush Interval:10s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1f94624]

goroutine 36 [running]:
github.com/influxdata/telegraf/plugins/processors/converter.(*Converter).convertFields(0xc0002e2ae0, 0x2f59f00, 0xc000290000)
        /go/src/github.com/influxdata/telegraf/plugins/processors/converter/converter.go:278 +0x8c4
github.com/influxdata/telegraf/plugins/processors/converter.(*Converter).Apply(0xc0002e2ae0, 0xc000a69540, 0x1, 0x1, 0xc0002a4060, 0x1, 0xc000286780)
        /go/src/github.com/influxdata/telegraf/plugins/processors/converter/converter.go:86 +0x9c
github.com/influxdata/telegraf/plugins/processors.(*streamingProcessor).Add(0xc0001e6d00, 0x2f59f00, 0xc000290000, 0x2f4aea0, 0xc000294040, 0x1, 0x101)
        /go/src/github.com/influxdata/telegraf/plugins/processors/streamingprocessor.go:35 +0x81
github.com/influxdata/telegraf/models.(*RunningProcessor).Add(0xc0002e2c60, 0x2f59f00, 0xc000290000, 0x2f4aea0, 0xc000294040, 0x0, 0x0)
        /go/src/github.com/influxdata/telegraf/models/running_processor.go:95 +0x106
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc0005d3330, 0xc0001e7760)
        /go/src/github.com/influxdata/telegraf/agent/agent.go:559 +0x156
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
        /go/src/github.com/influxdata/telegraf/agent/agent.go:554 +0x90

So I conclude that the value extracted by [[proccessors.regex]] is not a string which can then be converted into Integer by [[proccessors.converter]].
Shouldn’t the pattern instead be: “^(\d{1,2})” to return a string of numbers?
I just wish I could retrieve the “XX” value as a numeric of the “XX PERCENT” value from the cupPercent and memUsed fields to keep track of it over time on a graph.
Please, can you explain to me what I need to do to fix this error.
Thank you for your kind response.
Cordially.

#17

In the converter process you are referencing memUsed as well.

[[processors.converter]]
   order = 2
   [processors.converter.fields]
     integer = ["cpuPercent", "memUsed"]

Are you pulling that data in the [[inputs.snmp.field]] portion?

#18

Hello,
To answer your question, you will find below the HIKvision part that I inserted in the telegraf.conf file.

>   [[inputs.snmp]]
>     agents = [  "192.168.2.21", "192.168.2.22", "192.168.2.23", "192.168.2.24" ]
>     interval = "60s"
>     timeout = "10s"
>     retries = 3
>     version = 2
>     community = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>     max_repetitions = 30
>     name = "snmp.HIK"
>     
>   [[inputs.snmp.field]]
>     name = "manufacturer"
>     oid = "HIK-DEVICE-MIB::manufacturer.0"
>   [[inputs.snmp.field]]
>     name = "deviceModel"
>     oid = "HIK-DEVICE-MIB::deviceType.0"
>   [[inputs.snmp.field]]
>     name = "deviceID"
>     oid = "HIK-DEVICE-MIB::deviceID.0"
>   [[inputs.snmp.field]]
>     name = "softwVersion"
>     oid = "HIK-DEVICE-MIB::softwVersion.0"
>   [[inputs.snmp.field]]
>     name="cpuPercent"
>     oid = "HIK-DEVICE-MIB::cpuPercent.0"
>   [[inputs.snmp.field]]
>     name="memUsed"
>     oid = "HIK-DEVICE-MIB::memUsed.0"
>   [[inputs.snmp.field]]
>     name="staticIpAddr"
>     oid = "HIK-DEVICE-MIB::staticIpAddr.0"
>   [[inputs.snmp.field]]
>     name="dynIpAddr"
>     oid = "HIK-DEVICE-MIB::dynIpAddr.0"
>   [[inputs.snmp.field]]
>     name="macAddr"
>     oid = "HIK-DEVICE-MIB::macAddr.0"
>   [[inputs.snmp.field]]
>     name="vidEncode"
>     oid = "HIK-DEVICE-MIB::videoEncode.0"

For information, the [[processors.regex]] and [[processors.converter]] parts are placed at the end of the telegraf.conf file.
Cordially

#20

Adding conversion = “int” in the [[inputs.snmp.fields]] portion will not work, that value does not produce an int.

[[inputs.snmp.field]]
name=“cpuPercent”
oid = “HIK-DEVICE-MIB::cpuPercent.0”
conversion = “int”

If you remove all of the conversion portions, and run the test, does it produce working results?

telegraf --test --config /etc/telegraf/telegraf.d/telegraf.conf
This path may be different then where you have your .conf

#21

Hello,
Sorry, but I’m a little lost there.
I have remove all of the previous conversion portions. Then I have inserted this at the end of telegraf.conf file:

> [[processors.strings]]
>   order = 1
>   namepass = "snmp"
>   [[processors.strings.left]]
>     field = "cpuPercent"
>     width = 2
>   [[processors.strings.trim]]
>     field = "cpuPercent"
>   [[processors.strings.left]]
>     field = "memUsed"
>     width = 2
>   [[processors.strings.trim]]
>     field = "memUsed"`

I run the test and it produce thoses working results:

snmp.HIK,agent_host=192.168.2.24,host=telegraf cpuPercent=“60”,deviceID=“4”,deviceModel=“DS-2CD2085FWD-I”,dynIpAddr=“192.168.2.24”,macAddr=“xx-xx-xx-xx-xx-xx”,manufacturer=“Hikvision”,memUsed=“75”,restartDev=0i,softwVersion=“V5.6.5 build 200316”,staticIpAddr=“0.0.0.0”,vidEncode=“H.264” 1603287403000000000

I am getting the correct values for cpuPercent and memUsed but they are not numeric.
If in addition, I add the portion [[processors.converter]] then telegraf fails with error.
Now , how then to convert this string to integer?
Cordially

Welcome
Events
Legal
Blog
Careers
Community
InfluxData
799 Market Street, Suite 400
San Francisco, CA 94103
© 2018 InfluxData, Inc.