SNMP reply does not reach telegraf

Telegraf
1

Hi,

We’re implementing telegraf to monitor some remote sensor installations, the sensor data itself already flows to influxdb. I enabled snmp on the modems and got it to work with snmpget and friends. So I went on to configure telegraf as below. Which I tested as follows:

./telegraf --config /etc/telegraf/telegraf.conf --input-filter snmp --test --debug
2021-04-21T08:12:28Z I! Starting Telegraf 1.18.1
2021-04-21T08:12:28Z D! [agent] Initializing plugins
2021-04-21T08:12:28Z D! [agent] Starting service inputs
2021-04-21T08:12:28Z D! [inputs.snmp] executing "snmptranslate" "-Td" "-Ob" "IF-MIB::ifOutOctets.3"
2021-04-21T08:13:08Z E! [inputs.snmp] Error in plugin: agent 192.168.204.1: performing get on field ifOutOctets: request timeout (after 3 retries)
2021-04-21T08:13:08Z D! [agent] Stopping service inputs
2021-04-21T08:13:08Z D! [agent] Input channel closed
2021-04-21T08:13:08Z D! [agent] Processor channel closed
2021-04-21T08:13:08Z D! [agent] Processor channel closed
2021-04-21T08:13:08Z D! [agent] Processor channel closed
2021-04-21T08:13:08Z D! [agent] Stopped Successfully
2021-04-21T08:13:08Z E! [telegraf] Error running agent: input plugins recorded 1 errors

So I checked snmpget:

snmpget -v 2c -cpublic 192.168.204.1 RFC1213-MIB::sysUpTime.0
RFC1213-MIB::sysUpTime.0 = Timeticks: (5760500) 16:00:05.00

Thats still works. So I checked tcpdump to see what is going on, with snmpget:

10:19:52.124573 IP (tos 0x0, ttl 64, id 64948, offset 0, flags [DF], proto UDP (17), length 71)
    ip-10-1-1-7.eu-central-1.compute.internal.41390 > ip-192-168-204-1.eu-central-1.compute.internal.snmp:  { SNMPv2c { GetRequest(28) R=1551454767  system.sysUpTime.0 } }
10:19:52.125122 IP (tos 0x0, ttl 64, id 12974, offset 0, flags [DF], proto UDP (17), length 72)
    ip-10-1-1-7.eu-central-1.compute.internal.44501 > ip-10-0-0-2.eu-central-1.compute.internal.domain: 29541+ PTR? 1.204.168.192.in-addr.arpa. (44)
10:19:52.131363 IP (tos 0x0, ttl 253, id 0, offset 0, flags [DF], proto UDP (17), length 132)
    ip-10-0-0-2.eu-central-1.compute.internal.domain > ip-10-1-1-7.eu-central-1.compute.internal.44501: 29541 1/0/0 1.204.168.192.in-addr.arpa. PTR ip-192-168-204-1.eu-central-1.compute.internal. (104)
10:19:52.158332 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 74)
    ip-10-1-1-9.eu-central-1.compute.internal.snmp > ip-10-1-1-7.eu-central-1.compute.internal.41390:  { SNMPv2c { GetResponse(31) R=1551454767  system.sysUpTime.0=5760500 } }

That works as expected, now the same tcpdump with telegraf:

10:13:03.276940 IP (tos 0x0, ttl 64, id 44897, offset 0, flags [DF], proto UDP (17), length 73)
    ip-10-1-1-7.eu-central-1.compute.internal.52153 > ip-192-168-204-1.eu-central-1.compute.internal.snmp:  { SNMPv2c { GetRequest(30) R=2067745588  interfaces.ifTable.ifEntry.ifOutOctets.3 } }
10:13:03.324231 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 77)
    ip-10-1-1-9.eu-central-1.compute.internal.snmp > ip-10-1-1-7.eu-central-1.compute.internal.52153:  { SNMPv2c { GetResponse(34) R=2067745588  interfaces.ifTable.ifEntry.ifOutOctets.3=48063378 } }
10:13:03.324277 IP (tos 0xc0, ttl 64, id 43571, offset 0, flags [none], proto ICMP (1), length 105)
    ip-10-1-1-7.eu-central-1.compute.internal > ip-10-1-1-9.eu-central-1.compute.internal: ICMP ip-10-1-1-7.eu-central-1.compute.internal udp port 52153 unreachable, length 85
        IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 77)
    ip-10-1-1-9.eu-central-1.compute.internal.snmp > ip-10-1-1-7.eu-central-1.compute.internal.52153:  { SNMPv2c { GetResponse(34) R=2067745588  interfaces.ifTable.ifEntry.ifOutOctets.3=48063378 } }

Which shows the modem is replying, but telegraf is not accepting the packet??? I also tried snmp v3, but with the same results…

Everything runs in a docker container and is connected to the modem via an openvpn connection. Did I configure telegraf wrong, is something else in the setup wrong, or is there a bug in (the) telegraf(plugin)? I have very similar setups (without openvpn) that work.
Your help would be much appreciated,

Dirkjan Krijnders

The relevant telegraf.conf part:

[[inputs.snmp]]
  agents = [ "192.168.204.1" ]

  ## SNMP version; can be 1, 2, or 3.
  version = 2

  ## SNMP community string.
  community = "public"

  ## Number of retries to attempt.
  retries = 3

  ## The GETBULK max-repetitions parameter.
  max_repetitions = 10

  ## SNMPv3 authentication and encryption options.
  ##
  ## Security Name.
  sec_name = "admin"
  ## Authentication protocol; one of "MD5", "SHA", or "".
  auth_protocol = "MD5"
  ## Authentication password.
  auth_password = "censored"
  ## Security Level; one of "noAuthNoPriv", "authNoPriv", or "authPriv".
  sec_level = "authNoPriv"


  [[inputs.snmp.field]]
     name = "ifOutOctets"
     oid = "IF-MIB::ifOutOctets.3"