I have been battling an issue for quite some time that I have been unable to resolve as of yet. The issue that I am having is that if I upgrade my telegraf install from version 1.5.3 to something newer I am no longer seeing any data within Grafana.
I was running Ubuntu version 18.04 and decided it was time to upgrade to version 22.04. Along with this dist upgrade, I decided to upgrade influxdb, grafana and telegraf to the latest versions even though I knew that I would no longer see data within Grafana.
I am hoping this community can point me in the right direction as to what is going on and why the upgrade of telegraf from version 1.5.3 to version 1.24.0 is causing my data to no longer show up in Grafana.
I am struggling on how to debug this issue and confirm that telegraf is collecting the data and sending it to influxdb properly?
Here are some highlights. I can clearly see in Grafana that I am no longer seeing data on the graphs from the time the upgrade took place. In the past, I have upgraded to different versions of telegraf and had the same issue and I always ended up reverting back to telegraf version 1.5.3. This time I am determined to proceed with the update.
Iâve modified the telegraf.conf file to send the metrics to stdout as well as to a file to confirm that Telegraf is still collecting the information that its configured to do so. Below is a small snippet of the output seen in the output file.
âsnmp,agent_host=192.168.0.5,host=ubuntu,hostname=SW-2960-8.gotti.net,ifAlias=Gotti-Desktop,ifDescr=GigabitEthernet0/8,ifName=Gi0/8 ifLinkUpDownTrapEnable=1i,ifCounterDiscontinuityTime=0i,ifInBroadcastPkts=8449719i,ifOutMulticastPkts=17461705i,ifHCInOctets=6829494354993i,ifHCInMulticastPkts=3856540i,ifPromiscuousMode=2i,ifConnectorPresent=1i,ifInMulticastPkts=3856540i,ifHCInUcastPkts=10544020938i,ifHCInBroadcastPkts=8449719i,ifHCOutOctets=6777258262154i,ifHCOutMulticastPkts=17461705i,ifOutBroadcastPkts=7451597i,ifHCOutUcastPkts=9116329503i,ifHCOutBroadcastPkts=7451597i,ifHighSpeed=1000i 1663031490000000000â
Iâve also tried to debug influxdb to see if its receiving data and as far as I can tell it is. I am suspecting that the metrics that it is receiving from telegraf is going into a different database table or measurement, but I donât know enough to confirm. Below is some output that I have collected from influxdb, both in a working state and a non working state:
Working State:
âSep 12 21:32:32 ubuntu influxd-systemd-start.sh[66113]: ts=2022-09-13T01:32:32.444588Z lvl=info msg=âExecuting queryâ log_id=0cu72hml000 service=query query=âSELECT non_negative_derivative(mean(ifHCOutOctets), 1s) * 8 FROM telegraf.autogen.snmp WHERE (agent_host = â192.168.0.1â AND ifName = âem1â) AND time >= 461764h AND time <= 1662436799999ms GROUP BY time(15m)â
Sep 12 21:32:32 ubuntu influxd-systemd-start.sh[66113]: ts=2022-09-13T01:32:32.457232Z lvl=info msg=âExecuting queryâ log_id=0cu72hml000 service=query query=âSELECT last(ifHCInOctets) FROM telegraf.autogen.snmp WHERE (agent_host = â192.168.0.1â AND ifName = âem1â) AND time >= 461764h AND time <= 1662436799999ms GROUP BY time(15m)ââ
Non Working State:
Sep 12 21:30:38 ubuntu influxd-systemd-start.sh[66113]: ts=2022-09-13T01:30:38.455258Z lvl=info msg=âExecuting queryâ log_id=0cu72hml000 service=query query=âSELECT last(ifHCInOctets) FROM telegraf.autogen.snmp WHERE (agent_host = â192.168.0.1â AND ifName = âem1â) AND time >= now() - 30m AND time <= now() GROUP BY time(20s)â
Sep 12 21:30:38 ubuntu influxd-systemd-start.sh[66113]: ts=2022-09-13T01:30:38.460442Z lvl=info msg=âExecuting queryâ log_id=0cu72hml000 service=query query=âSELECT non_negative_derivative(mean(ifHCInOctets), 1s) * 8 FROM telegraf.autogen.snmp WHERE (agent_host = â192.168.0.1â AND ifName = âem1â) AND time >= now() - 30m AND time <= now() GROUP BY time(20s)â
Finally, below is the conf file that I am using to poll the network device that I am collecting SNMP data from.
[[inputs.snmp]]
agents = [ â192.168.0.1:161â ]
version = 2
community = âredactedâ
name = âsnmpâ
[[inputs.snmp.field]]
name = âhostnameâ
oid = âRFC1213-MIB::sysName.0â
is_tag = true
[[inputs.snmp.table]]
name = âsnmpâ
inherit_tags = [ âhostnameâ ]
oid = âIF-MIB::ifXTableâ
[[inputs.snmp.table.field]]
name = "ifName"
oid = "IF-MIB::ifName"
is_tag = true
Current versions installed:
Telegraf 1.24.0 (git: HEAD@3c4a6516)
InfluxDB shell version: 1.8.10
Grafana v9.1.4 (2186d0bbeb)
I am sure I am leaving out some important information, please let me know what I need to provide. I appreciate anyoneâs help in resolving my issue.
