Incorrect data in influxdb

telegraf
influxdb
grafana

#1

I have a single stat panel in Grafana, that is set to show how many hours it’s been since a server has backed up. The panel is set to pull the last() value from a log file, and the stat is set to current. Here’s the last couple of lines of my log file … notice Last_Backup_Hours=11.

Last_Backup_Time=20190213070054,Last_Backup_Hours=11,Data_Transferred=13326848,Files_Transferred=6,Backup_Status=1
Last_Backup_Time=20190213070054,Last_Backup_Hours=11,Data_Transferred=13326848,Files_Transferred=6,Backup_Status=1
Last_Backup_Time=20190213070054,Last_Backup_Hours=11,Data_Transferred=13326848,Files_Transferred=6,Backup_Status=1

But when my board refreshes, sometimes I get random numbers. See screenshots below.

It will jump from 11 hours, the correct time, to 23 hours. Which is a big difference from what it just showed 1 minute ago. Then in a couple minutes it goes back to showing the correct value.

When I run the query in influxdb directly, I get this result.

> SELECT last("Last_Backup_Hours") FROM "Backup_Jobs" WHERE ("Server" = 'nameofserver') AND time >= now() - 5m GROUP BY time(1m) fill(null)
name: Backup_Jobs
time                last
----                ----
1550082300000000000 11
1550082360000000000 11
1550082420000000000 11
1550082480000000000 11
1550082540000000000 11
1550082600000000000 23
> 

My problem is, I don’t know where influx got the “23” from. The last 30 lines of my log files have the same value listed as 11. Telegraf is sending the log file to influxdb every 60 seconds. The default “time interval” on my influxdb datasource in Grafana is “1m”.

Is it possible I have something mis-configured for my single stat? Or do I not understand how influxdb works.


#2

I was able to resolve this issue by writing a POST command directly to influxdb, instead of having telegraf send a influx line protocol file to influxdb. For whatever reason when telegraf was sending the log files, it was highly inconsistent.