(I asked about this topic about a week ago, but I can’t find the thread anymore. The thread may have disappeared due to the hacker attack on the forum (e.g. a backup was restored)???)
I’m using Telegraf to convert several MQTT streams into Line Protocol and sending them to an InfluxDB2 instance running on the same machine as Telegraf (plugins " MQTT Consumer Input" & “InfluxDB v2.x Output”). Although all LP metrics generated by Telegraf look correct to me, data from an MQTT source does not appear in the database. Or rather, only a few minutes of this data stream were visible (in InfluxDB2’s Data Explorer). The data from the other MQTT sources is present and complete.
Is there a way to check which data is actually in the database, i.e. all points individually and not aggregated in a time window, as the Data Explorer does? Strangely enough, the “show raw data” button did not help me. I would like to have a way of knowing for each incoming LP line whether it was accepted and added to the database, or if not, what was the reason.
Thanks!
Best,
conne914
I don’t think there is an easy way of checking that, not with that level of detail.
I’m not familiar with the data explorer but I’m sure you can simply query the DB to get the raw data you want without any aggregation (maybe by using InfluxQL).
about the ingestion “issue” you can find sometihng in the logs:
- check the telegraf log as it should log how many points gets written in each flush
- if that’s not the case, raise the log level or remove the quiet option
- Telegraf’s log don’t usually tell you why a point has been rejected and there is no way of knowing that afaik (if the point has been sent succesfully Telegraf is happy, if the destinations drops it that’s not a its problem)
- Check the InfluxDB log, it logs errors/warnings like points dropped because beyond retention data type conflicts and other stuff
But most importantly, remember that the InfluxDB point key is made of:
tags keys and values + field key + time
if you are writing multiple streams with the same structure you must be able to recognize/separate them (by giving them a proper key) otherwise the values received will just update each other effectively losing data. this is the expected behaviour and you won’t find any log about it.
A quick way of checking that is writing to a text file, I’d suggest using the line-protocol format as it allows to immediately see tags and fields, if some points are sharing the same key, only one of them will be in the DB (the latest one)