I now have confirmation that InfluxDB has lost data.
A user contacted us to say he was looking at his charts last week and noticed there was no data showing for March, April or May this year (see attached). The chart he is looking is generated from aggregated data and I have been aware for a couple of weeks that there is a problem with aggregated data in this system, specifically for the period March, April and May. I was planning to rebuild the relevant database from the raw measurements, and assumed that this was just another example of that problem.
(In our system raw measurements get loaded into one database and then aggregated into 5- and 30-minute time intervals, into a separate database for each.)
The chart in the attached screenshot was generated on June 7 (we only received notice today, via a third party).
Investigating this problem today (June 13) the same chart shows NO data for January and February even though it is in the screen shot from last week.
Checking the raw (un-aggregated) and aggregated measurement data for this user I see no measurements prior to April 10 in the ‘raw’ database and nothing prior to May 31 in the two ‘aggregated’ databases. Yet we know for a fact that this user should have data going back to December last year.
Thinking this might be a case of in-memory index problems, I have done a clean shutdown of our application followed by a clean shutdown of Influx and then rebooted the server, but this did not bring any measurement data back.
I have now run
influx_inspect to dump the ‘raw’ measurement database for January and there are NO entries for this customer in the dump. (The dump is 17G in size and contains lots of other data.)
How can we diagnose this further to work out what is going on?