I’m using Influxdb v2.6.1 on a Raspberry pi 4b with a 64bit system and RPi OS. The Influxdb database has grown to about 2GB on disk since 2020.
Yesterday evening I made a mistake and accidentally deleted half my /var/lib/influxdb
folder. I was able to restore from a backup and everything is now back running again (it seems), but despite NOT deleting recent data files from the data directory, I’m missing about 40 hours of data, specifically data from May 10, 07:00 until May 11, 23:00. Example:
However, the data must be there, because - as luck would have it - I did not delete recent data files (.TSM) from the influxdb folder, just ones with modification timestamps >1 week ago (and restored these from a backup after stopping influxd
). I also know the data is “in there somehow”, looking at them with a hex editor will show the measurement names I’m looking for. Example:
/var/lib/influxdb/engine/data/xxxxx/autogen/1170# ls -la
insgesamt 29252
drwxr-x--- 3 influxdb influxdb 4096 11. Mai 22:01 .
drwx------ 654 influxdb influxdb 12288 11. Mai 22:57 ..
-rw-r----- 1 influxdb influxdb 29913959 11. Mai 11:50 000000008-000000002.tsm
-rw-r----- 1 influxdb influxdb 14719 11. Mai 22:01 fields.idx
drwxr-x--- 10 influxdb influxdb 4096 1. Mai 02:00 index
/var/lib/influxdb/engine/wal/xxxxx/autogen/1178# ls -l
insgesamt 19204
-rw-r----- 1 influxdb influxdb 10485785 12. Mai 08:16 _00001.wal
-rw-r----- 1 influxdb influxdb 9165042 12. Mai 13:51 _00002.wal
There are also two .WAL files with (it seems) recent data which InfluxDB did not commit to the main index (yet?), and whose data does not show up when querying.
How do I get this data back into my reports and Grafana dashboards?
Can I force Influx somehow to recreate its indexes? Or do I just need to wait until Influx commits the WAL files into the main data store as this will update indexes anyway?
Can I manually export the data to line protocol and reimport it? (I tried influxd inspect export-lp ...
but this returns zero datapoints in the respective time interval too.)
In any case, will Influx continue working normally (it seems it’s doing so right now) after this kind of restore action, or do I need to watch out for silent corruption? If so, how can I test whether this is the case? There seemed to be nothing suspicious in the log files when starting Influx, and I only restored data files from backups that (according to timestamps) hadn’t been modified in weeks.
Thank you!