Data in Filesystem but not Queryable

I am running InfluxDB on AWS, persisting an EBS volume as the single instance comes and goes. I started with version 1.4.2 and inmem index. The instance was running for several months. When I rerolled the instance (I don’t think this was a graceful shutdown, which may have caused the problem?) and reattached the volume on a new one, all of the previous data was not queryable. It’s almost like Influx thought the start of time was when the new service started.

The data seems dormant in the file system. If I run an influx_inspect export, I can get all the data in line protocol. When I tried to reimport that, it brought the data back, but seemed to cause the filesystem to grow (so guessing it left these “shadow points” of missing data instead of overwriting it).

I have since upgraded to 1.6.1, and switched over to tsi1 index. During that instance reroll, the data issue occurred once again. I have tried to rebuild the indexes using influx_inspect buildtsi, but that didn’t seem to have any affect. On startup, it appears to be loading all the previous shards and takes 5-10 mins, but it seems it just doesn’t let it be queryable.

Does anyone have any pointers or ideas on how I can get Influx to recognize all the data?

Thanks!

In the InfluxDB config file, are all the directory paths pointing to the EBS volume? It sounds like the meta directory is set to a directory on the local boot volume and is being lost every time the EBS volume with the data is moved to a new instance.

The meta directory holds the mapping of databases and shards to the underlying files on disk in the main data directory. If the meta directory is lost, InfluxDB loads the files in the data directory but the data isn’t queryable because it’s not mapped to it’s corresponding database and shard.

The fastest way to re-instate the data would be to dig up the contents of the meta directory from the boot volume of the previous instance. Otherwise, you’ll need to reimport the data again to rebuild the meta info.