We are attempting to migrate an InfluxDB to a different EC2 instance using backup and restore tools in the following steps:
- We created an EC2 instance of the same size and ran the ‘influx backup’ command on the original database.
- Copied the created backup files to the new instance.
- On the new instance we ran an ‘influx restore’ on the files. There were a total of three databases restored.
The versions on the databases are the same, both are running 1.8.0 .
The problem:
Two of the databases are working with no issues. However, one of the databases, the largest one, has significant delays on query speed. Selecting the last value from a measurement takes 5-10 minutes to return. The same query on the original EC2 instance takes under a second.
While the query runs, the CPU utilization on the instance is close to 100%. No other processes are running on this instance.
We have tried the following:
- Loading less data, taking a 3 months chunk, but the problem persists.
- Disabling log storing to improve performance.
- Loading just one shard of data → query speed is improved. Then creating another database from a second shard and trying to copy the second shard into the first one → query speed > 30 minutes and still running.
We are at this point stuck as to how to proceed and are hoping to get this done by the end of the week. Are there any recommendations or solutions for this issue? Closest similar issue we found was here ( InfluxDB high CPU usage for one simple query · Issue #9354 · influxdata/influxdb · GitHub), but no solution is posted.
