Hi Influx Community,
I’m running an influxdb 1.8.0 docker container and on a Ubuntu 20.04.1 LTS system. In production I insert data with a influx-client, implemented in NodeJS. When I make a backup, I use the incremental backup command, like this:
root@7aaa716f23a0:/# influxd backup -portable -db EXAMPLE_DATABASE -start 2021-02-04T06:00:00Z -end 2021-02-04T09:00:00Z -host influxdb:8088 "/var/backups/temp"
2021/02/11 16:08:14 backing up metastore to /var/backups/temp/meta.00
2021/02/11 16:08:14 backing up db=EXAMPLE_DATABASE
2021/02/11 16:08:14 backing up db=EXAMPLE_DATABASE rp=autogen shard=184 to /var/backups/temp/EXAMPLE_DATABASE.autogen.00184.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:08:14 backing up db=EXAMPLE_DATABASE rp=autogen shard=185 to /var/backups/temp/EXAMPLE_DATABASE.autogen.00185.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:08:14 backup complete:
2021/02/11 16:08:14 /var/backups/temp/20210211T160814Z.meta
2021/02/11 16:08:14 /var/backups/temp/20210211T160814Z.s184.tar.gz
2021/02/11 16:08:14 /var/backups/temp/20210211T160814Z.s185.tar.gz
2021/02/11 16:08:14 /var/backups/temp/20210211T160814Z.manifest
Restoring this backup works. But the backup contains incomplete data. It only contains 2 field keys with data instead of 25 field keys.
I tried to vary the start and end time. With a time interval with 3 days or sometimes more I achieved an “entire” backup.
Another thing I tried was to query all of the data from the database with an NodeJS-Client and write it into a temporary database. Then Influx stores the data in 10 shards - instead of 2 shards like the backed up database.
Backing up and restore now this temporary database works like I expect it!
root@7aaa716f23a0:/# influxd backup -portable -db TEMP_DATABASE -start 2021-02-04T06:00:00Z -end 2021-02-04T09:00:00Z -host influxdb:8088 "/var/backups/temp"
2021/02/11 16:02:19 backing up metastore to /var/backups/temp/meta.00
2021/02/11 16:02:19 backing up db=TEMP_DATABASE
2021/02/11 16:02:19 backing up db=TEMP_DATABASE rp=autogen shard=114 to /var/backups/temp/TEMP_DATABASE.autogen.00114.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:02:19 backing up db=TEMP_DATABASE rp=autogen shard=118 to /var/backups/temp/TEMP_DATABASE.autogen.00118.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:02:19 backing up db=TEMP_DATABASE rp=autogen shard=124 to /var/backups/temp/TEMP_DATABASE.autogen.00124.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:02:19 backing up db=TEMP_DATABASE rp=autogen shard=119 to /var/backups/temp/TEMP_DATABASE.autogen.00119.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:02:19 backing up db=TEMP_DATABASE rp=autogen shard=121 to /var/backups/temp/TEMP_DATABASE.autogen.00121.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:02:19 backing up db=TEMP_DATABASE rp=autogen shard=126 to /var/backups/temp/TEMP_DATABASE.autogen.00126.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:02:19 backing up db=TEMP_DATABASE rp=autogen shard=130 to /var/backups/temp/TEMP_DATABASE.autogen.00130.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:02:19 backing up db=TEMP_DATABASE rp=autogen shard=138 to /var/backups/temp/TEMP_DATABASE.autogen.00138.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:02:19 backing up db=TEMP_DATABASE rp=autogen shard=146 to /var/backups/temp/TEMP_DATABASE.autogen.00146.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:02:19 backing up db=TEMP_DATABASE rp=autogen shard=156 to /var/backups/temp/TEMP_DATABASE.autogen.00156.00 with boundaries start=2021-02-04T06:00:00Z, end=2021-02-04T09:00:00Z
2021/02/11 16:02:21 backup complete:
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.meta
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.s114.tar.gz
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.s118.tar.gz
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.s124.tar.gz
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.s119.tar.gz
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.s121.tar.gz
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.s126.tar.gz
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.s130.tar.gz
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.s138.tar.gz
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.s146.tar.gz
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.s156.tar.gz
2021/02/11 16:02:21 /var/backups/temp/20210211T160219Z.manifest
The backup contains all field keys in the correct time span.
I assumed that the amount of shards could be a reason. I checked the retention policies…They are in both cases the same - the autogen with the 7days shard duration…
Did anybody obseve this behavior before? Why I’m not able to back up a small time intervall in my original database?
Cheers Marius
