InfluxDB Failed Backup

,
1

Hi, I’m trying to backup my InfluxDB database, and I can’t get it to work… I keep getting the below errors no matter what I try. I was on 1.3.2 but just upgraded to 1.3.5 to see if that would help, but it didn’t seem to.

influxd backup -database production backups/
2017/08/31 19:53:15 backing up db=production since 0001-01-01 00:00:00 +0000 UTC
2017/08/31 19:53:15 backing up metastore to backups/meta.01
2017/08/31 19:53:15 backing up db=production rp=autogen shard=5 to backups/production.autogen.00005.00 since 0001-01-01 00:00:00 +0000 UTC
2017/08/31 19:53:15 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (0)...
2017/08/31 19:53:16 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (1)...
2017/08/31 19:53:17 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (2)...
2017/08/31 19:53:18 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (3)...
2017/08/31 19:53:19 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (4)...
2017/08/31 19:53:20 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (5)...
2017/08/31 19:53:22 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (6)...
2017/08/31 19:53:23 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (7)...
2017/08/31 19:53:24 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (8)...
2017/08/31 19:53:25 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (9)...

Any thoughts?

2

Hmm, I had something similar happen*, I restarted influxdb and it started working again.

(* and I didn’t notice it happened until a week later, showing that my backup script wasn’t being monitored correctly!)

3

Ping! Do we have an authoritative answer from the developers as to why this happens?

Same thing happened to me on v1.4.3rc0 - one of the databases fails to backup on shard #7 - which is “healthy” according to influx_inspect. The only different thing about that particular database vs all other ones is that it has been restored from backup. So backup #1 succeeded, restore succeeded and backup #2 fails.

These messages repeat in log:

Feb 13 20:17:26 pauli influxd[1147]: [I] 2018-02-14T02:17:26Z Snapshot for path /opt/lib/influxdb/data/futures_daily/autogen/7 written in 4.899645ms engine=tsm1
Feb 13 20:17:26 pauli influxd[1147]: [I] 2018-02-14T02:17:26Z error creating tsm hard link: “link /opt/lib/influxdb/data/daily/autogen/7/000000001-000000001.tsm /opt/lib/influxdb/data/daily/autogen/7/1.tmp/000000001-000000001.tsm: operation not permitted” service=snapshot

These 1.tmp, 2.tmp etc directories are created but are indeed empty.