InfluxDB Failed Backup

Hi, I’m trying to backup my InfluxDB database, and I can’t get it to work… I keep getting the below errors no matter what I try. I was on 1.3.2 but just upgraded to 1.3.5 to see if that would help, but it didn’t seem to.

influxd backup -database production backups/
2017/08/31 19:53:15 backing up db=production since 0001-01-01 00:00:00 +0000 UTC
2017/08/31 19:53:15 backing up metastore to backups/meta.01
2017/08/31 19:53:15 backing up db=production rp=autogen shard=5 to backups/production.autogen.00005.00 since 0001-01-01 00:00:00 +0000 UTC
2017/08/31 19:53:15 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (0)...
2017/08/31 19:53:16 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (1)...
2017/08/31 19:53:17 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (2)...
2017/08/31 19:53:18 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (3)...
2017/08/31 19:53:19 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (4)...
2017/08/31 19:53:20 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (5)...
2017/08/31 19:53:22 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (6)...
2017/08/31 19:53:23 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (7)...
2017/08/31 19:53:24 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (8)...
2017/08/31 19:53:25 Download shard 5 failed copy backup to file: err=<nil>, n=0.  Retrying (9)...

Any thoughts?

Hmm, I had something similar happen*, I restarted influxdb and it started working again.

(* and I didn’t notice it happened until a week later, showing that my backup script wasn’t being monitored correctly!)

Ping! Do we have an authoritative answer from the developers as to why this happens?

Same thing happened to me on v1.4.3rc0 - one of the databases fails to backup on shard #7 - which is “healthy” according to influx_inspect. The only different thing about that particular database vs all other ones is that it has been restored from backup. So backup #1 succeeded, restore succeeded and backup #2 fails.

These messages repeat in log:

Feb 13 20:17:26 pauli influxd[1147]: [I] 2018-02-14T02:17:26Z Snapshot for path /opt/lib/influxdb/data/futures_daily/autogen/7 written in 4.899645ms engine=tsm1
Feb 13 20:17:26 pauli influxd[1147]: [I] 2018-02-14T02:17:26Z error creating tsm hard link: “link /opt/lib/influxdb/data/daily/autogen/7/000000001-000000001.tsm /opt/lib/influxdb/data/daily/autogen/7/1.tmp/000000001-000000001.tsm: operation not permitted” service=snapshot

These 1.tmp, 2.tmp etc directories are created but are indeed empty.