"Failed to download shard backup" when running a backup

I’m getting a Failed to download shard backup: An internal error has occurred when trying to run a backup. See the entire printout below.

I’m using the 2.0 OSS docker image (2.0.9) hosted in Azure Container Instances. One thing that might be relevant here is that I have an Azure File Share mounted to my container, and that I have both the Bolt path and engine path pointed to this file share (using --bolt-path=/aci/db/influxd.bolt and --engine-path=/aci/db/engine). Also note that I’m not using the standard port 8086, but rather 9999.

My backup command is /usr/local/bin/influx backup /aci/db/backup --host http://<name>.northeurope.azurecontainer.io:9999 --token <token>.

And here’s the printout:

2021-10-15T05:05:36.220021Z     info    Backing up KV store     {"log_id": "0XCaJ1pl000", "path": "/aci/db/backup/20211015T050536Z.bolt"}
2021-10-15T05:05:36.362044Z     info    Resources opened        {"log_id": "0XCaJ1pl000", "path": "/aci/db/backup/20211015T050536Z.bolt"}
2021-10-15T05:05:36.373678Z     info    Backing up organization {"log_id": "0XCaJ1pl000", "id": "cde2791a53d987d4", "name": "XXXXXX"}
2021-10-15T05:05:36.378281Z     info    Backing up bucket       {"log_id": "0XCaJ1pl000", "id": "50df484acf706ca0", "name": "_monitoring"}
2021-10-15T05:05:36.378399Z     info    Backing up bucket       {"log_id": "0XCaJ1pl000", "id": "9f983ea8987ecaf6", "name": "_tasks"}
2021-10-15T05:05:36.378551Z     info    Backing up bucket       {"log_id": "0XCaJ1pl000", "id": "54885bd02d196ccf", "name": "(some bucket)"}
2021-10-15T05:05:36.378590Z     info    Backing up bucket       {"log_id": "0XCaJ1pl000", "id": "ad8b1a11184c7561", "name": "(another bucket)"}
2021-10-15T05:05:36.378662Z     info    Backing up shard        {"log_id": "0XCaJ1pl000", "id": 332, "path": "/aci/db/backup/20211015T050536Z.s332.tar.gz"}
Error: Failed to download shard backup: An internal error has occurred.
See 'influx backup -h' for help

Thank you!

@MattiasB Azure File Share unfortunately doesn’t support hard-links, which are used in our backup logic. We recently merged a PR that changes our Windows builds to use copies instead of hard-links for semi-related reasons, and have been talking about generalizing the logic to support other situations where links don’t work. I’ll add a link to this post to the relevant issue as a +1

Actually it seems a +1 isn’t needed to convince the team to work on it :grinning_face_with_smiling_eyes: so I’ll just link you to the relevant issue, so you can track progress: Backup failure: Download shard n failed copy backup to file · Issue #16739 · influxdata/influxdb · GitHub

Oh! That explains it. Good to hear that a fix is in the pipeline.

Does this mean that it’s a bad idea in general to have the engine path pointing to an Azure File Share (or perhaps even an NTFS file system)?

I wouldn’t say it’s a bad idea in general, it’s a common use-case we see so it’s something we intend to make work well.

1 Like