Rm error during the migration process

InfluxDB 2
,
1

Hi,
I encounter a few issues during the upgrade process of my influxdb docker 1.8 to 2.0.4

in the upgrade.log all is ok

{“level”:“info”,“ts”:1616033352.2994857,“caller”:“upgrade/upgrade.go:395”,“msg”:“Starting InfluxDB 1.x upgrade”}
{“level”:“info”,“ts”:1616033352.2999074,“caller”:“upgrade/upgrade.go:398”,“msg”:“Upgrading config file”,“file”:“/etc/influxdb/influxdb.conf”}
{“level”:“info”,“ts”:1616033352.39868,“caller”:“upgrade/upgrade.go:402”,“msg”:“Config file upgraded.”,“1.x config”:“/etc/influxdb/influxdb.conf”,“2.x config”:“/etc/influxdb2/config.toml”}
{“level”:“info”,“ts”:1616033352.3988657,“caller”:“upgrade/upgrade.go:412”,“msg”:“Upgrade source paths”,“meta”:“/var/lib/influxdb/meta”,“data”:“/var/lib/influxdb/data”}
{“level”:“info”,“ts”:1616033352.3990054,“caller”:“upgrade/upgrade.go:413”,“msg”:“Upgrade target paths”,“bolt”:“/var/lib/influxdb2/influxd.bolt”,“engine”:“/var/lib/influxdb2/engine”}
{“level”:“info”,“ts”:1616033352.5881813,“caller”:“bolt/bbolt.go:76”,“msg”:“Resources opened”,“service”:“bolt”,“path”:“/var/lib/influxdb2/influxd.bolt”}
{“level”:“info”,“ts”:1616033352.6661072,“caller”:“migration/migration.go:153”,“msg”:“Bringing up metadata migrations”,“service”:“migrations”,“migration_count”:14}
{“level”:“info”,“ts”:1616033362.1642268,“caller”:“upgrade/setup.go:148”,“msg”:“CLI config has been stored.”,“path”:“/etc/influxdb2/influx-configs”}
{“level”:“info”,“ts”:1616033362.164388,“caller”:“upgrade/database.go:201”,“msg”:“Checking available disk space”}
{“level”:“info”,“ts”:1616033363.0342376,“caller”:“upgrade/database.go:222”,“msg”:“Computed disk space”,“free”:“2.3 TB”,“required”:“1.6 GB”}
{“level”:“info”,“ts”:1616033363.0881262,“caller”:“upgrade/database.go:51”,“msg”:“Upgrading databases”}
{“level”:“warn”,“ts”:1616033458.8931978,“caller”:“upgrade/database.go:157”,“msg”:“Empty retention policy”}
{“level”:“info”,“ts”:1616033473.4203157,“caller”:“upgrade/database.go:194”,“msg”:“Database upgrade complete”,“upgraded_count”:4}
{“level”:“info”,“ts”:1616033473.4777935,“caller”:“upgrade/security.go:30”,“msg”:“There are no users in 1.x, nothing to upgrade”}
{“level”:“info”,“ts”:1616033473.47907,“caller”:“upgrade/upgrade.go:482”,“msg”:“Upgrade successfully completed. Start the influxd service now, then log in”,“login_url”:"http://localhost:8086"}

but the container crash with an exit code 1 and I have a lot of this line in my container log

rm: cannot remove ‘/var/lib/influxdb2/engine/data/4d18dcaf0212e7bd/autogen/336/index/4’: Directory not empty
rm: cannot remove ‘/var/lib/influxdb2/engine/data/4d18dcaf0212e7bd/autogen/336/index/5’: Directory not empty
rm: cannot remove ‘/var/lib/influxdb2/engine/data/4d18dcaf0212e7bd/autogen/336/index/3’: Directory not empty
rm: cannot remove ‘/var/lib/influxdb2/engine/data/4d18dcaf0212e7bd/autogen/336/index/6’: Directory not empty
rm: cannot remove ‘/var/lib/influxdb2/engine/data/4d18dcaf0212e7bd/_series/06’: Directory not empty
rm: cannot remove ‘/var/lib/influxdb2/engine/data/4d18dcaf0212e7bd/_series/05’: Directory not empty

When restarting the upgrade process restart and crash in the same way

My parameters are:

DOCKER_INFLUXDB_INIT_USERNAME myuser
DOCKER_INFLUXDB_INIT_PASSWORD mypassword
DOCKER_INFLUXDB_INIT_ORG myorg
DOCKER_INFLUXDB_INIT_BUCKET init-bucket
DOCKER_INFLUXDB_INIT_UPGRADE_V1_CONFIG /etc/influxdb/influxdb.conf
DOCKER_INFLUXDB_INIT_UPGRADE_V1_DIR /var/lib/influxdb
PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GOSU_VER 1.12
INFLUXDB_VERSION 2.0.4
INFLUX_CONFIGS_PATH /etc/influxdb2/influx-configs
INFLUXD_INIT_PORT 9999
DOCKER_INFLUXDB_INIT_MODE upgrade
2

I have been experiencing the same, with Docker and an NFS/SSD backed volume. So far I have found that whilst InfluxDB2 is migrating/starting up it’s leaving lots of .nfs files in the directories it claims it can’t delete, such as ./data/06e53227f17296ba/autogen/6466/index/3/.nfs00000000002e400f0002475c

I see InfluxDB Enterprise (which I’m not running, it’s just the community version) states NFS is not supported, and InfluxDB fails on NFS with "stale NFS file handle" error · Issue #9047 · influxdata/influxdb · GitHub says the same.

If you’re using NFS volumes within Docker like I am, I suspect we’re straight out of luck with regards to InfluxDB being usable.

3

Hello @kylegordon,
I’m afraid I don’t have a solution for you. I’m sharing your question with the InfluxDB team though.

4

@dream @kylegordon in at least one case, the rm errors are a red herring. The real “failure” is that the server takes longer to boot up post-upgrade than the entrypoint.sh is hard-coded to accept: Upgrade error rm: cannot remove '/var/lib/influxdb2/engine/wal/4d18dcaf0212e7bd/autogen/456': Directory not empty · Issue #471 · influxdata/influxdata-docker · GitHub

I’m working on a fix for this today.

1 Like
5

This appears to now be fixed, thank you for working on it @dan-moran

1 Like