Influs v2.7.0 from 1.8.10 Upgrade fails - "Structure needs cleaning "

InfluxDB 2
1

I have seemingly successfully installed Influx 2.7.0 on my raspberry pi. And when running the upgrade process, it’s failing with an error around “structure needs cleaning” I have no idea where to start, any help would be appreciated - here’s the output I get :

pi@raspberrypi:~ $ sudo -u influxdb influxd upgrade
{"level":"info","ts":1683389063.2464552,"caller":"upgrade/upgrade.go:401","msg":"Starting InfluxDB 1.x upgrade"}
{"level":"info","ts":1683389063.2467244,"caller":"upgrade/upgrade.go:404","msg":"Upgrading config file","file":"/etc/influxdb/influxdb.conf"}
{"level":"info","ts":1683389063.2472906,"caller":"upgrade/upgrade.go:408","msg":"Config file upgraded.","1.x config":"/etc/influxdb/influxdb.conf","2.x config":"/var/lib/influxdb/.influxdbv2/config.toml"}
{"level":"info","ts":1683389063.2473936,"caller":"upgrade/upgrade.go:418","msg":"Upgrade source paths","meta":"/usb/influxdb/meta","data":"/usb/influxdb/data"}
{"level":"info","ts":1683389063.2474802,"caller":"upgrade/upgrade.go:419","msg":"Upgrade target paths","bolt":"/var/lib/influxdb/.influxdbv2/influxd.bolt","engine":"/var/lib/influxdb/.influxdbv2/engine"}
{"level":"info","ts":1683389063.2697685,"caller":"bolt/bbolt.go:83","msg":"Resources opened","service":"bolt","path":"/var/lib/influxdb/.influxdbv2/influxd.bolt"}
{"level":"info","ts":1683389063.2724388,"caller":"migration/migration.go:175","msg":"Bringing up metadata migrations","service":"migrations","migration_count":20}
> Welcome to InfluxDB 2.0!
? Please type your primary username admin
? Please type your password ********
? Please type your password again ********
? Please type your primary organization name my_org
? Please type your primary bucket name ruuvi2
? Please type your retention period in hours, or 0 for infinite 0
? Setup with these parameters?
  Username:          admin
  Organization:      my_org
  Bucket:            ruuvi2
  Retention Period:  infinite
 Yes
{"level":"info","ts":1683389120.1779838,"caller":"upgrade/setup.go:73","msg":"CLI config has been stored.","path":"/var/lib/influxdb/.influxdbv2/configs"}
{"level":"info","ts":1683389120.1781554,"caller":"upgrade/database.go:202","msg":"Checking available disk space"}
{"level":"info","ts":1683389120.1903756,"caller":"upgrade/database.go:223","msg":"Computed disk space","free":"6.0 GB","required":"3.4 GB"}
? Proceeding will copy all V1 data to "/var/lib/influxdb/.influxdbv2"
  Space available: 6.0 GB
  Space required:  3.4 GB
 Yes
{"level":"info","ts":1683389129.735762,"caller":"upgrade/database.go:51","msg":"Upgrading databases"}
{"level":"info","ts":1683389129.7486033,"caller":"upgrade/database.go:101","msg":"Creating mapping","database":"ruuvi","retention policy":"autogen","orgID":"57de2885eeb31a77","bucketID":"369e2864746be981"}
{"level":"error","ts":1683389236.3776467,"caller":"upgrade/upgrade.go:467","msg":"Database upgrade error, removing data","error":"error copying v1 data from /usb/influxdb/data/ruuvi/autogen to /var/lib/influxdb/.influxdbv2/engine/wal/369e2864746be981/autogen: sync /var/lib/influxdb/.influxdbv2/engine/wal/369e2864746be981/autogen/181/000000003-000000002.tsm: structure needs cleaning","stacktrace":"github.com/influxdata/influxdb/v2/cmd/influxd/upgrade.runUpgradeE\n\t/root/project/cmd/influxd/upgrade/upgrade.go:467\ngithub.com/influxdata/influxdb/v2/cmd/influxd/upgrade.NewCommand.func1\n\t/root/project/cmd/influxd/upgrade/upgrade.go:157\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/root/project/cmd/influxd/main.go:61\nruntime.main\n\t/go/src/runtime/proc.go:250"}
Error: error copying v1 data from /usb/influxdb/data/ruuvi/autogen to /var/lib/influxdb/.influxdbv2/engine/wal/369e2864746be981/autogen: sync /var/lib/influxdb/.influxdbv2/engine/wal/369e2864746be981/autogen/181/000000003-000000002.tsm: structure needs cleaning
See 'influxd -h' for help
2

Hello @Paul_P,
Wow! In all my years I’ve never seen that error before. lolol
It looks like or sounds like the TSM tree got corrupted somehow.

I have no idea how to begin to fix that.
Any chance you can try restarting InfluxDB and trying the process again?
I’m asking around to see if anyone can help.

3

Hi @Anaisdg

That doesn’t fill me with confidence :smiley: What is really weird is that it was running without errors on 1.8.10. I tried stopping and restarting the process and also rebooting it, and nothing really helped. I then also tried rerunning the upgrade process which it didn’t allow me to do. Luckily I have a backup of it, so I can then restore it and start from my “fresh” 1.8.10 database.

Is there perhaps another way that I can then not another way that I can do a backup from 1.8 and then an import into 2.7, I couldn’t find any easy way of doing that. Hopefully one of your colleagues can help me out here - :crossed_fingers: :slightly_smiling_face: