Fatal error during rebuild DB

After a power outage, my influx database was no longer working. I tried to rebuild the index:

systemctl stop influxdb
rm -r /var/lib/influxdb/data///*/index
su --command “influx_inspect buildtsi -datadir /var/lib/influxdb/data -waldir /var/lib/influxdb/wal” influxdb

goroutine 1 [running]:
runtime.throw(0x6f2e43, 0x5)
/usr/local/go/src/runtime/panic.go:774 +0x5c fp=0x2c9ea04 sp=0x2c9e9f0 pc=0x3f948
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:391 +0x378 fp=0x2c9ea1c sp=0x2c9ea04 pc=0x55220
githubcom/influxdata/influxdb/tsdb.ReadSeriesEntry(0xa549f000, 0x3ca000, 0x3ca000, 0x35ff7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, …)
/go/src/githubcom/influxdata/influxdb/tsdb/series_segment.go:417 +0x20 fp=0x2c9ea54 sp=0x2c9ea20 pc=0x4b4af8
githubcom/influxdata/influxdb/tsdb.(*SeriesSegment).ForEachEntry(0x2c241b0, 0x2c9eaa8, 0x2, 0x0)
/go/src/githubcom/influxdata/influxdb/tsdb/series_segment.go:244 +0x64 fp=0x2c9ea8c sp=0x2c9ea54 pc=0x4b4174
githubcom/influxdata/influxdb/tsdb.(*SeriesSegment).MaxSeriesID(0x2c241b0, 0x0, 0x0)
/go/src/githubcom/influxdata/influxdb/tsdb/series_segment.go:232 +0x50 fp=0x2c9eab0 sp=0x2c9ea8c pc=0x4b40e8
githubcom/influxdata/influxdb/tsdb.(*SeriesPartition).openSegments(0x2c7e070, 0x0, 0x0)
/go/src/githubcom/influxdata/influxdb/tsdb/series_partition.go:124 +0x2a4 fp=0x2c9eb24 sp=0x2c9eab0 pc=0x4b0260
githubcom/influxdata/influxdb/tsdb.(*SeriesPartition).Open.func1(0x2c7e070, 0x2b, 0x1ff)
/go/src/githubcom/influxdata/influxdb/tsdb/series_partition.go:78 +0x1c fp=0x2c9eb58 sp=0x2c9eb24 pc=0x4b96e8
githubcom/influxdata/influxdb/tsdb.(*SeriesPartition).Open(0x2c7e070, 0x2c24150, 0x1)
/go/src/githubcom/influxdata/influxdb/tsdb/series_partition.go:95 +0x90 fp=0x2c9eb80 sp=0x2c9eb58 pc=0x4aff14
githubcom/influxdata/influxdb/tsdb.(*SeriesFile).Open(0x2c97d40, 0x0, 0x0)
/go/src/githubcom/influxdata/influxdb/tsdb/series_file.go:91 +0x260 fp=0x2c9ed38 sp=0x2c9eb80 pc=0x4ab6a4
githubcom/influxdata/influxdb/cmd/influx_inspect/buildtsi.(*Command).processDatabase(0x2ca8cd0, 0x2c94917, 0x9, 0x2c94960, 0x20, 0x2c94980, 0x1f, 0x0, 0x0)
/go/src/githubcom/influxdata/influxdb/cmd/influx_inspect/buildtsi/buildtsi.go:257 +0x154 fp=0x2c9ee38 sp=0x2c9ed38 pc=0x5b78a0
githubcom/influxdata/influxdb/cmd/influx_inspect/buildtsi.(*Command).run(0x2ca8cd0, 0xbecc7e18, 0x16, 0xbecc7e37, 0x15, 0x0, 0x7)
/go/src/githubcom/influxdata/influxdb/cmd/influx_inspect/buildtsi/buildtsi.go:126 +0x2cc fp=0x2c9eec8 sp=0x2c9ee38 pc=0x5b65f4
githubcom/influxdata/influxdb/cmd/influx_inspect/buildtsi.(*Command).Run(0x2ca8cd0, 0x2c98010, 0x4, 0x4, 0x8, 0x2c98010)
/go/src/githubcom/influxdata/influxdb/cmd/influx_inspect/buildtsi/buildtsi.go:81 +0x4b0 fp=0x2c9eef8 sp=0x2c9eec8 pc=0x5b6234
main.(*Main).Run(0x2c9ef88, 0x2c98010, 0x4, 0x4, 0x0, 0x14f01)
/go/src/githubcom/influxdata/influxdb/cmd/influx_inspect/main.go:94 +0xb7c fp=0x2c9ef58 sp=0x2c9eef8 pc=0x5e0c88
main.main()
/go/src/githubcom/influxdata/influxdb/cmd/influx_inspect/main.go:29 +0x110 fp=0x2c9efa4 sp=0x2c9ef58 pc=0x5e0038
runtime.main()
/usr/local/go/src/runtime/proc.go:203 +0x208 fp=0x2c9efe4 sp=0x2c9efa4 pc=0x41c28
runtime.goexit()
/usr/local/go/src/runtime/asm_arm.s:868 +0x4 fp=0x2c9efe4 sp=0x2c9efe4 pc=0x6dd38

InfluxDB shell version: 1.8.10

any advice for me?
Ralf

Hello,
Did you double check to make sure your paths are correct?
Hmm those logs aren’t very useful huh. Let me see if anyone else knows.

1 Like

Hi @Ralf_L,
I see in your commands that you are following this guide: https://docs.influxdata.com/influxdb/v1/administration/rebuild-tsi-index/

Did you also carry out step 2? As I see you are deleting the index files but not the _series directories

Many thanks,
Jay

1 Like

Hi Jay,
you’re right, haven’t deleted these folders yet. I followed another instruction that didn’t include this step.
After I deleted the _series directories, the rebuilding of the index worked and the database is running again.
Thank you very much for your help!

1 Like

Ah glad to hear it!!

1 Like