Hello,
I am running influxdb-2.7.1 on OpenBSD and every day influxdb crashes right after compacting the TSI.
The log shows that tsdb.ReadSeriesKeyMeasurement fails to access the 3rd item of a list than only contains 2.
Here is the INFO level log that I recovered:
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.080793Z lvl=info msg="Cache snapshot (start)" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=start
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.098220Z lvl=info msg="Cache snapshot (start)" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=start
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.115050Z lvl=info msg="Snapshot for path written" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9 duration=34.257ms
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.115136Z lvl=info msg="Cache snapshot (end)" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=end op_elapsed=34.362ms
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.131786Z lvl=info msg="Snapshot for path written" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10 duration=33.566ms
> ./daemon:Oct 24 13:10:01 matrix influxdb[56502]: ts=2023-10-24T11:10:01.131842Z lvl=info msg="Cache snapshot (end)" log_id=0l487GTG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=end op_elapsed=33.635ms
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081461Z lvl=info msg="TSM compaction (start)" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=start
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081529Z lvl=info msg="Beginning compaction" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=8
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081548Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000001-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081562Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000002-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081574Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000003-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081588Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000004-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081600Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000005-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081612Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000006-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081623Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000007-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.081636Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000008-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101320Z lvl=info msg="TSM compaction (start)" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=start
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101366Z lvl=info msg="Beginning compaction" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=8
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101380Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000001-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101392Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000002-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101404Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000003-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101416Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000004-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101428Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000005-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101440Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000006-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101451Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000007-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.101462Z lvl=info msg="Compacting file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000008-000000001.tsm
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.155109Z lvl=info msg="Compacted file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/influxdb/.influxdbv2/engine/data/937ca33353e67cc1/autogen/9/000000008-000000002.tsm.tmp
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.155168Z lvl=info msg="Finished compacting files" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=1
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.155184Z lvl=info msg="TSM compaction (end)" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=end op_elapsed=73.960ms
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.169039Z lvl=info msg="Compacted file" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/influxdb/.influxdbv2/engine/data/25e9a86d4be24daa/autogen/10/000000008-000000002.tsm.tmp
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.169091Z lvl=info msg="Finished compacting files" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=1
> ./daemon:Oct 24 13:10:02 matrix influxdb[56502]: ts=2023-10-24T11:10:02.169106Z lvl=info msg="TSM compaction (end)" log_id=0l487GTG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=end op_elapsed=67.859ms
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: ts=2023-10-24T11:34:37.379351Z lvl=info msg="TSI log compaction (start)" log_id=0l487GTG000 service=storage-engine index=tsi tsi1_partition=8 op_name=tsi1_compact_log_file tsi1_log_file_id=2 op_event=start
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: ts=2023-10-24T11:34:37.385303Z lvl=info msg="TSI log compaction (end)" log_id=0l487GTG000 service=storage-engine index=tsi tsi1_partition=8 op_name=tsi1_compact_log_file tsi1_log_file_id=2 op_event=end op_elapsed=6.005ms
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: panic: runtime error: index out of range [1] with length 0
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]:
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: goroutine 249206 [running]:
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: encoding/binary.bigEndian.Uint16(...)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: encoding/binary/binary.go:139
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb.ReadSeriesKeyMeasurement(...)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/series_file.go:380
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb.parseSeriesKey({0x4757243cf?, 0x32c14cc?, 0x5c1aa0?}, {0x0, 0x36f2236?, 0x0})
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/series_file.go:417 +0x300
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb.ParseSeriesKey(...)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/series_file.go:400
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb.(*SeriesFile).Series(0x400783bea0?, 0x40078cb7c0?)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/series_file.go:242 +0x30
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).seriesSketches.func1(0x40029a1488?)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1/log_file.go:1055 +0x34
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb.(*SeriesIDSet).ForEach(0x40029a1510?, 0x4007cc1510)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/series_set.go:181 +0xe0
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).seriesSketches(0x4001bcfc20)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1/log_file.go:1054 +0x78
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).CompactTo(0x4001bcfc20, {0xcf83a0?, 0x40082a7a10?}, 0x3?, 0x3?, 0x4001ba1920)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1/log_file.go:869 +0x350
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).compactLogFile(0x4001a25440, 0x4001bcfc20)
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1/partition.go:1285 +0x3ec
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).compact.func1()
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1/partition.go:1030 +0x28
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: created by github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).compact in goroutine 249187
> ./daemon:Oct 24 13:34:37 matrix influxdb[56502]: github.com/influxdata/influxdb/v2/tsdb/index/tsi1/partition.go:1029 +0x2e0
Do you have any idea on a fix or a bypass?
I thank you in advance.
