InfluxDB 2.2.0 fatal error: concurrent map read and map write

Hey,

We use “nimon” to load data from nearly 100 systems directly into InfluxDB. Unfortunately, fatal errors occur in the process. The more systems report into InfluxDB, the more such errors occur (currently around 60 errors per day). Has anyone had such an error before or knows how to fix the problem?

Environment info:

  • Red Hat Enterprise Linux 8.4
  • InfluxDB Version: InfluxDB 2.2.0
  • Build date: 2022-04-05

Errors in the log:
Nov 16 10:21:39 influxvm influxd-systemd-start.sh[127277]: ts=2022-11-16T09:21:39.322620Z lvl=info msg=“Snapshot for path written” log_id=0eBsHbvW000 service=storage-engine engine=tsm1 op_name=tsm1_c
ache_snapshot path=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638 duration=2591.686ms
Nov 16 10:21:39 influxvm influxd-systemd-start.sh[127277]: ts=2022-11-16T09:21:39.322684Z lvl=info msg=“Cache snapshot (end)” log_id=0eBsHbvW000 service=storage-engine engine=tsm1 op_name=tsm1_cache_
snapshot op_event=end op_elapsed=2591.758ms
Nov 16 10:21:39 influxvm influxd-systemd-start.sh[127277]: ts=2022-11-16T09:21:39.322700Z lvl=info msg=“Cache snapshot (start)” log_id=0eBsHbvW000 service=storage-engine engine=tsm1 op_name=tsm1_cach
e_snapshot op_event=start
Nov 16 10:21:44 influxvm influxd-systemd-start.sh[127277]: ts=2022-11-16T09:21:44.351961Z lvl=info msg=“Snapshot for path written” log_id=0eBsHbvW000 service=storage-engine engine=tsm1 op_name=tsm1_c
ache_snapshot path=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638 duration=5029.256ms
Nov 16 10:21:44 influxvm influxd-systemd-start.sh[127277]: ts=2022-11-16T09:21:44.352157Z lvl=info msg=“Cache snapshot (end)” log_id=0eBsHbvW000 service=storage-engine engine=tsm1 op_name=tsm1_cache_
snapshot op_event=end op_elapsed=5029.460ms
Nov 16 10:21:44 influxvm influxd-systemd-start.sh[127277]: ts=2022-11-16T09:21:44.352175Z lvl=info msg=“Cache snapshot (start)” log_id=0eBsHbvW000 service=storage-engine engine=tsm1 op_name=tsm1_cach
e_snapshot op_event=start
Nov 16 10:21:45 influxvm influxd-systemd-start.sh[127277]: ts=2022-11-16T09:21:45.741979Z lvl=info msg=“Cache snapshot (start)” log_id=0eBsHbvW000 service=storage-engine engine=tsm1 op_name=tsm1_cach
e_snapshot op_event=start
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: fatal error: concurrent map read and map write
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 6251450 [running]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: runtime.throw({0x103c71351, 0x21})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /usr/local/go/src/runtime/panic.go:1198 +0x68 fp=0xc0a5a695c0 sp=0xc0a5a69580 pc=0x101b3d628
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: runtime.mapaccess1_faststr(0x10544b260, 0xc0356c0de0, {0xc0b97e6300, 0xb6})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /usr/local/go/src/runtime/map_faststr.go:21 +0x490 fp=0xc0a5a69640 sp=0xc0a5a695c0 pc=0x101b14f60
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/tsdb/engine/tsm1.(*partition).write(0xc0718f6320, {0xc0b97e6300, 0xb6, 0xc0}, {0xc0b93e7040, 0x1, 0x1})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/engine/tsm1/ring.go:225 +0xc0 fp=0xc0a5a696c0 sp=0xc0a5a69640 pc=0x10298dea0
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/tsdb/engine/tsm1.(*ring).write(0xc001faa7c8, {0xc0b97e6300, 0xb6, 0xc0}, {0xc0b93e7040, 0x1, 0x1})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/engine/tsm1/ring.go:95 +0xb4 fp=0xc0a5a69730 sp=0xc0a5a696c0 pc=0x10298cff4
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/tsdb/engine/tsm1.(*Cache).WriteMulti(0xc023ef4120, 0xc06fa7b590)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/engine/tsm1/cache.go:343 +0x3cc fp=0xc0a5a698a8 sp=0xc0a5a69730 pc=0x102919f2c
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/tsdb/engine/tsm1.(*Engine).WritePoints(0xc01989e280, {0x10583ee00, 0xc0ff4acc00}, {0xc0b912e000, 0x1e9, 0x
1e9})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1344 +0x11c fp=0xc0a5a69a98 sp=0xc0a5a698a8 pc=0x1029499ac
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/tsdb.(*Shard).WritePoints(0xc035656240, {0x10583ee00, 0xc0ff4acc00}, {0xc0b912e000, 0x1e9, 0x1e9})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/shard.go:675 +0x2f8 fp=0xc0a5a69b98 sp=0xc0a5a69a98 pc=0x102882428
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/tsdb.(*Store).WriteToShard(0xc000c8c000, {0x10583ee00, 0xc0ff4acc00}, 0x19ee, {0xc0b912e000, 0x1e9, 0x1e9}
)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/store.go:1497 +0x2cc fp=0xc0a5a69c50 sp=0xc0a5a69b98 pc=0x10289b76c
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/v1/coordinator.(*PointsWriter).writeToShard(0xc000c8bc20, {0x10583ee00, 0xc0ff4acc00}, 0xc0b91355a0, {0xc0
adc17520, 0x10}, {0x103c16d9f, 0x7}, {0xc0b912e000, 0x1e9, …})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/v1/coordinator/points_writer.go:431 +0x88 fp=0xc0a5a69e68 sp=0xc0a5a69c50 pc=0x1031a2828
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/v1/coordinator.(*PointsWriter).WritePointsPrivileged.func1(0xc000c8bc20, {0x10583ee00, 0xc0ff4acc00}, 0xc0
32a12f00, 0xc0b91355a0, {0xc0adc17520, 0x10}, {0x103c16d9f, 0x7}, {0xc0b912e000, …})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/v1/coordinator/points_writer.go:393 +0x94 fp=0xc0a5a69f60 sp=0xc0a5a69e68 pc=0x1031a25e4
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: runtime.goexit()
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /usr/local/go/src/runtime/asm_ppc64x.s:855 +0x4 fp=0xc0a5a69f60 sp=0xc0a5a69f60 pc=0x101b784a4
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by github/influxdata/influxdb/v2/v1/coordinator.(*PointsWriter).WritePointsPrivileged
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/v1/coordinator/points_writer.go:392 +0x34c
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 1 [chan receive, 77 minutes]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/cmd/influxd/launcher.cmdRunE.func1()
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/cmd/influxd/launcher/cmd.go:127 +0x394
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/kit/cli.NewCommand.func1(0xc000b16dc0, {0x10c36ac40, 0x0, 0x0})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/kit/cli/viper.go:54 +0x48
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/spf13/cobra.(*Command).execute(0xc000b16dc0, {0xc00004e210, 0x0, 0x0})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/pkg/mod/github/spf13/cobra@v1.0.0/command.go:842 +0x748
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/spf13/cobra.(*Command).ExecuteC(0xc000b16dc0)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/pkg/mod/github/spf13/cobra@v1.0.0/command.go:950 +0x464
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/spf13/cobra.(*Command).Execute(…)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/pkg/mod/github/spf13/cobra@v1.0.0/command.go:887
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: main.main()
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/cmd/influxd/main.go:61 +0x61c
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 74 [chan receive, 78 minutes]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/flux/execute.createOperatorProfiler.func1(0xc000aae110)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/pkg/mod/github/influxdata/flux@v0.161.0/execute/profiler.go:108 +0xf8
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by github/influxdata/flux/execute.createOperatorProfiler
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/pkg/mod/github/influxdata/flux@v0.161.0/execute/profiler.go:106 +0xe4
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 75 [select]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: go.opencensus.io/stats/view.(*worker).start(0xc00015e180)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xc8
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by go.opencensus.io/stats/view.init.0
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x9c
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 54 [syscall, 78 minutes]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: os/signal.signal_recv()
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /usr/local/go/src/runtime/sigqueue.go:169 +0x128
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: os/signal.loop()
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /usr/local/go/src/os/signal/signal_unix.go:24 +0x38
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by os/signal.Notify.func1.1
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /usr/local/go/src/os/signal/signal.go:151 +0x50
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 55 [select, 78 minutes]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/kit/signals.WithSignals.func1(0xc002006700, {0x10583ed58, 0xc0017c9b40}, 0xc002db87e0)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/kit/signals/context.go:17 +0xb8
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by github/influxdata/influxdb/v2/kit/signals.WithSignals
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/kit/signals/context.go:15 +0xc8
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 57 [select, 19 minutes]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/bolt.(*pluginMetricsCollector).pollTelegrafStats(0xc0014117a0, 0xc000138900)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/bolt/metrics.go:109 +0xa4
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by github/influxdata/influxdb/v2/bolt.(*pluginMetricsCollector).Open
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/bolt/metrics.go:104 +0x5c
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 59 [select, 78 minutes]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: database/sql.(*DB).connectionOpener(0xc000b3b040, {0x10583ed58, 0xc0000fc540})
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /usr/local/go/src/database/sql/sql.go:1196 +0xc4
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by database/sql.OpenDB
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /usr/local/go/src/database/sql/sql.go:794 +0x16c
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 1360 [select, 18 minutes]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).runPeriodicCompaction(0xc001f767e0)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/index/tsi1/partition.go:911 +0x160
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by github/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).Open
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/index/tsi1/partition.go:243 +0x710
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 2091 [select, 18 minutes]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).runPeriodicCompaction(0xc02af7b560)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/index/tsi1/partition.go:911 +0x160
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by github/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).Open
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/index/tsi1/partition.go:243 +0x710
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 1829 [select, 18 minutes]:
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).runPeriodicCompaction(0xc012924fc0)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/index/tsi1/partition.go:911 +0x160
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by github/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).Open
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/index/tsi1/partition.go:243 +0x710
[…]
[…]
[…]
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: github/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).runPeriodicCompaction(0xc022c15680)
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/index/tsi1/partition.go:911 +0x160
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: created by github/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).Open
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: /root/go/src/github/influxdata/influxdb/tsdb/index/tsi1/partition.go:243 +0x710
Nov 16 10:21:47 influxvm influxd-systemd-start.sh[127277]: goroutine 4450 [select, 18 minutes]:
Nov 16 10:21:47 influxvm systemd[1]: influxdb.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 16 10:21:47 influxvm systemd[1]: influxdb.service: Failed with result ‘exit-code’.
Nov 16 10:21:48 influxvm systemd[1]: influxdb.service: Service RestartSec=100ms expired, scheduling restart.
Nov 16 10:21:48 influxvm systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 2434.
Nov 16 10:21:48 influxvm systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Nov 16 10:21:48 influxvm systemd[1]: Starting InfluxDB is an open-source, distributed, time series database…
Nov 16 10:21:50 influxvm influxd-systemd-start.sh[144031]: InfluxDB API at localhost:8086/ready unavailable after 3 attempts…
Nov 16 10:21:50 influxvm systemd-journald[1539]: Suppressed 17710 messages from influxdb.service
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.233532Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5999 error=“[shard 5999] stat /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5999/index: permission denied”
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.233791Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5947 error=“[shard 5947] stat /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5947/index: permission denied”
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.233930Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5957 error=“[shard 5957] stat /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5957/index: permission denied”
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.234258Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5960 error=“[shard 5960] stat /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5960/index: permission denied”
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.234604Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5954 error=“[shard 5954] open /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5954/index/0/L2-00000004.tsi: no such file or directory”
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.234976Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5975 error=“[shard 5975] open /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5975/index/0/L1-00000001.tsi: no such file or directory”
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.235103Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5951 error=“[shard 5951] open /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5951/index/0/L1-00000001.tsi: no such file or directory”
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.235298Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5924 error=“[shard 5924] open /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5924/index/1/L1-00000001.tsi: no such file or directory”
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.235306Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5931 error=“[shard 5931] open /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5931/index/1/L1-00000001.tsi: no such file or directory”
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.235292Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5961 error=“[shard 5961] open /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5961/index/0/L2-00000004.tsi: no such file or directory”
Nov 16 10:21:51 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:51.235381Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5991 error=“[shard 5991] stat /var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/5991/index: permission denied”
[…]
Nov 16 10:21:55 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:55.405070Z lvl=info msg=“Opened file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 service=filestore path=/var/
lib/influxdb/engine/data/8305eaa97bc97328/autogen/6596/000002653-000000002.tsm id=0 duration=231.432ms
Nov 16 10:21:55 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:55.405388Z lvl=info msg=“Opened shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_open index_v
ersion=tsi1 path=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6596 duration=607.578ms
Nov 16 10:21:55 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:55.411194Z lvl=info msg=“Opened file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 service=filestore path=/var/
lib/influxdb/engine/data/8305eaa97bc97328/autogen/6316/000002769-000000003.tsm id=0 duration=324.840ms
Nov 16 10:21:55 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:55.420441Z lvl=info msg=“Opened file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 service=filestore path=/var/
lib/influxdb/engine/data/8305eaa97bc97328/autogen/6316/000002769-000000005.tsm id=2 duration=342.334ms
Nov 16 10:21:55 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:21:55.440467Z lvl=error msg=“Failed to open shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_ope
n db_shard_id=5918 error=“[shard 5918] open /var/lib/influxdb/engine/data/83aca633621a27bd/autogen/5918/index/2/L1-00000001.tsi: no such file or directory”
[…]
[…]
[…]
Nov 16 10:21:59 influxvm influxd-systemd-start.sh[144031]: InfluxDB API at localhost:8086/ready unavailable after 12 attempts…
Nov 16 10:22:00 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:00.113341Z lvl=info msg=“Opened shard” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_open index_v
ersion=tsi1 path=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638 duration=4896.037ms
Nov 16 10:22:00 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:00.640213Z lvl=info msg=“Open store (end)” log_id=0eBwkycl000 service=storage-engine service=store op_name=tsdb_open op_
event=end op_elapsed=12177.230ms
Nov 16 10:22:00 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:00.640270Z lvl=info msg=“Starting retention policy enforcement service” log_id=0eBwkycl000 service=retention check_inter
val=30m
Nov 16 10:22:00 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:00.640290Z lvl=info msg=“Starting precreation service” log_id=0eBwkycl000 service=shard-precreation check_interval=10m a
dvance_period=30m
Nov 16 10:22:00 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:00.641222Z lvl=info msg=“Starting query controller” log_id=0eBwkycl000 service=storage-reads concurrency_quota=1024 init
ial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
Nov 16 10:22:00 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:00.643481Z lvl=info msg=“Configuring InfluxQL statement executor (zeros indicate unlimited).” log_id=0eBwkycl000 max_sel
ect_point=0 max_select_series=0 max_select_buckets=0
Nov 16 10:22:00 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:00.657668Z lvl=info msg=Starting log_id=0eBwkycl000 service=telemetry interval=8h
Nov 16 10:22:00 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:00.657921Z lvl=info msg=Listening log_id=0eBwkycl000 service=tcp-listener transport=http addr=:8086 port=8086
Nov 16 10:22:00 influxvm influxd-systemd-start.sh[144031]: InfluxDB started
Nov 16 10:22:00 influxvm systemd[1]: Started InfluxDB is an open-source, distributed, time series database.
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.113886Z lvl=info msg=“Cache snapshot (start)” log_id=0eBwkycl000 service=storage-engine engine=tsm1 op_name=tsm1_cach
e_snapshot op_event=start
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.115375Z lvl=info msg=“TSM compaction (start)” log_id=0eBwkycl000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1
strategy=level op_name=tsm1_compact_group op_event=start
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.115400Z lvl=info msg=“Beginning compaction” log_id=0eBwkycl000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_s
trategy=level op_name=tsm1_compact_group tsm1_files_n=8
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.115408Z lvl=info msg=“Compacting file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strate
gy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638/000001509-000000001.tsm
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.115416Z lvl=info msg=“Compacting file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strate
gy=level op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638/000001510-000000001.tsm
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.115424Z lvl=info msg=“Compacting file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strate
gy=level op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638/000001511-000000001.tsm
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.115432Z lvl=info msg=“Compacting file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strate
gy=level op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638/000001512-000000001.tsm
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.115439Z lvl=info msg=“Compacting file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strate
gy=level op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638/000001513-000000001.tsm
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.115447Z lvl=info msg=“Compacting file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strate
gy=level op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638/000001514-000000001.tsm
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.115455Z lvl=info msg=“Compacting file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strate
gy=level op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638/000001515-000000001.tsm
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.115463Z lvl=info msg=“Compacting file” log_id=0eBwkycl000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strate
gy=level op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/var/lib/influxdb/engine/data/8305eaa97bc97328/autogen/6638/000001516-000000001.tsm
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.327739Z lvl=info msg=“Cache snapshot (start)” log_id=0eBwkycl000 service=storage-engine engine=tsm1 op_name=tsm1_cach
e_snapshot op_event=start
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.848974Z lvl=info msg=“Snapshot for path written” log_id=0eBwkycl000 service=storage-engine engine=tsm1 op_name=tsm1_c
ache_snapshot path=/var/lib/influxdb/engine/data/83aca633621a27bd/autogen/6639 duration=521.231ms
Nov 16 10:22:01 influxvm influxd-systemd-start.sh[144031]: ts=2022-11-16T09:22:01.849027Z lvl=info msg=“Cache snapshot (end)” log_id=0eBwkycl000 service=storage-engine engine=tsm1 op_name=tsm1_cache

Kind regards
Loeni01