Lvl=error msg="Failed to open shard"; "cannot allocate memory"

InfluxDB 1
1

I have started getting this error from my influxdb enterprise v1.10.

Aug 06 08:49:30 abc.xyz.com influxd[4429]: ts=2024-08-06T14:49:30.953236Z lvl=info msg=“Opened shard” log_id=0qr2VH5W000 service=store trace_id=0qr2VHB0000 op_name=tsdb_open index_version=tsi1 path=/pgdata/influxdata/data/tsesre_gms/testbucket/88817 duration=9.208ms
Aug 06 08:49:30 abc.xyz.com influxd[4429]: ts=2024-08-06T14:49:30.953425Z lvl=error msg=“Failed to open shard” log_id=0qr2VH5W000 service=store trace_id=0qr2VHB0000 op_name=tsdb_open db_shard_id=88727 error=“[shard 88727] cannot allocate memory”

And because of this issue telegraf agents are unable to write to it.

Seems like too many shards are getting created.

Currently the influxdb is having to manage 93,539 shards and I’m not sure if that is part of what’s overwhelming the system.

2

The error would indicated that you’re running out of memory. Do you also see that in your system resource metrics?

influxdb is having to manage 93,539 shards

What is your shard-group-duration? Is data not being expired? Are these all hot (uncompacted) shards or does this include cold (compacted) shards?