Seeing huge memory spikes roughly every hour(see Image) which results in OOM and Influx being killed.
Memory is increasing from 14Gb to 47GB and causing OOM killer to kick in and kill Influxdb
Problem can be rerpoduced by doing a 'drop shard ’
Retention policy is below;-
show retention policies;
name duration shardGroupDuration replicaN default
cmk_retention 2160h0m0s 24h0m0s 1 true
Also noticed that I appear to have an orphaned shard which can be seen in /influxdb/data/cmk/cmk_retention but does not show up in ‘show shard groups’. If I attempt to drop this shard memory can be seen to increase as mentioned above, but shared fails to be removed;-
drop shard 666;
ERR: no data received
Can this shard simply be deleted from /influxdb/data/cmk/cmk_retention?
and why does ‘drop shard’ result in massive memory usage?