I am running telegraf in k8s with a very large config (~40k tags / 23MB config size). When the pod starts in k8s I see a brief memory spike up to 2.4GB, and then it quickly settles down to around 500MB.
I have other telegraf instances with smaller configs (~10k tags / 6MB config size), and that does not show any memory spikes when booting.
The problem is that due to how k8s works, I have to give each pod around 2.5GB of memory, rather than 0.75GB. Since I have dozens of pods, this makes the memory requirements unreasonable.
Does anyone have an idea of what might be causing this memory spike on startup? Is there some sort of pre-processing I can do ahead of time that will negate the need for the spike?