Nothing else using an unusual amount of memory. The largest is Icinga2. Beyond that there’s small things to support monitoring and visualization: mysql (for icinga’s config), apache, saslauthd, postfix, grafana.
The culprit is clearly influxd – it grows slowly from about 6% of memory to 60-70%. I haven’t let the OOM-killer get it yet because I get alerting on low memory, and I’ve added a cron job that just restarts influxd every 6 hours (which has solved the problem for very low values of “solved”).
Neither influx or icinga logs say much interesting. Influx is doing detailed httpd logging now – I’ll turn that off.
A few years ago I ran into a problem where a version of Icinga2 wouldn’t reconnect after losing an SSL connection to influxd, queue up data, and eventually blow up. That’s not the case here – influxd is the big process, not icinga2.
