Hi, I am using InfluxDB as the remote storage backend for Prometheus to archive metrics of a relatively small kubernetes cluster: 20 nodes, about 800 pods.
InfluxDB (1.7.9) is running in single instance with these limits: 16GB memory, 4 CPU.
I used the default settings described here: Configuring InfluxDB OSS | InfluxDB OSS 1.7 Documentation
It works, but it consume all of the 16GB and has a steady 1.5 CPU usage, just to handle prometheus ingestion (no. grafana queries running).
I was wondering if there are specific settings for using influxdb as prometheus backend, or any hint on tuning the DB parameters, or prometheus itself.
To give some context of the load, reverting to Prometheus only (native storage), with the same load, results in a 0.2 CPU usage and 2GB of RAM.
Also I noted that even if InfluxDB is not receiving any metrics, it keeps “Compacting” it’s files for about 15 minutes before becoming idle.
Best regards,
Roberto
