We are using Prometheus input plugin for gathering metrics. I want to understand what happens if the collected metrics are more than ‘metrics_batch_size’ ? Will the output plugin collect only ‘metrics_batch_size’ number of metrics ? or will it fetch all of them and store additional metrics in output plugin buffer ? or the additional metrics stored in buffer of input plugin ? Do input plugins have buffer for storing metrics uncollected by output plugins ? If additional metrics are stored in input plugin, will they get fetched in next interval of collection from output plugin ? How this works ? I have been using http output plugin customized to add auth headers ?
Each output plugin has a limit equal to the setting metric_buffer_limit
Each output plugin will send the metrics to the output once either the interval or metric_batch_size has been met
If you have a buffer limit of 10,000 and a batch size of a 1000. Once you get to 1000 metrics, it will try to push them forward. If for some reason it is slow or down, it will queue the metrics internally.
Once you get to 10k metrics queued, it will begin dropping metrics
