InfluxDB Checks and Notifications Best Practices

Hi. For the last couple of months, I’ve been using InfluxDB OSS to work on a proof of concept / prototype and starting to work towards a solution design. We are using OSS as on-prem is a requirement for our use-case.

Solution will have a service to collect sensor metrics which have been configured by end-users and write to InfluxDB. End users would define their thresholds via an application front-end, with rules stored in a separate SQL database. The application service would create the corresponding Check in InfluxDB using the Checks API (–alerting). System Admins and devs would have access to Influx UI, but otherwise a user would interact with the system through a separate application.

I have read:
and watched Checks and Notifications in Action - YouTube

which have been useful for overall understanding how checks and how alerts/checks could be used in a real world scenario, however I hope community will help with these questions:

  1. Implementation - for a dynamic number of metrics and threshold rules, would we define one alert for each rule, or is there a pattern for running checks against a set of different metrics and thresholds within a single alert task?

  2. Scale / Performance - at a rough estimate there could be in the order of 50 checks running at different frequencies and thresholds. Are there any limits on number of tasks / checks before we might run into performance issues?

  3. Retention Policy - we would like to be able to examine history of threshold checks, but these are stored in _monitoring bucket which has 7 day retention policy. Is there a way this could be extended, or should we follow the practice of writing the data we need from _monitoring _statuses measurement to a bucket with longer retention policy.

I look forward to advice from those more experienced, and I hope I will be able to contribute to the community in the future.