InfluxDB Checks and Notifications Best Practices

Hi. For the last couple of months, I’ve been using InfluxDB OSS to work on a proof of concept / prototype and starting to work towards a solution design. We are using OSS as on-prem is a requirement for our use-case.

Background:
Solution will have a service to collect sensor metrics which have been configured by end-users and write to InfluxDB. End users would define their thresholds via an application front-end, with rules stored in a separate SQL database. The application service would create the corresponding Check in InfluxDB using the Checks API (https://github.com/influxdata/influxdb-client-csharp/tree/master/Client#monitoring–alerting). System Admins and devs would have access to Influx UI, but otherwise a user would interact with the system through a separate application.

I have read:
https://www.influxdata.com/blog/influxdbs-checks-and-notifications-system/
and watched Checks and Notifications in Action - YouTube

which have been useful for overall understanding how checks and how alerts/checks could be used in a real world scenario, however I hope community will help with these questions:

  1. Implementation - for a dynamic number of metrics and threshold rules, would we define one alert for each rule, or is there a pattern for running checks against a set of different metrics and thresholds within a single alert task?

  2. Scale / Performance - at a rough estimate there could be in the order of 50 checks running at different frequencies and thresholds. Are there any limits on number of tasks / checks before we might run into performance issues?

  3. Retention Policy - we would like to be able to examine history of threshold checks, but these are stored in _monitoring bucket which has 7 day retention policy. Is there a way this could be extended, or should we follow the practice of writing the data we need from _monitoring _statuses measurement to a bucket with longer retention policy.

I look forward to advice from those more experienced, and I hope I will be able to contribute to the community in the future.

Hi @pgoh,

No answer unfortunately. But I’m basically trying to do the same thing.

Did you make any progress?

Hello @pgoh,

  1. You could create invokable script with parameters.
    Call that script with http package in flux instead of cURL.
    TL;DR InfluxDB Tech Tips: API Invokable Scripts in InfluxDB Cloud | InfluxData
    http.post() function | Flux 0.x Documentation

You don’t need to create a check and notification rule. You can do these simultaneously. See this:

and this:

So essentially you could create invokable scripts out of the two resources directly above to check and send a notification with your data BUT you’d parameterize the scrips and create an invokable script. Then you would create tasks where you pass in different filters and thresholds into your invokable script and invoke the script with the http.post function in a task. Does that make sense? If not, where can I help?

  1. Hmm I’m not sure. Are you using InfluxDB Cloud or OSS? If it’s OSS what HW? For cloud at least, we have customers with 10,000’s of tasks, the problem is usually they run into timeout limits for task execution, so somewhat short tasks you can run a ton of.

  2. Yes you could include logic in the scripts above to write whatever metadata you want about your check to another bucket.

Here’s an example of calling an invokable script in a task:

Where this is the invokable script