What is the recommended practice for "protecting" some data from a retention policy?

Store
1

I have high rate data, and over time I want to eliminate old data and instead store lower rate aggregate data. Exact intended use case for CQs and retention policies. Great.

However, some data I want to keep forever no matter their age. Specifically, I want to keep the previous X minutes of full rate data prior to a fault. Fault event records are also stored in InfluxDB.

What’s the best way to achieve this in InfluxDB 1.6? Is it possible to automate with CQs or should I plan to deliberately execute a custom query to achieve this (say, after each fault is detected)? If so, what would such a query look like?

Thanks.

2

Hello,

Would using Kapacitor and event handlers to identify a fault and then backfill fuller resolution data with a CQ help you? (You could also use Kapacitor to create a TICK script with threshold parameters for fault metrics)

3

If I understand correctly, I don’t think Kapacitor will help me here. This is on an embedded system. The fault conditions are identified by dedicated hardware circuits, so I don’t need to do any kind of streaming processing to find the fault; I just look for the right hardware signal and log it into InfluxDB. Kapacitor could do this but it’s a much more heavyweight solution compared to adding a line or two of Python.

And I believe the link about CQs that you list shows that backfilling cannot be achieved with a CQ but must be done using a regular INTO query. The most basic solution I can think of is to manually execute an INTO query to copy data to special, separate database used for saving fault data every time a fault condition occurs. I didn’t know if this was best practice or not. Again, Kapacitor could do this but executing it in Python isn’t difficult.

4

Hello,

Yes, you’re correct. Using the INTO query is the best way to alternate between downsampling data and retaining full resolution data. Using Kapacitor becomes really handy when you want to detect fault events with specific metric conditions and perform various aggregate functions. There is no best practice your case. Go with your personal preference.