We are storing telemetry values(sensor values) from different IOT devices into influxDB Enterpise. Total number of devices that sends telemetry are around 2000. Each sensor data is stored under a measurement. So effectively there will be 2000 measurements. Each sensor needs a different retention requirement. For example, sensor1 needs to be retained for a week, Sensor2 needs to be retained for 20 years and the retention needs might also change in future.
Hence it makes sense to create 1 retention policy per sensor, thereby I get the flexibility to change the retention time in future.
My question is, Is this a bad design? Having 2000 retention policies for 2000 measurements affects Influx performance?
@ponsakthi,
Thanks for your question. 2000 RP is too many–each RP on the database means extra shards. Every RP has it’s own set of shard files on disk and buffers in memory so every new RP requires that additional scaffolding. A better approach is to create a small set of RPs, like 2weeks , 1year , 20years
Then make sure that the larger retention policies have long shard durations. Like the 20 year one should have shard duration of 1 year or 5 years.
Rule of Thumb: less than 20 RPs actively receiving writes. Generally, creating a new RP per measurement is an anti-pattern
I would try to add up the max shard quantity across all retention policies and try to keep that number below 100. The sum of each RP’s duration divided by it’s shard duration = max shard quantity.
