Hi folks ! I’m new here, I’ll try to make this as readable as possible.
We are cascading retention policies and continuous queries to fill measurements with per minute/hour/day/week/year granularity. Due to our data set, high cardinality is expected. Using InfluxDB 1.7.4 with TSM index.
We encountered max-series-per-database (1M limit) errors, digging out a little bit we found out some interesting things :
- ‘show series cardinality’ yelds 300k series, same with summing “show series exact cardinality from RP.measurement” for each measurement. (looks like exact cardinality only looks in default RP without from clause)
- ‘show tag values from RP.measurement with key = “my_only_key”’ sum yelds 1M if logged as admin, 300k otherwise (?!)
- restarting influxdb brings “show tag values” seen by admin to 300k
Splitting out the numbers provided by ‘show tag values’ we found out there is much more difference between short RPs than long RPs. As this system has been running for a few month only our guesses are the following :
- when values are taken out of the DB due to RP, index is not cleaned and series remain
- series without values are not shown by ‘show series’ / ‘show series cardinality’
- max-series-per-database is based on both “shown” and “hidden” series
- there is a user privilege influencing ‘show tag values’ output
What do you think about those guesses ?
How do we fix our “hidden series” problem ?
- I have read there is no way to purge index, everybody seems to be just restarting InfluxDB which is not an option here (anywhere in prod ?
) - problem should already be fixed per [0.12] Retention policy cleanup does not remove series · Issue #6457 · influxdata/influxdb · GitHub + Remove series from index when shard is closed by jwilder · Pull Request #6485 · influxdata/influxdb · GitHub
- I guess we should move to TSI index due to our high cardinality, will this solve our problem ?
Thank you very much!
Edit: some pieces of the message were dropped
