Cardinality Limitations with AWS Managed InfluxDB (v2.7) for IoT Data Ingestion

InfluxDB 2
1

Hi InfluxDB Community,

We are exploring AWS Managed InfluxDB, which currently supports InfluxDB v2.7, and I have a question regarding cardinality limitations. According to the InfluxDB documentation, if series cardinality exceeds 10 million, InfluxDB recommends using the Enterprise version. However, I haven’t found any detailed explanation of the implications if we exceed this 10-million cardinality threshold.

Here’s an outline of our use case:

  • Data Type: IoT data
  • Measurement Names: device_type (e.g., sensors, camera, gateway, adv_security)
  • Tags:
    • account_id (10 million unique account IDs)
    • partner (up to 4 unique values)
    • device_id (each account can have approximately 5 unique devices)
  • Fields: Varies based on device_type

From my understanding, InfluxDB calculates cardinality based on the measurement name and unique tag key values. In our case, the unique combination of account_id and device_id should represent our maximum cardinality. However, in my testing, the cardinality seems to reflect only the unique account_id, which I’m uncertain about.

My questions are:

  1. What will be the true cardinality calculation for our data structure?
  2. What are the potential implications or performance impacts if our cardinality exceeds 10 million on the AWS Managed InfluxDB (v2.7) setup?

Our estimated ingestion rate is around 200-300 events per second, with read requests in the range of 10-20 per second. Any insights on how this cardinality limit might affect performance or stability at these rates would be greatly appreciated!

Thanks in advance for your help!

2

Hello @Durga_Prasad_Anugam and Welcome!
Cardinality in InfluxDB is calculated based on series or dimentionality rather than raw data points. A series is a unique combination of:

  • Measurement name
  • Tag keys
  • Tag values
  • Field keys

For your setup a cardinality estimation looks like:

Cardinality = Number of Measurements * Unique account_ids * Average devices per account * Unique partner values * Field Keys

For your use case you’re significantly exceeding 10 million, meaning you are likely to face performance and storage challenges.

This could result in read and write latency issues.

Do you mind me asking what is your use case? What security solution are you providing?

3

@Durga_Prasad_Anugam
For this scale/throughput use case I recommend talking to someone in sales who knows more about the impact on performance you’re looking at and whether v3 might solve some of those issues. Would you like me to pass your information along?

You can also elect to do so here: