Influx Cloud TSM adjustable quotas and bulk imports

InfluxDB 2
,
1

Hi there!

We are currently using an InfluxDB 2 Cloud TSM instance on Azure, which we got via the Azure Marketplace offering: Microsoft Azure Marketplace

We are faced with the problem, that our ingest regularly surpasses the ingestion limit of 300MB per 5 mintutes, which is documented on the “InfluxDB Cloud limits and adjustable quotas” page.

The reason for this is that we ingest data which is exported every hour from other systems as CSV data with up to millisecond timestamp resolution. Our clients us the golang v2 client library: GitHub - influxdata/influxdb-client-go: InfluxDB 2 Go Client

Spread out across the whole hour, the data would comfortably fit in the Data-In quota. However since the data arrives all at once every hour, we get rate limited regularly, which we can also see in the usage dashboard.

So I have the following questions:

  1. What would be the recommended way to handle this scenario?
  2. Can the quota be increased or adjusted to work on a 1-hour window instead of a 5-minute window? Is it possible to upgrade the usage based plan and if so, how?
  3. Is there a way to (automatically) ingest file-based data rather than ingesting through the ingest API?
  4. Or is the only option to implement custom rate limiting in our clients and coordinate it across multiple clients?

Any help is appreciated, thanks!

2

@csh I’ll try to answer your questions

I would recommend batching your writes in a way so that each batch doesn’t exceed the API rate limit and then running a batch each 5 minutes until the queue is fully processed (essentially rate-limiting client-side).

The enforcement interval cannot be changed, but you may be able to upgrade the enforced limit. I’m not 100% though. You could reach out to InfluxData support.

Kind of. For this case, you could use Telegraf as the write agent. It could read data from files or even a message broker like Kafka and write the data to InfluxDB. Telegraf actually has some rate-limiting capabilities built in that you could use to stay within your InfluxDB Cloud rate limits. Since you’re already writing data to CSV, you could use Telegraf to parse the CSV files and write them to InfluxDB.

This is an option, but not the only option. My recommendation would actually be to have your clients write to a “middle-man” that handles the batching/rate-limiting as it writes to InfluxDB. I think the simplest implementation would use Telegraf, but it all depends on your current data pipeline and how much you are willing to change it.

3

I see, thanks a lot!