I’m trying to push some data from Influx Cloud to Bigquery but I’m getting a “jobRateLimit Exceeded” error.
In my influx query I’m using the sql.to approach described here: Write to Google BigQuery | Flux Documentation
When I look at the BigQuery Logs I can see that influx is doing a CREATE TABLE IF NOT EXISTS followed by a INSERT INTO. So for each of the rows I need to insert (10s of thousands) influx does this check.
The rate limit in bigquery that seems to apply is “up to five table metadata update operations per 10 seconds per table.” I can see in the logs that it is the CREATE TABLE IF NOT EXISTS operation triggering the rateLimit issue.
Is there anyway to stop Influx running that operation? I’d love to be able to do it once and then just do inserts (even a batch insert).
I should add that I’m getting very few rows inserted. First time I got 12. Second time another 8.