I am running with an issue and not sure how to move forward with it.
I have integrated influxdb insertion with python library and have around 10million records that needs to be inserted in influxDb.
Here’s what I do:
- I perform some operation on the data.
- make it in the json format needed for insertion
- then write it to influxdb.
My problem is – Data operation takes 4 units of time and influxdb insertion takes 10 units of time. To handle this, I have used threads, but it is not giving significant results.
This entire process of inserting ~10 million records in database takes > 4hrs.
But, if I write the data to a file and then use import command, it just takes 15 mins to be imported.
Is there a way I can speedup my insertions without making an intermediate file and then importing?