I encountered a new problem with my python influxdb injection script.
I have an object that is in charge of injecting data in line format that I generate myself from a special kind of file format.
Usually I want to write big chuncks of time series data. So I use batching, and I configure the batch size according to the number of datapoints that I have (divided by 3). And it seems to work pretty well with my large (hundered thousands data points) writes.
I also have another method for injecting a single line, which I use for a different goal.
Both of these methods work well, with good enough performance.
I’m trying to use the first method, but for a low amount of data as well.
For example: 28K lines, with 10 fields each.
When doing that, the batching system doesn’t work well as it doesn’t send all the batches.
I follow the batches and the lengths of them using the successful/error/retry callbacks, and I see that I’m missing batches (also tested using tcpdump and looking at the messages).
When I default the batch size to 5000 - it all works well. When only 2-3 batches are required - only the first one (or sometimes two) are sent.
What should I do? is it an issue that I should report here/on github?
I can set the write option to synchronous, which works well for the small writes, but I’m unsure of the large ones (especially performance wise).
Thanks for the help,