I’m currently writing a golang system that essentially simulates a system of entities over an entire day (which can be viewed on a website.) While the simulation itself is fast to calculate this data, the weakest link of the system is InfluxDB’s golang client, which is failing to cope with the large amount of data.
For an example, say there are 20 entities. They operate over an entire day in seconds, so 86400 steps of simulation. For each step of simulation, they write 8 points of data. This means that, roughly, there are 13,824,000 points to be written. The batch size is 5000; the Retry Buffer limit is 25,000; the Flush Interval is 5000ms. While I am writing this data, I am also simultaneously querying the database every second for information about these entities (in this case, 3 points of data per entity.)
The entire simulation takes 10 seconds to complete, at which point the system flushes all remaining data. It writes data to the database successfully for a while, but begins to slow over time, with high RAM and CPU usage (to be expected). After about 2 minutes, I get this error:
Post "http://localhost:9999/api/v2/query?org=MyOrg": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Following which the golang system panics, and the signal is killed, with no more data being sent.
Is there a way to set up the system to handle such a large quantity of data to be written? Do I need to consider higher performance computing for this operation?
EDIT: It turns out that the sim is failing before it gets to the flush stage, not after. This means it’s something to do with writing rather than flushing.