Simulate and ingest high frequency live stream data in Influx for stress test


#1

Hi all,

I would like to simulate sensor data (i.e. float numerical data coming from machines, at very high frequency, say 10kHz), and send it to a server, where InfluxDB runs.

I have tried, by now, a simple multi-threaded python script, but i cannot reach that 10k values-er-second flow.

What else can I do to reach my goal?


#2

Hey @luca15f,

You may have better luck with our stress testing tool, available on GitHub.

Let us know how you get on :smile:


#3

Hi @rawkode ,

I gave a look (and then extensively used) the mentioned stressing tool. It actually works as expected: I was able to simulate a very high frequency (peak frequency: 120kHz).

Activable flags for the insert cmd are very useful; the only thing that drove me crazy is that I could not specify how many concurrent writers to use for a test. For example (with -f flag always activated):

influx-stress insert … -b 5000 … led to the creation of 40 concurrent writers;
influx-stress insert … -b 10000 … (default batch) led to the creation of 20 concurrent writers;
influx-stress insert … -b 20000 … led to the creation of 10 concurrent writers;

influx-stress insert … -b 200000 … led to the creation of 1 (non-)concurrent writer.

It seems that #writers gets lowered by an half everytime i double the size of the used batch. In fact, all above tests had, more or less, the same result in terms of PPS inserted.

I am really interested in varying batch size while fixing concurrent writers number.

How can I achieve this goal?

P.S. I have noticed that, 49 times out of 50, if I specify a time limit (with the -r flag), the script actually crashes when time runs out, giving no (fundamental) information on number of inserted points and PPS (the results I were referring to above were measured everytime counting all points in the database, making me waste a lot of time). Is it a known issue, or can it depend on my machine?

Thanks, Luca