We have the idea to build a system which can manage 100k devices
Each one of them send 1 message/sec to our system so one day our system must process 100k x 60x60x24 messages
And we’re using apache nifi to route messages from kafka to influxdb ( influxdb processor)
I see sometimes there are problems so that messages cannot be inserted into influxdb. I think it’s because we call influxdb api too many times( via influxdb processor). If we use rest api, it means we need to create 100k connections per second.
Could you please share your approach to overcome this kind of problem
Thank you so much
Can you aggregate the messages and batch them? If I am not mistake there is an FSync triggered on every write be it for 1 message or 1000 (or more)
Aggregation takes a little and increase latency, and our purpose is to be able to visualize data real time so I think this approach is the last resort
Thank you
