I have installed influxDB together with Grafana, Node-Red on Raspberry Pi 4 and have been playing since a month, impressed what can I achieve with them. However, I need a faster solution.
I have a project in my mind, to create a car data logger (inverse kinematics) to log from CAN bus and multiple acceleration and GPS sensors. Sample rate and the amount of data to record is quite high.
What would you recommend, how to handle bandwidth heavy applications? Specifically, I would like to know:
How to aggregate data faster without Node-Red, e.g. with Python or C? Alternative agents?
How to record data to something like HDF5 then dumb into influxDB?
Hello @nebulae,
Welcome and thanks for your question. Why do you want to record to HDF5 first? I don’t think HDF5 files compress as well as InfluxDB compresses. What version are you using (if you’re using an older version, you might want to enable TSI). What sort of aggregation are you performing? You have a couple of options when it comes to aggregations: telegraf aggregator plugins, kapacitor, or flux. I also recommend checking out the new python client. You can use multiprocessing to perform writes. That being said, I’m not very familiar with IoT or networking. Perhaps @davidgs has a better answer for you.
Hello Anais. Thank you very much for your response.
One reason is, IMO it is more efficient than quering 300 messages per second, because, I think it will also reduce the lifetime of the SD card. Another reson is, it would be handy to have HDF5 files for further Analysis in LabVIEW.
Furthermore, I have been thinking to make an offline logger which can load the data when I am back home to the home server. But before I have long way to go.
I am using the latest version. But I do not think that TSI will solve my problem. I need to be able to store bandwidth heavy data, without overloading the system.
Right now I am feeding the serial data to influxDB using Node-Red. But, yes, Telegraf and Kapacitor was in my prespective. Also going to check Flux. Especialy, for my purpose I find Python very useful.
@nebulae,
Thanks for your response. Just to clarify, do you still need help? I will bug others to help you, but please know it might be a while until someone gets back to you due to holidays…
Using a direct library and writing a small script – Python or Go – is probably the most efficient way to go here. I typically use Go and collect/write data directly to InfluxDB via the library. I usually do batch writes for efficiency but you don’t have to.
Thanks David!
Do you mean that you write data to a file and then dump it into influxDB or send batches of points? You probably flush data when certain number of points get buffered. It should be possible to configure influxDB like that setting batch size.
No, I do not write data to a file. The client libraries allow you to write the data directly to an InfluxDB instance. I just read and batch up points in the script, then send them as needed.
