Delayed replication to an InfluxDB available intermittently

1

I’m looking for an architecture feedback.

For a home project, I’m considering to store measurements into InfluxDB on a RaspberryPi. I’m considering a retention policy of a few weeks (2-3 months worst case). However I’d like to keep the full detailed history (call me data hoarder :)) on a somewhat more beefy server. Which, let’s say for sake of electricity consumption optimisation, is not up all the time. And worse than that, it is booted quite irregularly whenever I want it up.

What I’d like to reach is that have that server to replicate (synchronize) the data of the influxdb on RaspberryPi since the last time it connected (or at least what is still available due to the retention policy setup).

How would you do it?

My first idea is to keep the original data on the RaspberryPi, synchronize the original data whenever the server is brought up and handle an automatic injection in the full history influxdb on the server.

But that is not fun. Is there a clever use of Telegraf, Kapacitor, Influx subscriptions (or something else I missed) that would allow some kind of replication “whenever the slave is available”.

I’d say that the only constraint would be not to loose too many measurements between 2 synchronizations. For example buffering data in memory in the RaspberryPi might exceed the available RAM or I would loose everything since last sync if the power is cut.

Thanks for any pointer you’d have. There is no wrong answer here so just shoot!

1 Like
2

I too have a similar use case in mind and would also be very interested to see if anyone has a solution for this. I’m just starting out with the whole Influx stack so unfortunately no usable input from me at this time :sweat_smile:

I was thinking of doing some OBD data logging with RaspberryPi which resides in my car and also intermittently connects to the internet over wifi.

3

An OBD use case. That is interesting. I have a WiFi dongle in my car. I could give it a shoot too :thinking:

4

So, I actually realized that I would not necessarily need an InfluxDb instance on the RasPi. Instead I was thinking of pushing the metrics to a persistent RabbitMQ queue and then use Telegraf (AMQP input, InfluxDb output) to push it forward to my remote InfluxDb when ever there is a connection. I think you might be able to pull it off with multiple InfluxDbs if you need to have one on your Pi as well. This is just an idea so far, I will let you know when I get around to actually testing it :smiley: