Looking for some insights as I am currently designing a monitoring platform for some nano computers (ARM), with SSD disks. We currently have a few hundred devices on field with very random Internet connection (some of which will never connect). Our frontend is Graphite and don’t plan to replace it.
Went out looking for collectd, statd, but telegraf/influxdb seems to do the job quite well (small fingerprint! <3).
Now the tricky bit seems to be the very unreliable Internet connections for these devices. Most of the time connecting once a month. So I’m struggling to find a way to store all these metrics, while controlling the db size (good point to influxdb with its retention policy) and uploading everything to our Graphite instance when connection is back. Maybe MQTT/AMQP would do the trick? But from what I read, data is kept either in memory or in a file on disk, which is a nogo as device often reboot and I don’t fancy working with huge text files.
Seen any similar use case around?
Thanks for any idea/help.
I think the idea of using a lightweight durable queue that Telegraf writes to is a good idea. You would probably need some custom code for reading from the queue and writing when the network is available, since Telegraf was designed for cases where the network is mostly up and I don’t think it could currently handle the sending in this use case.
Since the device reboots it seems to me the data will have to be stored on disk in some form.
Thank you Daniel for your reply.
Just for follow-up : We ended writing with outputs.file with data-format=graphite.
When our boxes come back online they’ll just push the logs and we’ll do the necessary job (uncompress logs, push to carbon) directly on the log server. We decided to keep the poll interval quite low (60s) with 1 year retention and logrotating to make sure volume is kept lightweight (for our use case we shouldn’t have more than 50Mb of logs, which is correct considered some of our devices are on very bad Internet connections such as African remote locations).
