Hello, I’m new to this community and was wondering if anyone could advise me on some questions. For my thesis I’m currently planning a house monitoring system (raspberry pi) that contains multiple sensors and needs somwhere to store mass data over time. I’ve picked InfluxDB as it seems to be the top dog for time-series data.
But i am new to this kind of stuff and am trying to wrap my head around what is possible.
My plan is:
Deploy an InfluxDB database on AWS (potentially on a docker container on a EC2 instance, as i will be deploying a frontend side to display the data.)
Have a Python program on the pi read the sensors and converts to json format.
Use Telegraf on the pi to insert data in the database on AWS every period of a time (maybe 10 minutes).
Then the use the frontend side i mentioned early to query and display the data to a user checking for events and readings from their house. RASPI → TELEGRAF → INFLUXDB(deployed alongside frondend program on docker on EC2)
My questions are if this sounds possible and am I overcomplicating things with this method?
(I haven’t quite figured out what i want to do for the frontend, i was thinking maybe a simple api with React or integrating with grafana or some graphing software. But these are all things i have not fully researched).
I appreciate any help and guidance from you influxDB experts, again i am new to InfluxDB and am sorry if this seems like trivial questions. Thanks for your time.
We have InfluxDB on an AWS EC2 instance. It was a pain to set up (so many security hoops to jump through, which may not apply to your case).
Anyway, the signal chain we have is:
Node-RED scrapes the sensor data, massages the data to be how we want to format it (e.g. convert to JSON format), and passes it on to Influx at the rate we specify…the Node-RED community is extremely active and extremely helpful.
→
InfluxDB v2.4 (we originally started with v1.8 and used InfluxQL, but made the jump to v2.3 /later 2.4/ and converted everything to Flux. I also realized the importance of how to properly set up the fieldnames, tagnames, measurement names, buckets, etc. rather than just jumping in. I can provide you with some helpful links if you have not already gone through this exercise.
—>
Grafana (OSS free v9.2) to view the data and alert if something goes wrong. We also use InfluxDB alerts.
Even though it is already quite old, I found this webinar from Influxdb about Schema Design for IoT to be extremely helpful. Watch it all the way through, then think about your own data (and even what data you eventually intend to collect). Write down all of your field names & tag names and then even post here if you wish and I (or others) can make sure you are headed in the right direction. If you start incorrectly and realize it ~3 or ~6 months into your data collection, you might be forced to start over.
Also, beginner tip (from another beginner)…when I am setting up this stuff at first, I often goof up something here or there. I usually put everything in a bucket that I created called “JunkBucket”, populate it with my data, build the query in InfluxDB Explorer, paste into Grafana, etc. Once I am happy with the outcome, I simply change the bucket name to the actual bucket name I want to use. Then delete JunkBucket.
Node-RED comes pre-installed on most Raspberry Pi’s (and is dead simple to install). I’d definitely look into it if you have not already.
Yes, the EC2 instance (once set up and running) has been fine, but I prefer on-prem devices. Remember you can also use InfluxDB Cloud and skip AWS altogether.
