Basic question about storage efficiency

Hi - I currently am running a home-made solution to monitor things in my house such as well pressure, status of pumps, fans, furnace, temperatures etc. I store the data in an SQL database and was thinking of switching to influxdb, however I have a question.

Currently I store the info in the db using a sorta “run-length” encoding scheme. That is, most sensors don’t change much with time. For example a pump may run for a few minutes, and then not run for days or weeks. So I only store the values in the db when they are changing. Thus my database if relatively small PLUS I have every piece of information going back for years. I have read where influxdb has some methods to save space on historiical data by storing averages etc, however that scheme would lose data and not be useful to me. Does influxdb also have a scheme to store data as I am doing now by preserving every piece of data and saving space?

As a side note, with my current scheme I was able to diagnose a few problems (I’ll describe two). First, I was looking at graphs and suspected my septic pump was running more than usual. I looked at the graph going back months and years to confirm it. I then open the lid on the septic tank and could see a tiny spray/leak from the pipe going to the field. Fluid was leaking back from the field. After some more analysis I confirmed it started after the fellow came to pump out the tank. I suspect he was slamming the hose around and damaged the joint on my pipe. I had another problem where I suspected that my well pump was short cycling. Being that I store every second of data, I looked at the slope of the pressure graph going back years for each pump cycle. I could see the system would pressure-up at an increasing rate as time went by. I then checked the pressure of the air charge in the pressure tank using a tire gauge and found it was low and needed air added. So it is important for me to store all data efficiently and not lose any.

Hello @btdb,
Welcome!
Yes you can do that.
You can write all your raw data to one bucket with a small retention policy.
Then you can run a task to determine if values have changed and only write those changes to a new bucket with a longer retention policy.
https://docs.influxdata.com/influxdb/v2.0/process-data/get-started/
You can use the monitor package to collect the data that is changing.
https://docs.influxdata.com/influxdb/v2.0/reference/flux/stdlib/monitor/statechangesonly/
Use the to() function to write those changes to a new bucket
https://docs.influxdata.com/influxdb/v2.0/reference/flux/stdlib/built-in/outputs/to/

Finally this blog could be useful to you as well:

That’s a really cool use case! Please let me know if you want any more information on any topic. :slight_smile: