Not sure if I should store non-changing data

1

Folks, first post so please go easy on me if I am not on the right track.

My question is about the storage of the data in the influxdb where the all fields (20+, numerical) of the data isn’t really changing much all the time. e.g. inverter data which is charging a large battery and during charge/discharge cycle battery_soc value won’t change much but some fields like home_load will change quite frequently, let’s say once every few second.

If I feed that data over mqtt to the influxdb I am filtering out unchanged data to reduce the noise and traffic. However this produces additional complexity where every second only a changing fields get updated in the db and we don’t know when next change in value will arrive. As result query becomes inaccurate as I don’t know how long the aggregate window has to be resulting in mean values being inaccurate. May be there is a way out of it and I need to learn how to handle this situation on the query but I am sure it will have additional overhead on the server processing. Is it worth the effort and cpu cycles to save some bandwidth and storage space?

So at this point I have two choices:

  1. Send only the changing fields. Pro: Less data to sent across saving bandwidth and influx db storage, Con: Difficulty in queries.
  2. Send all fields: Pro: Ease on query, Con: Repeated data unnecessarily flowing over the network and consing storage space where 20+ fields are being send across every second.

Has anyone been in this dilemma, any suggestions from the experience?

2

My advice is always to “record the data you are going to want to process
later”.

In most cases, this means “record a data point every timestamp, so that later
you can display what the value at that timestamp was”.

There are dozens (at least) of questions on this list and the Grafana list of
people doing things like measuring when water gets added to a tank, or removed
from it, and then later they want to display the level (quantity) of water in
the tank over some period of time.

This is far from simple to do, because they did not record the quantity of
water in the tank - they recorded when this changed, but that’s not what they
want to display afterwards.

So, in summary, think about what it is you want to see on your instrumentation
system at the end of the whole measurement / analysis process, and make sure
you are recording the data that’s needed to show that.

Antony.

1 Like