I’d appreciate some help on the stuff below.
A little context:
- Our application saves timeseries data representing sensor reports.
- We have event processing software, which reads the data saved in the last N seconds and processes it.
- The data does not always come on time. Sometimes the sensors’ connection is being interrupted, so the data is buffered and sent when connection is available. This results in saving data with older timestamp relative to the current time.
- Our problem is the event processing program is retrieving the data for the last 30 seconds. If the data is saved now with older timestamp (therefore does not exist in the last 30 seconds), it is skipped.
What we have tried to do is save the processing time in the database in unix format and use it, instead of the time field.
Two attempts are done here:
- Save the processed time as a tag - does not work, because tags are string values and range queries cannot be performed (>, <)
- Save the processed time as a field - works, but every query takes up to 10 seconds
Opting out the primary time is not an option, we need to know when the data is originally taken.
The way the event processing program works also can not be changed.
What would you advice me to do?
All ideas are appreciated.