Querying by secondary date in influxdb

1

Hello everyone,

I’d appreciate some help on the stuff below.

A little context:

  • Our application saves timeseries data representing sensor reports.
  • We have event processing software, which reads the data saved in the last N seconds and processes it.
  • The data does not always come on time. Sometimes the sensors’ connection is being interrupted, so the data is buffered and sent when connection is available. This results in saving data with older timestamp relative to the current time.
  • Our problem is the event processing program is retrieving the data for the last 30 seconds. If the data is saved now with older timestamp (therefore does not exist in the last 30 seconds), it is skipped.

What we have tried to do is save the processing time in the database in unix format and use it, instead of the time field.
Two attempts are done here:

  • Save the processed time as a tag - does not work, because tags are string values and range queries cannot be performed (>, <)
  • Save the processed time as a field - works, but every query takes up to 10 seconds

Opting out the primary time is not an option, we need to know when the data is originally taken.
The way the event processing program works also can not be changed.

What would you advice me to do?
All ideas are appreciated.

Thank you,
Angelin

2

Hi Angelin,
is adding “and time > now()-1h”
in your query an option or can sensors’ be interrupted for longer than 1h ?

Best regards

3

Hi MarcV,

This query will fetch a big percentage of the data that has already been processed.
We want to fetch only data from the last processing time till now, so there is no overlapping.

Greetings

4

Do you mean you already have a “ and time > ‘something’ “ in your query ?

5

Yes.
time > "last processing time" and time < now()