Best way to fetch incremental data

Store
1

Hello,

I have a time series data coming from external system on daily or weekly basic and below is the basic data we store,

[
{
"time": "2021-03-30T08:20:40.2368102Z",
"Device": "A-65",
"value": 10
},
{
"time": "2021-03-31T08:20:40.2368102Z",
"Device": "A-65",
"value": 20
}
]

Here on 2 different date we got some device and value. Device we stored as a TAG while Value as a Field.

  • Let’s assume yesterday on April 6 we got above data and here time = actual collection time (back date).

  • We process this data and we keep time = “2021-03-31T08:20:40.2368102Z” as a checkpoint that next time we need to process data which has time (greater than) > “2021-03-31T08:20:40.2368102Z”.

  • But today we got data from device = “A-1” which has collection time = “2021-03-15T08:20:40.2368102Z”.

    [
    {
    “time”: “2021-03-15T08:20:40.2368102Z”,
    “Device”: “A-1”,
    “value”: 30
    }
    ]

  • Now if we query to influx give me all data where time>“2021-03-31T08:20:40.2368102Z”, I won’t get any data since my above data row has time < “2021-03-31T08:20:40.2368102Z”.

So the question is how to get only incremental data and discard the data which already processed.

As a solution I tried to add one new FIELD “InsertTime” as epoch integer which basically now() datetime, so that each row we have one indicator when we have insert this record and this always be incremental.

[
{
"time": "2021-03-15T08:20:40.2368102Z",
"Device": "A-1",
 "InsertTime": 1617697306266,
"value": 30
}
]

Then I can query like this,

select * from measurement where InsertTime > 1617697306266

But since “InsertTime” I am storing as Field and this is not indexed, so there will be query performance issue with huge data.

So what should be the best option? Please sugget