Writing data to InfluxDB with UDP

Hi,

I’m considering a change to the way we write data to our InfluxDB database.
Currently we write batches of points via UDP if their timestamps are recent enough to be picked up by the next CQ.
If any timestamps are older than that the entire batch is written via HTTP and we do the data aggregation manually.

I know that writing via UDP does not provide any error indication if the write failed.
If all the writes fail, that’s not a problem for us because we will detect that when more data arrives.
I am curious to know what happens if the UDP listener fails (for whatever reason) part-way through processing a batch.
Can we assume that if the batch is supplied in timestamp order, it will be written to the database in that order?

Thanks,
Jeremy Begg

I believe the timestamps are processed in the order they are received. Are you trying to prioritize the points based on this so that CQ takes precedence? You can also expand the duration of “CQ” operation; it could be resource expensive but does help if your continuously running into a situation of missing CQ results.

Hi,
I’m actually thinking of dropping the CQs altogether, and instead doing all aggregation manually.
The reason is that that there can be quite a delay between taking a measurement at a given point in time and uploading it to InfluxDB, so we have to do manual aggregation for “old” data. That delay varies from a minute or two, up to several days in some cases.

When the monitoring device wants to upload raw data to InfluxDB it first queries InfluxDB to get the timestamp of the last measurement in the database, and then uploads all measurements taken since that timestamp. What I’m considering is deleting the CQs and replacing them by a daemon process that uses similar logic to create aggregated data points; the process would be triggered on receipt of new raw measurements for a device.

If the UDP listener is guaranteed to process a batch in the order in which the measurements appear in the batch, and I know my code creates the batch in timestamp order, it won’t matter if the UDP listener fails to write part of a batch; we can assume that if a data point appears in InfluxDB, the UDP listener has written all older data, or that data does not exist anywhere.

Jeremy

I’ve exactly the same setup using the HTTP connector. A bit expensive but gives the reliability. I also pre-process the data before uploading it to the Influxdb.

So I never ran into issue where the last time stamp has a mismatch or is in incorrect order. So I firmly believe the db entries are processed in FIFO order. But in case of UDP it could get tedious, since you are validating it based on the timestamps from the DB, If you have a case of failure and success with different timestamps (worst case: Success for an entry which has recent timestamp and failure for other timestamps before that). The “Last” function may not be of much use in this case rather a count of entries for a period of time could help, but it will he hard to devise.