Fill usePrevious with threshold

Fluxlang
, , , ,
7

I don’t think we are on the same page about the source data yet.

The source does not contain any null values, it does however have variable intervals because of the subscription nature of the datasource. As discussed in the replication topic, my data is replicated at (for example) 1 minute intervals if communication is still good. But if communication goes bad or telegraf stops working larger data gaps occur. These are the ones I do not want to fill with the previous value.

All queries start with the usual:

from(bucket: "ProcessLog")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] =~ /PT|PDT/)
  |> filter(fn: (r) => r["_field"] == "Val_ActPV")
  |> keep(columns: ["_time", "_measurement","_field","_value"])
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: true)
  |> yield(name: "mean")

Resulting in this kind of table:

image
image911×501 33.9 KB

Adding the fill makes the output much more usable because each row has all the values. *Setting createEmpty to false creates the same kind of table only eliminating the completely empty windows.

I need the null values after the allowed fill period to make the gap in the data visible to the user. There is no reliable data after the threshold period so the fill function should not fill windows with stale values. Otherwise end users will make assumptions based on wrong data.

So my question is how do I add a fill function to my example query without filling the windows with stale data. Mind you that the example query with a regular fill works in all situations where there are no errors in the data pipeline.

Hope this makes the issue a bit more clear.

1 Like