Long filter clause

Fluxlang
,
2

@danielgonnet You have a few options here, but there are downsides to each.

Option 1: Use contains to filter

The first solution is to use contains() to check for the existence of a value in an array. You could pass the list of device IDs as an array and check for the existence of those IDs in a filter:

deviceIDs = ["device_1", "device_2", "device_3", ... ]

data = from(bucket: "prod_readouts/high")
    |> range(start: time(v: "2021-05-01T22:00:00.000Z"), stop: time(v: "2021-05-07T21:59:59.999Z"))
    |> filter(fn: (r) => r._measurement == "{measurement_for_customer}")
    |> filter(fn: (r) => contains(set: deviceIDs, value: r.deviceId))
    // ... more flux

The downside of this is that there are known performance issues with contains():

Option 2: Programmatically build streams and union them together

The 2nd option is only theoretical, so I’m not 100% sure it would work. It also depends on how you’re building your Flux queries. But, you could create multiple streams where each stream represents a filter of 10 (just throwing a number out there) device IDs. So you’d have to programatically build the Flux code for each stream, but if you’re able to do that, you could then union all the streams together:

baseData = () => from(bucket: "prod_readouts/high")
    |> range(start: time(v: "2021-05-01T22:00:00.000Z"), stop: time(v: "2021-05-07T21:59:59.999Z"))

stream0 = baseData() |> filter(fn: (r) => r.deviceId == "{device_1}" or r.deviceId == "{device_2}" or ....)
stream1 = baseData() |> filter(fn: (r) => r.deviceId == "{device_11}" or r.deviceId == "{device_12}" or ....) 
stream2 = baseData() |> filter(fn: (r) => r.deviceId == "{device_21}" or r.deviceId == "{device_22}" or ....) 
stream3 = baseData() |> filter(fn: (r) => r.deviceId == "{device_31}" or r.deviceId == "{device_32}" or ....) 
// ...

union(tables: [stream0, stream1, stream2, stream3, ...]

Downsides here are that there is additional programatic burden in building the Flux code AND I’m not 100% sure it would work. I think it would, but I haven’t tested it.