Accumulating lag in pipeline

I’ve got data that looks like:

   "series" : [

      {
         "name" : "measurement_name",
         "columns" : [
            "time",
            "count"
         ]
         "values" : [
            [
               "2017-10-17T17:04:20Z",
               2
            ]
         ],
         "tags" : {
            "pool" : "a",
            "reporting_lb" : "1"
         },
      },

      {
         "name" : "measurement_name",
         "columns" : [
            "time",
            "count"
         ],
         "values" : [
            [
               "2017-10-17T17:04:20Z",
               2
            ]
         ]
         "tags" : {
            "pool" : "a",
            "reporting_lb" : "2"
         },
      }

   ]

I process this to get the average of the value across all "pool"s, using TICKscript like so:

var flaps = stream
    |from()
        .database(db)
        .retentionPolicy(rp)
        .measurement('loadbalancing_member_events')
        // use only down events (going down then up is a full flap; don't count halves)
        .where(lambda: "event" == 'readiness change' AND "transition" == 'down')
    |delete()
        .field('text')
        .tag('transition')
    |window()
        .period(5m)
        .every(10s)
    |groupBy(['pool', 'reporting_lb'])
    |count('event')
    |groupBy(['pool'])
    |mean('count')

The calculated average will sometimes lag minutes behind the input data. There seems to be an accumulating delay at the mean() node.

These stats might help. The mean() node is I believe “mean7” here:

stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
from9 [avg_exec_time_ns="521ns" errors="0" working_cardinality="0" ];
window10 [avg_exec_time_ns="1.25µs" errors="0" working_cardinality="1" ];
groupby11 [avg_exec_time_ns="1.635879ms" errors="0" working_cardinality="233" ];
mean12 [avg_exec_time_ns="15.782µs" errors="0" working_cardinality="0" ];
influxdb_out13 [avg_exec_time_ns="9.7µs" errors="0" points_written="1148901" working_cardinality="0" write_errors="0" ];
from1 [avg_exec_time_ns="2.843µs" errors="0" working_cardinality="0" ];
delete2 [avg_exec_time_ns="4.833µs" errors="0" fields_deleted="155872" tags_deleted="155872" working_cardinality="0" ];
window3 [avg_exec_time_ns="4.046µs" errors="0" working_cardinality="1" ];
groupby4 [avg_exec_time_ns="3.79276ms" errors="0" working_cardinality="10" ];
count5 [avg_exec_time_ns="43.52µs" errors="0" working_cardinality="0" ];
groupby6 [avg_exec_time_ns="4.889µs" errors="0" working_cardinality="0" ];
mean7 [avg_exec_time_ns="14m18.119870102s" errors="0" working_cardinality="208" ];
join15 [avg_exec_time_ns="24.9µs" errors="0" working_cardinality="235" ];
eval17 [avg_exec_time_ns="69.2µs" errors="0" working_cardinality="206" ];
alert19 [alerts_triggered="92" avg_exec_time_ns="24.613µs" crits_triggered="8" errors="0" infos_triggered="0" oks_triggered="44" warns_triggered="40" working_cardinality="206" ];
influxdb_out20 [avg_exec_time_ns="6.025µs" errors="1" points_written="90" working_cardinality="0" write_errors="1" ];
influxdb_out18 [avg_exec_time_ns="3.005µs" errors="3" points_written="16737" working_cardinality="0" write_errors="3" ];
influxdb_out16 [avg_exec_time_ns="2.475µs" errors="0" points_written="16743" working_cardinality="0" write_errors="0" ];
influxdb_out8 [avg_exec_time_ns="6.733µs" errors="0" points_written="160454" working_cardinality="0" write_errors="0" ];

Any ideas why this might be happening?

Thanks.

Has anyone experienced strange bottlenecks in their pipelines before?