Kapacitor - High memory usage over time

1

Hello all,
We are using kapacitor to collect and aggregate the data from local influxdb to send it to the remote influx. We have seen high memory usage by both, but mainly kapacitor.

We have around 25 tasks.
One of the task definition is given below,

ID: traffic_bytes
Error:
Template:
Type: stream
Status: enabled
Executing: true
Created: 04 May 18 08:12 CDT
Modified: 22 Oct 18 01:19 CDT
LastEnabled: 22 Oct 18 01:18 CDT
Databases Retention Policies: [“traffic”.“autogen”]
TICKscript:
var trafficBytes = stream
|from()
.database(‘traffic’)
.measurement(‘bytes’)

trafficBytes
|influxDBOut()
.cluster(‘smpCluster’)
.database(‘traffic’)
.measurement(‘bytes’)

DOT:
digraph traffic_bytes {
graph [throughput=“0.00 points/s”];

stream0 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
stream0 -> from1 [processed=“0”];

from1 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
from1 -> influxdb_out2 [processed=“0”];

influxdb_out2 [avg_exec_time_ns=“0s” errors=“0” points_written=“0” working_cardinality=“0” write_errors=“0” ];
}

We also have one other process that writes to data influxdb from the incoming traffic, and the traffic is usually from 4Mbps to 1Gbps.

I tested with 1 Gbps, and the kapacitor uses memory quickly(in some 5 minutes), to a point of restart of the server itself.
With 4Mbps, the memory usage kept growing, but once i stopped the traffic, the kapacitor is not freeing memory.

kapacitor version: Kapacitor OSS 1.5.0 (git: HEAD 4f10efc41b4dcac070495cf95ba2c41cfcc2aa3a)
influxdb version: InfluxDB shell version: 1.5.2

Please let me know, if you need any more information.

Thank you.

2

This task is not likely the cause of the large memory usage. Do you have tasks that have large windows or perform joins or unions? Those kinds of tasks tend to use much more RAM in order to buffer the data for the operation.

3

Yeah, we have some 14 of the below tasks,

ID: top_router_bytes
Error:
Template:
Type: stream
Status: enabled
Executing: true
Created: 12 Jan 18 05:56 CST
Modified: 22 Oct 18 01:18 CDT
LastEnabled: 22 Oct 18 01:18 CDT
Databases Retention Policies: [“traffic”.“autogen”]
TICKscript:
var topRouters = stream
|from()
.database(‘traffic’)
.measurement(‘bytes’)
.groupBy(‘router’)
|window()
.period(60s)
.every(60s)
.align()
|sum(‘value’)

var topR = topRouters
|top(5, ‘sum’, ‘router’)
.as(‘bytes’)

topR
|influxDBOut()
.database(‘traffic’)
.measurement(‘top_routers’)

DOT:
digraph top_router_bytes {
graph [throughput=“0.00 points/s”];

stream0 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
stream0 -> from1 [processed=“0”];

from1 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
from1 -> window2 [processed=“0”];

window2 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
window2 -> sum3 [processed=“0”];

sum3 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
sum3 -> top4 [processed=“0”];

top4 [avg_exec_time_ns=“0s” errors=“0” working_cardinality=“0” ];
top4 -> influxdb_out5 [processed=“0”];

influxdb_out5 [avg_exec_time_ns=“0s” errors=“0” points_written=“0” working_cardinality=“0” write_errors=“0” ];
}

4

please see if this helps - Kapacitor High Memory Usage while using UDFs | by Tamil Selvan V | Medium