Impossible to aggregate net measurement in telegraf database

Telegraf
, , ,
1

Hey everyone!!

We use telegraf for storing metrics in our InfluxDB. As we’re running out of disk space we’re using RPs and aggregating data and store them to different RPs. We ran into problem with aggregation of measurement called net which has around 94 fields which I can see in Grafana Dashboard. The problem is that it is even impossible to select one line of measurement with limited select. About counting the records I would even speak.

When I try to run query for aggregation with mean function that would mean data every 1m the machine is swiftly running out of RAM. We even tried to add RAM memory for 30GB which is insane and it still is running to 29 and then use 2GB swap so machine drops.

If anyone ran into the similar problem please offer a hand.

Thanks in advance!!

2

Hello @vladpov,
Have you considered using flux? You can perform aggregations easily across all fields in a measurement like so:

 from(bucket: "mybucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "mymeasurement")
  |> mean(column: "_value")
3

Hello @Anaisdg,

first of all thank you for your reply! The solution you offer sounds great :blush: If I understand it correctly example of code you sent would take the measurement and run mean function on specified column and time range.

If you could clarify couple questions that occur in my mind :smiley: Is necessary to define time range? Can I run similar code on every record in the column? How and where is output stored? I mean if I run it on every column, is it possible to store mean output to one measurement column after column?

Thank you again for your help! :pray: Cheers :v:

4

Hello @vladpov,
You’re welcome! :slight_smile:
Yes, you need to define a time range, however you can query all the data in your bucket if your bucket doesn’t have too much data.
The output is stored in memory, but you can write the data to a new bucket with the to() and you can do this on a schedule with a task.

Does this help answer your questions?

Thank you! :))

1 Like
5

Great! Thank you @Anaisdg, I’ll give it a try!

Speaking of scheduling, I ran only to these options of scheduling a task. Is anyhow possible to schedule it with kind a if statement? Like if task is done successfully continue to the next column…

Best regards.
VP

6

This topic was automatically closed 60 minutes after the last reply. New replies are no longer allowed.

7

You can create multiple tasks and write the outputs to a new measurement or bucket. Then you can write a new task to operate on that output. Does that help?

1 Like