Telegraf: route aggregations to different buckets

Telegraf
1

Hi,
I have few sensors sending data over MQTT and for which I would like to create 3 buckets:

  • one with raw data, to be kept for max 30 days
  • one with aggregated data, per hour, to be kept for 90 days
  • one with aggregated data, per day, to be kept for long periods
    Below is the way I implemented the first two but I have the feeling could be done easier. In addition, I don’t know how to do the 3rd one.
[[aggregators.basicstats]]
  alias = "basic_stat"
  ## The period on which to flush & clear the aggregator.
  period = "1h"

  ## If true, the original metric will be dropped by the
  ## aggregator and will not get sent to the output plugins.
  drop_original = false

  ## Configures which basic stats to push as fields
  # stats = ["count","diff","rate","min","max","mean","non_negative_diff","non_negative_rate","stdev","s2","sum","interval"]  
  stats = ["count","min","max","mean"]
  
  fieldpass = ["humidity" , "temperature_C" , "rssi"]

[[outputs.influxdb_v2]]
    alias = "raw_data"

  urls = ["http://${DOCKER_INFLUXDB_INIT_HOST}:8086"]

  ## Token for authentication.
  token = "$DOCKER_INFLUXDB_INIT_ADMIN_TOKEN"

  ## Organization is the name of the organization you wish to write to; must exist.
  organization = "$DOCKER_INFLUXDB_INIT_ORG"

  ## Destination bucket to write into.
  bucket = "$DOCKER_INFLUXDB_INIT_BUCKET_RAW"

  insecure_skip_verify = false
  fielddrop = ["*_count", "*_min", "*_max", "*_mean"]

# # Configuration for sending metrics to InfluxDB
[[outputs.influxdb_v2]]
    alias = "aggregated_data"

  urls = ["http://${DOCKER_INFLUXDB_INIT_HOST}:8086"]

  ## Token for authentication.
  token = "$DOCKER_INFLUXDB_INIT_ADMIN_TOKEN"

  ## Organization is the name of the organization you wish to write to; must exist.
  organization = "$DOCKER_INFLUXDB_INIT_ORG"

  ## Destination bucket to write into.
  bucket = "$DOCKER_INFLUXDB_INIT_BUCKET_AGG"

  insecure_skip_verify = false
  fieldpass = ["*_count", "*_min", "*_max", "*_mean"]

I tried to tag, as below, all aggregated data but didn’t work.

    [aggregators.basicstats.tags]
	  influxdb_database = "aggregated"

In my view, this would solve the implementation of 3rd bucket (bucket tags for: inputs.mqtt_consumer, aggregation per hour and aggregation per day) but not working put me in the position to remain with the first 2 buckets.

Thank you

2

Hi,

Are you sending an enormous amount of data that you only want to keep 30 days of raw data? Is it possible to use the aggregation in your queries in InfluxDB, rather than trying to do this at collection time?

3

Hi,
In the end I would like to have the 3 buckets to keep a reasonable low size.
The question could be who is doing the aggregation: Telegraf or InfluxDB?
What would be the pros and cons of one or another?
Thanks