How can i use low interval telegarf input for both low frequency influxdb output and high frequency influxdb output

Telegraf
1

Hello,

  1. I have modbus telegraf input plugin to collect data from various devices (intervals = 5 minutes).
    Need to write those data to 2 different influxdb bucket. 1st bucket with update intervals of 5 minutes(This bucket is with data retention policy of only 3 days storage). 2nd bucket with update intervals of 1hour( data retention policy of forever).

  2. I don’t want to create another separate sets of inputs to get data (because there are 77 devices currently and it will increase on the future)

Need suggestion on this situation. (is there a way to do this other than influxdb task script)

My current telegraf conf file looks like: (with 1st bucket with data retention policy of 3 days)
Need to add 2nd telegraf config for another influxdb

# Configuration for sending metrics to InfluxDB 2.0
[[outputs.influxdb_v2]]
  ## The URLs of the InfluxDB cluster nodes.
  ##
  ## Multiple URLs can be specified for a single cluster, only ONE of the
  ## urls will be written to each interval.
  ##   ex: urls = ["https://us-west-2-1.aws.cloud2.influxdata.com"]
  urls = ["http://127.0.0.1:8086"]

  ## Token for authentication.
  token = "influxdbRead_Token"

  ## Organization is the name of the organization you wish to write to.
  organization = "OrgName"

  ## Destination bucket to write into.
  bucket = "demo"


# Retrieve data from MODBUS slave devices
[[inputs.modbus]]
  ## Connection Configuration
  ##
  ## The plugin supports connections to PLCs via MODBUS/TCP, RTU over TCP, ASCII over TCP or
  ## via serial line communication in binary (RTU) or readable (ASCII) encoding
  ##
  ## Device name
  name = "Device name"
  
  interval = "300s"

  ## Timeout for each request
  timeout = "1s"

  configuration_type = "request"
  
  tagexclude = ["host", "slave_id", "type"]


  # TCP - connect via Modbus/TCP
  controller = "tcp://192.168.0.103:502"

  [[inputs.modbus.request]]
    ## Holding example
    ## All of those examples will result in FLOAT64 field outputs
    slave_id = 33
    byte_order = "CDAB"
    register = "holding"
	
    fields = [
      { address=3960, name="Active Energy",       type="FLOAT32", scale=0.001, measurement="measurementName" },
    ]


  [[inputs.modbus.request]]
    ## Holding example
    ## All of those examples will result in FLOAT64 field outputs
    slave_id = 34
    byte_order = "CDAB"
    register = "holding"
	
    fields = [
      { address=223, name="Active Energy",       type="FLOAT32", scale=0.001, measurement="measurementName" },
    ]

...
...
...
	  
  [inputs.modbus.workarounds]
    ## Pause after connect delays the first request by the specified time.
    ## This might be necessary for (slow) devices.
    # pause_after_connect = "0ms"

    ## Pause between read requests sent to the device.
    ## This might be necessary for (slow) serial devices.
     pause_between_requests = "500ms"

    ## Close the connection after every gather cycle.
    ## Usually the plugin closes the connection after a certain idle-timeout,
    ## however, if you query a device with limited simultaneous connectivity
    ## (e.g. serial devices) from multiple instances you might want to only
    ## stay connected during gather and disconnect afterwards.
    # close_connection_after_gather = false

    ## Force the plugin to read each field in a separate request.
    ## This might be necessary for devices not conforming to the spec,
    ## see https://github.com/influxdata/telegraf/issues/12071.
    # one_request_per_field = false

    ## Enforce the starting address to be zero for the first request on
    ## coil registers. This is necessary for some devices see
    ## https://github.com/influxdata/telegraf/issues/8905
    # read_coils_starting_at_zero = false
2

Hi,

You can add a second influxdb_v2 output and even set a different flush_interval on each of them.

3

Hi @jpowers,

But flush_interval writes all the captured data during that period. I try to capture the data at that period only. If i choose to use metric_batch_size, metric_buffer_limit than it may drop some data if the devices added future and need to calculate the buffer which also differ based on input collection.

4

Hi @jpowers,

Here i’m using the cpu input plugin and influxdb_v2 output plugin as test case. With 2 output of different flush_interval.

Test config looks like:

# Configuration for sending metrics to InfluxDB 2.0
[[outputs.influxdb_v2]]
  ## The URLs of the InfluxDB cluster nodes.
  ##
  ## Multiple URLs can be specified for a single cluster, only ONE of the
  ## urls will be written to each interval.
  ##   ex: urls = ["https://us-west-2-1.aws.cloud2.influxdata.com"]
  urls = ["http://127.0.0.1:8086"]

  ## Token for authentication.
  token = "token1"

  ## Organization is the name of the organization you wish to write to.
  organization = "OrgName"

  ## Destination bucket to write into.
  bucket = "testBucket1"
  
  flush_interval = "60s"



# Configuration for sending metrics to InfluxDB 2.0
[[outputs.influxdb_v2]]
  ## The URLs of the InfluxDB cluster nodes.
  ##
  ## Multiple URLs can be specified for a single cluster, only ONE of the
  ## urls will be written to each interval.
  ##   ex: urls = ["https://us-west-2-1.aws.cloud2.influxdata.com"]
  urls = ["http://127.0.0.1:8086"]

  ## Token for authentication.
  token = "token2"

  ## Organization is the name of the organization you wish to write to.
  organization = "OrgName"

  ## Destination bucket to write into.
  bucket = "testBucket2"
  
  flush_interval = "10s"


# Read metrics about cpu usage
[[inputs.cpu]]
  ## Whether to report per-cpu stats or not
  percpu = true
  ## Whether to report total system cpu stats or not
  totalcpu = true
  ## If true, collect raw CPU time metrics
  collect_cpu_time = false
  ## If true, compute and report the sum of all non-idle CPU states
  ## NOTE: The resulting 'time_active' field INCLUDES 'iowait'!
  report_active = false
  ## If true and the info is available then add core_id and physical_id tags
  core_tags = false

testBucket1 data looks like:

testBucket1
testBucket11600×820 158 KB

testBucket2 data looks like:

testBucket2
testBucket21600×821 152 KB

Expected behaviour for testBucket1 is not all the data that input gathered before the flush_interval. Only need the agent input data collection at the time of flush_interval loop.

Need suggestion on this.

5

@ManojKumar_Stru what you can do is to downsample the data using the final aggregator which only outputs the last datapoint in the set period:

[[aggregators.final]]
  period = "1h"
  output_strategy = "periodic"
  [aggregators.final]
    downsampled = "true"

I would also set a custom tag on those metrics to use the tagpass / tagdrop on the output side for routing the metrics. Is this what you want?

6

Hi @srebhan,

If aggregator is used then all the input data gathered will be down sampled right, but i am trying use the same input for both buckets( 1 with 5 minutes frequency, another with 1 hour frequency).

If i’m wrong about aggregators correct me.