Telegraf config help please

Telegraf
1

Hi All,

I am having some issues with my telegraf config file.

Requirements:

  • I have a host that is outputting metrics and performance stats as multicast traffic on my network. These are getting collected by a node.js application and posted to influx / telegraf. These metrics are to go into the batrium database.
  • I want to also record and save the metrics for the telegraf instance into the telegraf database
  • Both databases are on the same influx host.
  • I want to use telegraf as a proxy between the host and influx in case influx goes offline, so I need it to buffer the results in memory until the influx host is back online. This will depend on available resources, but hoping for potential long term capacity (think 1 week)

I am unsure how many metrics are getting spewed out per second / minute so im hoping to also use the internal monitor to work this out so i can determine the possibility of how long it will buffer. 1 week is not a requirement, but honestly, as long as i can get I’ll take.

Initially for testing, telegraf is deployed on a NUC, but my plan is to move it to a PI Zero W as containers. There will only be 2 containers running on the pi, Telegraf and the node.js agent that collects the metrics from the server.

I have tried all sorts of things to get this to work, but no matter what, it seems that the host metrics and the telegraf metrics are getting posted into both databases. Below is some examples of config files i have tried. but none have worked. I have even tried using chatGPT to assist, but none of its outputs are having the desired outcome either.

I am new to telegraf, and perhaps its not the right tool for this, and i should look to move the data into RMQ as the proxy. However the node.js application only posts to influx, and im no expert in node.js, so im hoping to leave it as it is, and then introduce middleware to broker and proxy as needed.

Config 1:

[[inputs.influxdb_listener]]
  service_address = ":8086" 

[[outputs.influxdb]] 
  urls = ["http://192.168.100.250:8088"] 
  database = "batrium" 
  data_format = "influx" 


[[inputs.internal]] 
  collect_memstats = true

[[outputs.influxdb]] 
  urls = ["http://192.168.100.205:8088"] 
  database = "telegraf"
  data_format = "influx"

Config 2:

[[inputs.influxdb_listener]] 
  service_address = ":8086" 


[[outputs.influxdb]] 
  urls = ["http://192.168.100.250:8088"] 
  database = "batrium" 
  [[outputs.influxdb.tags]]  
    influxdb_database = "batrium" 



[[inputs.internal]] 
  collect_memstats = true

[[outputs.influxdb]] 
  urls = ["http://192.168.100.250:8088"] 
  database = "telegraf" 
  [[outputs.influxdb.tags]] 
    influxdb_database = "telegraf"

Config 3:

[[inputs.influxdb_listener]]

  tags = ["watchmon"]
  service_address = ":8086"




[[inputs.internal]]

  tags = ["telegraf"]
  collect_memstats = true



# OUTPUTS
[[outputs.influxdb]]
  tags = ["watchmon"]
  url = "http://192.168.100.250:8088"
  database = "batrium" # required.
  precision = "1s"
  username = "batrium"
  password = "batrium"
  namedrop = ["telegraf"]

[[outputs.influxdb]]
  tags = ["telegraf"]
  url = "http://192.168.100.250:8087"
  database = "telegraf"
  precision = "1s"
  username = "telegraf"
  password = "telegraf"
  namedrop = ["batrium"]

So im kinda stumped on where the issue lies, but happy to take feedback.

Thanks.

2

Doing what you ask isn’t hard, but it must be clear how the data are routed into different Databases.

To do that you can either use:

  • Telegraf selectors, to filter what data passes into the output, for your case probably based on measurement names (namepass or namedrop)
  • The InfluxDB output and its database_tag. this requires you to put an additional tag in your data to route them properly.
    (static tags can be added even via telegraf, with a mix of processors and metric filtering, but for your case I think its it’s just a longer route…)

The internal plugin produces a set of measurements starting with “internal”, therefore using something like the sample below should work

[[outputs.influxdb]] 
  urls = ["http://192.168.100.250:8088"] 
  database = "batrium" 
  data_format = "influx"
  namedrop = ["internal*"]

[[outputs.influxdb]] 
  urls = ["http://192.168.100.205:8088"] 
  database = "telegraf"
  data_format = "influx" 
  namepass = ["internal*"]
3

You sir. Are a legend. Thanks for that.

Now, i just need to play with the buffer size to get it as large as possible.

I did some trsting with it at 100k, that got ne about 4 hours of buffering. Next will try 1MM see how that goes. And will watch the memory usage on the docker host whilst im at it.

1 Like