Telegraf metric_buffer_limit can't exceed 20

Telegraf
1

Hi everyone!
My issue is similar to Metric_buffer_limit can't exceed 1000? but I the resolution posted there didn’t work in my case. I’m pretty new in this area and I’m out of ideas, honestly. I’d be really thankful if you gave me any tips, what could I’ve been doing wrong.

Long story short - I’m trying to configure metric_buffer_limit to 10000 on the Telegraf agent, but as I’m testing it (by disabling connection between Telegraf and Influx), the actual buffer never exceeds 20 (despite MQTT messages still incoming).

On the InfluxDB side, I don’t see any of the metrics collected during the test time, even those 20 that should’ve been delivered. I only get the newest values for each of the subscribed topics (which there are 5 in total), but no “historical” data.

I’m running Telegraf 1.32.2 on Raspbian GNU/Linux 11 connected to InfluxDB v2.7.10 on Ubuntu 22.04.3.

Here’s my config file:

[agent]
  interval = "5s"
  round_interval = true
  flush_interval = "5s"
  flush_jitter = "0s"
  debug = true
  quiet = false
  logfile = ""
  hostname = ""
  metric_batch_size = 1000
  metric_buffer_limit = 10000

#####################################################################
#                            INPUT                                  #
#####################################################################
[[inputs.mqtt_consumer]]
  servers = ["tcp://localhost:1883"]
  topics = [
    "#"
  ]
  qos = 1

  # I've added this after consulting mentioned topic, but nothing changed
  max_undelivered_messages = 100000 

  connection_timeout = "30s"
  client_id = "telegraf_mqtt_consumer"
  data_format = "value"
  data_type = "float"
  persistent_session = true
  topic_tag = "topic"

  [[inputs.mqtt_consumer.topic_parsing]]
    topic = "home/+/+"
    measurement = "_/_/pozycja"
    tags = "location/device/_"


####################################################################
#                            OUTPUT                                #
####################################################################
[[outputs.influxdb_v2]]
  urls = ["http://10.0.0.1:8086"]
  token = "<TOKEN>"
  organization = "dzialka"
  bucket = "dzialka_monitoring_new"
  timeout = "10s"

My logs for telegraf:

Nov 16 22:35:08 raspberrypi systemd[1]: Starting Telegraf...
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z I! Loading config: /etc/telegraf/telegraf.conf
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z I! Starting Telegraf 1.32.2 brought to you by InfluxData the makers of InfluxDB
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z I! Available plugins: 235 inputs, 9 aggregators, 32 processors, 26 parsers, 62 outputs, 6 secret-stores
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z I! Loaded inputs: mqtt_consumer
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z I! Loaded aggregators:
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z I! Loaded processors:
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z I! Loaded secretstores:
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z I! Loaded outputs: influxdb_v2
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z I! Tags enabled: host=raspberrypi
Nov 16 22:35:10 raspberrypi systemd[1]: Started Telegraf.
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z I! [agent] Config: Interval:5s, Quiet:false, Hostname:"raspberrypi", Flush Interval:5s
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z D! [agent] Initializing plugins
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z D! [agent] Connecting outputs
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z D! [agent] Attempting connection to [outputs.influxdb_v2]
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z D! [agent] Successfully connected to outputs.influxdb_v2
Nov 16 22:35:10 raspberrypi telegraf[1945]: 2024-11-16T21:35:10Z D! [agent] Starting service inputs
Nov 16 22:35:11 raspberrypi telegraf[1945]: 2024-11-16T21:35:11Z I! [inputs.mqtt_consumer] Connected [tcp://localhost:1883]
Nov 16 22:35:11 raspberrypi telegraf[1945]: 2024-11-16T21:35:11Z D! [inputs.mqtt_consumer] Session found [tcp://localhost:1883]
Nov 16 22:35:16 raspberrypi telegraf[1945]: 2024-11-16T21:35:16Z D! [outputs.influxdb_v2] Wrote batch of 6 metrics in 78.513069ms
Nov 16 22:35:16 raspberrypi telegraf[1945]: 2024-11-16T21:35:16Z D! [outputs.influxdb_v2] Buffer fullness: 1 / 10000 metrics
Nov 16 22:35:21 raspberrypi telegraf[1945]: 2024-11-16T21:35:21Z D! [outputs.influxdb_v2] Wrote batch of 3 metrics in 52.808375ms
Nov 16 22:35:21 raspberrypi telegraf[1945]: 2024-11-16T21:35:21Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
Nov 16 22:35:26 raspberrypi telegraf[1945]: 2024-11-16T21:35:26Z D! [outputs.influxdb_v2] Wrote batch of 3 metrics in 38.762542ms
Nov 16 22:35:26 raspberrypi telegraf[1945]: 2024-11-16T21:35:26Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
Nov 16 22:35:31 raspberrypi telegraf[1945]: 2024-11-16T21:35:31Z D! [outputs.influxdb_v2] Wrote batch of 3 metrics in 55.832341ms
Nov 16 22:35:31 raspberrypi telegraf[1945]: 2024-11-16T21:35:31Z D! [outputs.influxdb_v2] Buffer fullness: 1 / 10000 metrics
Nov 16 22:35:36 raspberrypi telegraf[1945]: 2024-11-16T21:35:36Z D! [outputs.influxdb_v2] Wrote batch of 3 metrics in 47.874435ms
Nov 16 22:35:36 raspberrypi telegraf[1945]: 2024-11-16T21:35:36Z D! [outputs.influxdb_v2] Buffer fullness: 1 / 10000 metrics
Nov 16 22:35:41 raspberrypi telegraf[1945]: 2024-11-16T21:35:41Z D! [outputs.influxdb_v2] Wrote batch of 6 metrics in 47.957436ms
Nov 16 22:35:41 raspberrypi telegraf[1945]: 2024-11-16T21:35:41Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
Nov 16 22:35:46 raspberrypi telegraf[1945]: 2024-11-16T21:35:46Z D! [outputs.influxdb_v2] Wrote batch of 8 metrics in 43.179492ms
Nov 16 22:35:46 raspberrypi telegraf[1945]: 2024-11-16T21:35:46Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
Nov 16 22:35:51 raspberrypi telegraf[1945]: 2024-11-16T21:35:51Z D! [outputs.influxdb_v2] Wrote batch of 6 metrics in 41.540512ms
Nov 16 22:35:51 raspberrypi telegraf[1945]: 2024-11-16T21:35:51Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

#At this moment, I've simulated loosing connection to InfluxDB

Nov 16 22:35:56 raspberrypi telegraf[1945]: 2024-11-16T21:35:56Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:35:56 raspberrypi telegraf[1945]: 2024-11-16T21:35:56Z D! [outputs.influxdb_v2] Buffer fullness: 6 / 10000 metrics
Nov 16 22:35:56 raspberrypi telegraf[1945]: 2024-11-16T21:35:56Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:01 raspberrypi telegraf[1945]: 2024-11-16T21:36:01Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:01 raspberrypi telegraf[1945]: 2024-11-16T21:36:01Z D! [outputs.influxdb_v2] Buffer fullness: 10 / 10000 metrics
Nov 16 22:36:01 raspberrypi telegraf[1945]: 2024-11-16T21:36:01Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:06 raspberrypi telegraf[1945]: 2024-11-16T21:36:06Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:06 raspberrypi telegraf[1945]: 2024-11-16T21:36:06Z D! [outputs.influxdb_v2] Buffer fullness: 15 / 10000 metrics
Nov 16 22:36:06 raspberrypi telegraf[1945]: 2024-11-16T21:36:06Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:11 raspberrypi telegraf[1945]: 2024-11-16T21:36:11Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:11 raspberrypi telegraf[1945]: 2024-11-16T21:36:11Z D! [outputs.influxdb_v2] Buffer fullness: 19 / 10000 metrics
Nov 16 22:36:11 raspberrypi telegraf[1945]: 2024-11-16T21:36:11Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:16 raspberrypi telegraf[1945]: 2024-11-16T21:36:16Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:16 raspberrypi telegraf[1945]: 2024-11-16T21:36:16Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 16 22:36:16 raspberrypi telegraf[1945]: 2024-11-16T21:36:16Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:21 raspberrypi telegraf[1945]: 2024-11-16T21:36:21Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:21 raspberrypi telegraf[1945]: 2024-11-16T21:36:21Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 16 22:36:21 raspberrypi telegraf[1945]: 2024-11-16T21:36:21Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:26 raspberrypi telegraf[1945]: 2024-11-16T21:36:26Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:26 raspberrypi telegraf[1945]: 2024-11-16T21:36:26Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 16 22:36:26 raspberrypi telegraf[1945]: 2024-11-16T21:36:26Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:32 raspberrypi telegraf[1945]: 2024-11-16T21:36:32Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:32 raspberrypi telegraf[1945]: 2024-11-16T21:36:32Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 16 22:36:32 raspberrypi telegraf[1945]: 2024-11-16T21:36:32Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:36 raspberrypi telegraf[1945]: 2024-11-16T21:36:36Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:36 raspberrypi telegraf[1945]: 2024-11-16T21:36:36Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 16 22:36:36 raspberrypi telegraf[1945]: 2024-11-16T21:36:36Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:41 raspberrypi telegraf[1945]: 2024-11-16T21:36:41Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:41 raspberrypi telegraf[1945]: 2024-11-16T21:36:41Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 16 22:36:41 raspberrypi telegraf[1945]: 2024-11-16T21:36:41Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:46 raspberrypi telegraf[1945]: 2024-11-16T21:36:46Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:46 raspberrypi telegraf[1945]: 2024-11-16T21:36:46Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 16 22:36:46 raspberrypi telegraf[1945]: 2024-11-16T21:36:46Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 16 22:36:51 raspberrypi telegraf[1945]: 2024-11-16T21:36:51Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 16 22:36:51 raspberrypi telegraf[1945]: 2024-11-16T21:36:51Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 16 22:36:51 raspberrypi telegraf[1945]: 2024-11-16T21:36:51Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)

#Resumed normal network connectivity

Nov 16 22:36:56 raspberrypi telegraf[1945]: 2024-11-16T21:36:56Z D! [outputs.influxdb_v2] Wrote batch of 20 metrics in 73.432153ms
Nov 16 22:36:56 raspberrypi telegraf[1945]: 2024-11-16T21:36:56Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
Nov 16 22:37:01 raspberrypi telegraf[1945]: 2024-11-16T21:37:01Z D! [outputs.influxdb_v2] Wrote batch of 41 metrics in 58.559326ms
Nov 16 22:37:01 raspberrypi telegraf[1945]: 2024-11-16T21:37:01Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
Nov 16 22:37:06 raspberrypi telegraf[1945]: 2024-11-16T21:37:06Z D! [outputs.influxdb_v2] Wrote batch of 3 metrics in 34.574602ms
Nov 16 22:37:06 raspberrypi telegraf[1945]: 2024-11-16T21:37:06Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

Thank you for your help

2

Are you sure more messages arrive in the time when the “connection is lost”? Could you add an output like

[[outputs.file]]
  files = ["stdout"]

and check if you see new messages arriving!?

1 Like
3

Okay, so something very weird happened:
During the time when there was no connection to the Influx, outputs.file was displaying new messages only until buffer on outputs.influxdb_v2 reached 20. After that - silence, yet I could see on the MQTT broker that new messages were still incoming.
However, after restoring connection to Influx, I got flooded with 300+ metrics that got collected during the downtime on the stdout. Supposedly, the same batch was written to the Influx, but I cannot see it in the database.

As it comes to logs (sorry for them being so long):

######################## NORMAL OPERATION ########################


Nov 26 23:23:53 raspberrypi telegraf[1447]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=26.8 1732659830116380945
Nov 26 23:23:53 raspberrypi telegraf[1447]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5310623 1732659830321094270
Nov 26 23:23:53 raspberrypi telegraf[1447]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=26.7 1732659832879031349
Nov 26 23:23:53 raspberrypi telegraf[1447]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5294505 1732659832942483598
Nov 26 23:23:53 raspberrypi telegraf[1447]: humidity,device=rpi,host=raspberrypi,location=home,topic=home/rpi/humidity value=48.1 1732659833392978190
Nov 26 23:23:53 raspberrypi telegraf[1447]: 2024-11-26T22:23:53Z D! [outputs.file] Wrote batch of 5 metrics in 6.605126ms
Nov 26 23:23:53 raspberrypi telegraf[1447]: 2024-11-26T22:23:53Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:23:53 raspberrypi telegraf[1447]: 2024-11-26T22:23:53Z D! [outputs.influxdb_v2] Wrote batch of 5 metrics in 76.739456ms
Nov 26 23:23:53 raspberrypi telegraf[1447]: 2024-11-26T22:23:53Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

######################## DISABLED INFLUX AT THIS MOMENT ########################

Nov 26 23:23:58 raspberrypi telegraf[1447]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=26.8 1732659835348636136
Nov 26 23:23:58 raspberrypi telegraf[1447]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5286447 1732659837798894575
Nov 26 23:23:58 raspberrypi telegraf[1447]: 2024-11-26T22:23:58Z D! [outputs.file] Wrote batch of 2 metrics in 4.161065ms
Nov 26 23:23:58 raspberrypi telegraf[1447]: 2024-11-26T22:23:58Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:23:58 raspberrypi telegraf[1447]: 2024-11-26T22:23:58Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:23:58 raspberrypi telegraf[1447]: 2024-11-26T22:23:58Z D! [outputs.influxdb_v2] Buffer fullness: 2 / 10000 metrics
Nov 26 23:23:58 raspberrypi telegraf[1447]: 2024-11-26T22:23:58Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 26 23:24:03 raspberrypi telegraf[1447]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=26.7 1732659840255378085
Nov 26 23:24:03 raspberrypi telegraf[1447]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=49.7 1732659840458259992
Nov 26 23:24:03 raspberrypi telegraf[1447]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=26.8 1732659843019520841
Nov 26 23:24:03 raspberrypi telegraf[1447]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=49.9 1732659843135384288
Nov 26 23:24:03 raspberrypi telegraf[1447]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5125275 1732659843391159483
Nov 26 23:24:03 raspberrypi telegraf[1447]: 2024-11-26T22:24:03Z D! [outputs.file] Wrote batch of 5 metrics in 899.011µs
Nov 26 23:24:03 raspberrypi telegraf[1447]: 2024-11-26T22:24:03Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:24:03 raspberrypi telegraf[1447]: 2024-11-26T22:24:03Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:24:03 raspberrypi telegraf[1447]: 2024-11-26T22:24:03Z D! [outputs.influxdb_v2] Buffer fullness: 7 / 10000 metrics
Nov 26 23:24:03 raspberrypi telegraf[1447]: 2024-11-26T22:24:03Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 26 23:24:08 raspberrypi telegraf[1447]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=49.8 1732659845804718094
Nov 26 23:24:08 raspberrypi telegraf[1447]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5254212 1732659845864929776
Nov 26 23:24:08 raspberrypi telegraf[1447]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=26.7 1732659848359246718
Nov 26 23:24:08 raspberrypi telegraf[1447]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=49.9 1732659848549419547
Nov 26 23:24:08 raspberrypi telegraf[1447]: 2024-11-26T22:24:08Z D! [outputs.file] Wrote batch of 4 metrics in 9.236088ms
Nov 26 23:24:08 raspberrypi telegraf[1447]: 2024-11-26T22:24:08Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:24:08 raspberrypi telegraf[1447]: 2024-11-26T22:24:08Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:24:08 raspberrypi telegraf[1447]: 2024-11-26T22:24:08Z D! [outputs.influxdb_v2] Buffer fullness: 11 / 10000 metrics
Nov 26 23:24:08 raspberrypi telegraf[1447]: 2024-11-26T22:24:08Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 26 23:24:13 raspberrypi telegraf[1447]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5262271 1732659851005506541
Nov 26 23:24:13 raspberrypi telegraf[1447]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=26.8 1732659853470343231
Nov 26 23:24:13 raspberrypi telegraf[1447]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=50 1732659853538517705
Nov 26 23:24:13 raspberrypi telegraf[1447]: 2024-11-26T22:24:13Z D! [outputs.file] Wrote batch of 3 metrics in 5.475038ms
Nov 26 23:24:13 raspberrypi telegraf[1447]: 2024-11-26T22:24:13Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:24:13 raspberrypi telegraf[1447]: 2024-11-26T22:24:13Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:24:13 raspberrypi telegraf[1447]: 2024-11-26T22:24:13Z D! [outputs.influxdb_v2] Buffer fullness: 14 / 10000 metrics
Nov 26 23:24:13 raspberrypi telegraf[1447]: 2024-11-26T22:24:13Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 26 23:24:18 raspberrypi telegraf[1447]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5230037 1732659854510625205
Nov 26 23:24:18 raspberrypi telegraf[1447]: humidity,device=rpi,host=raspberrypi,location=home,topic=home/rpi/humidity value=48.2 1732659856181709303
Nov 26 23:24:18 raspberrypi telegraf[1447]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=49.8 1732659857151817521
Nov 26 23:24:18 raspberrypi telegraf[1447]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5213919 1732659857355773534
Nov 26 23:24:18 raspberrypi telegraf[1447]: 2024-11-26T22:24:18Z D! [outputs.file] Wrote batch of 4 metrics in 5.383024ms
Nov 26 23:24:18 raspberrypi telegraf[1447]: 2024-11-26T22:24:18Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:24:18 raspberrypi telegraf[1447]: 2024-11-26T22:24:18Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:24:18 raspberrypi telegraf[1447]: 2024-11-26T22:24:18Z D! [outputs.influxdb_v2] Buffer fullness: 18 / 10000 metrics
Nov 26 23:24:18 raspberrypi telegraf[1447]: 2024-11-26T22:24:18Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)

######################## LAST MESSAGES BEFORE OUTPUTS.INFLUXDB_V2 BUFFER REACHED 20 ########################

Nov 26 23:24:23 raspberrypi telegraf[1447]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=26.7 1732659859913799933
Nov 26 23:24:23 raspberrypi telegraf[1447]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5205861 1732659860121243713
Nov 26 23:24:23 raspberrypi telegraf[1447]: 2024-11-26T22:24:23Z D! [outputs.file] Wrote batch of 2 metrics in 3.875009ms
Nov 26 23:24:23 raspberrypi telegraf[1447]: 2024-11-26T22:24:23Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:24:23 raspberrypi telegraf[1447]: 2024-11-26T22:24:23Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:24:23 raspberrypi telegraf[1447]: 2024-11-26T22:24:23Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 26 23:24:23 raspberrypi telegraf[1447]: 2024-11-26T22:24:23Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 26 23:24:28 raspberrypi telegraf[1447]: 2024-11-26T22:24:28Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:24:28 raspberrypi telegraf[1447]: 2024-11-26T22:24:28Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:24:28 raspberrypi telegraf[1447]: 2024-11-26T22:24:28Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 26 23:24:28 raspberrypi telegraf[1447]: 2024-11-26T22:24:28Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 26 23:24:33 raspberrypi telegraf[1447]: 2024-11-26T22:24:33Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:24:33 raspberrypi telegraf[1447]: 2024-11-26T22:24:33Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:24:33 raspberrypi telegraf[1447]: 2024-11-26T22:24:33Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 26 23:24:33 raspberrypi telegraf[1447]: 2024-11-26T22:24:33Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 26 23:24:38 raspberrypi telegraf[1447]: 2024-11-26T22:24:38Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:24:38 raspberrypi telegraf[1447]: 2024-11-26T22:24:38Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:24:38 raspberrypi telegraf[1447]: 2024-11-26T22:24:38Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 26 23:24:38 raspberrypi telegraf[1447]: 2024-11-26T22:24:38Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)


######################## I LEFT IT RUNNING LIKE THAT FOR  ~7 MINUTES ########################


Nov 26 23:30:50 raspberrypi telegraf[1527]: 2024-11-26T22:30:50Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:30:50 raspberrypi telegraf[1527]: 2024-11-26T22:30:50Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:30:50 raspberrypi telegraf[1527]: 2024-11-26T22:30:50Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 26 23:30:50 raspberrypi telegraf[1527]: 2024-11-26T22:30:50Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 26 23:30:55 raspberrypi telegraf[1527]: 2024-11-26T22:30:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:30:55 raspberrypi telegraf[1527]: 2024-11-26T22:30:55Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:30:55 raspberrypi telegraf[1527]: 2024-11-26T22:30:55Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 26 23:30:55 raspberrypi telegraf[1527]: 2024-11-26T22:30:55Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 26 23:31:00 raspberrypi telegraf[1527]: 2024-11-26T22:31:00Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:31:00 raspberrypi telegraf[1527]: 2024-11-26T22:31:00Z E! [outputs.influxdb_v2] When writing to [http://10.0.0.1:8086/api/v2/write]: Post "http://10.0.0.1:8086/api/v2/write?bucket=dzialka_monitoring_new&org=dzialka": dial tcp 10.0.0.1:8086: connect: connection refused
Nov 26 23:31:00 raspberrypi telegraf[1527]: 2024-11-26T22:31:00Z D! [outputs.influxdb_v2] Buffer fullness: 20 / 10000 metrics
Nov 26 23:31:00 raspberrypi telegraf[1527]: 2024-11-26T22:31:00Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
Nov 26 23:31:05 raspberrypi telegraf[1527]: 2024-11-26T22:31:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:31:05 raspberrypi telegraf[1527]: 2024-11-26T22:31:05Z D! [outputs.influxdb_v2] Wrote batch of 20 metrics in 154.884459ms
Nov 26 23:31:05 raspberrypi telegraf[1527]: 2024-11-26T22:31:05Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

######################## STARTED INFLUX AT THIS MOMENT ########################

Nov 26 23:31:10 raspberrypi telegraf[1527]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=26.8 1732660265914252608
Nov 26 23:31:10 raspberrypi telegraf[1527]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=49.7 1732660265937515377
Nov 26 23:31:10 raspberrypi telegraf[1527]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5165568 1732660265937792374
Nov 26 23:31:10 raspberrypi telegraf[1527]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.503663 1732660265938623366
Nov 26 23:31:10 raspberrypi telegraf[1527]: humidity,device=rpi,host=raspberrypi,location=home,topic=home/rpi/humidity value=48.3 1732660265946688286
Nov 26 23:31:10 raspberrypi telegraf[1527]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.5165568 1732660265951599237

######################## 300+ METRICS GOT DISPLAYED HERE IN A SPAN OF 2 SECONDS, I CUT OUT MOST OF THEM FOR CLARITY ########################

Nov 26 23:31:12 raspberrypi telegraf[1527]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=52.1 1732660266347974271
Nov 26 23:31:12 raspberrypi telegraf[1527]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.3610256 1732660266348208269
Nov 26 23:31:12 raspberrypi telegraf[1527]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=27.3 1732660266348608265
Nov 26 23:31:12 raspberrypi telegraf[1527]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=51.9 1732660266349039261
Nov 26 23:31:12 raspberrypi telegraf[1527]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.3561905 1732660266350400247
Nov 26 23:31:12 raspberrypi telegraf[1527]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=27.2 1732660266350691244
Nov 26 23:31:12 raspberrypi telegraf[1527]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=51.7 1732660266350978241
Nov 26 23:31:12 raspberrypi telegraf[1527]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.3602198 1732660266351374237
Nov 26 23:31:12 raspberrypi telegraf[1527]: humidity,device=rpi,host=raspberrypi,location=home,topic=home/rpi/humidity value=49.1 1732660266351914232

######################## GOT 2 BIG BATCHES WRITTEN AT THE SAME TIME ########################


Nov 26 23:31:12 raspberrypi telegraf[1527]: 2024-11-26T22:31:10Z D! [outputs.file] Wrote batch of 365 metrics in 201.998919ms
Nov 26 23:31:12 raspberrypi telegraf[1527]: 2024-11-26T22:31:10Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:31:12 raspberrypi telegraf[1527]: 2024-11-26T22:31:11Z D! [outputs.influxdb_v2] Wrote batch of 365 metrics in 289.906013ms
Nov 26 23:31:12 raspberrypi telegraf[1527]: 2024-11-26T22:31:11Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
Nov 26 23:31:15 raspberrypi telegraf[1527]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=27.3 1732660271035151595
Nov 26 23:31:15 raspberrypi telegraf[1527]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=51.4 1732660271070267231
Nov 26 23:31:15 raspberrypi telegraf[1527]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.3594139 1732660271070522228
Nov 26 23:31:15 raspberrypi telegraf[1527]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=51.3 1732660271071047223
Nov 26 23:31:15 raspberrypi telegraf[1527]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.3569963 1732660271095440970
Nov 26 23:31:15 raspberrypi telegraf[1527]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=51.4 1732660272605758261
Nov 26 23:31:15 raspberrypi telegraf[1527]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.3481319 1732660272685875424
Nov 26 23:31:15 raspberrypi telegraf[1527]: temperature,device=esp1,host=raspberrypi,location=home,topic=home/esp1/temperature value=27.2 1732660275159780355
Nov 26 23:31:15 raspberrypi telegraf[1527]: humidity,device=esp1,host=raspberrypi,location=home,topic=home/esp1/humidity value=51.2 1732660275225959650
Nov 26 23:31:15 raspberrypi telegraf[1527]: co_voltage,device=esp1,host=raspberrypi,location=home,topic=home/esp1/co_voltage value=0.3489377 1732660275573069954
Nov 26 23:31:15 raspberrypi telegraf[1527]: 2024-11-26T22:31:15Z D! [outputs.file] Wrote batch of 10 metrics in 27.305709ms
Nov 26 23:31:15 raspberrypi telegraf[1527]: 2024-11-26T22:31:15Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 26 23:31:15 raspberrypi telegraf[1527]: 2024-11-26T22:31:15Z D! [outputs.influxdb_v2] Wrote batch of 10 metrics in 84.450101ms
Nov 26 23:31:15 raspberrypi telegraf[1527]: 2024-11-26T22:31:15Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics


######################## RESUMED NORMAL OPERATION ########################

Also, maybe to clarify my architecture.

I’m running something like this

ESP32 (with sensors) —> RPi (mosquitto —> Telegraf) —> Remote Ubuntu Server (InfluxDB).

Both MQTT broker and Telegraf are on the same physical device, so I doubt there are any connection issues between those two.

I’ll be very grateful for any insight that you might provide, thank you in advance.

4

Please check the max_undelivered_messages setting in the mqtt-consumer plugin. The MQTT consumer uses tracking metrics to not overflow the metric buffer and not loose data. Those special metrics notify the input as soon as they are delivered to the output and then stop the consumption of more than the mentioned undelivered messages exist.