Metric_buffer_limit can't exceed 1000?

Telegraf
,
1

Hi community,

In order to provice resiliency to an decentralized architecture (MQTT <–> Telegraf <–> Influxdb), I’m trying to increase the buffer size of Telegraf agent. But it looks like 1000 is the maximum buffer size I can reach for some reason. Am I missing something ?

Here is my configuration :

[agent]
  interval = "1s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 1000000
  collection_jitter = "0s"
  flush_interval = "1s"
  flush_jitter = "0s"
  debug = true
  quiet = false
  omit_hostname = false

###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################

[[outputs.influxdb]]
  urls = ["http://c_influxdb:8086"] # à adapter
  retention_policy = ""
  write_consistency = "any"
  timeout = "5s"

###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################

[[inputs.mqtt_consumer]]
   servers = ["tcp://192.168.1.200:1883"] # à adapter
   qos = 0
   connection_timeout = "30s"
   topics = ["topic/+"]
   client_id = "client"
   data_format = "influx"

Here is the log output, at 07:58:02 I stopped the influxdb container in order to simulate a breakdown (MQTT data are still published) :

2021-09-02T07:57:45Z I! Starting Telegraf 1.19.3
2021-09-02T07:57:45Z I! Using config file: /etc/telegraf/telegraf.conf
2021-09-02T07:57:45Z I! Loaded inputs: mqtt_consumer
2021-09-02T07:57:45Z I! Loaded aggregators:
2021-09-02T07:57:45Z I! Loaded processors:
2021-09-02T07:57:45Z I! Loaded outputs: influxdb
2021-09-02T07:57:45Z I! Tags enabled: host=aac5503d8de9
2021-09-02T07:57:45Z I! [agent] Config: Interval:1s, Quiet:false, Hostname:"aac5503d8de9", Flush Interval:1s
2021-09-02T07:57:45Z D! [agent] Initializing plugins
2021-09-02T07:57:45Z D! [agent] Connecting outputs
2021-09-02T07:57:45Z D! [agent] Attempting connection to [outputs.influxdb]
2021-09-02T07:57:45Z D! [agent] Successfully connected to outputs.influxdb
2021-09-02T07:57:45Z D! [agent] Starting service inputs
2021-09-02T07:57:45Z I! [inputs.mqtt_consumer] Connected [tcp://192.168.1.200:1883]
2021-09-02T07:57:46Z D! [outputs.influxdb] Wrote batch of 12 metrics in 9.37483ms
2021-09-02T07:57:46Z D! [outputs.influxdb] Buffer fullness: 2 / 1000000 metrics
2021-09-02T07:57:47Z D! [outputs.influxdb] Wrote batch of 110 metrics in 26.983634ms
2021-09-02T07:57:47Z D! [outputs.influxdb] Buffer fullness: 6 / 1000000 metrics
2021-09-02T07:57:48Z D! [outputs.influxdb] Wrote batch of 98 metrics in 10.499298ms
2021-09-02T07:57:48Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:57:49Z D! [outputs.influxdb] Wrote batch of 9 metrics in 6.769336ms
2021-09-02T07:57:49Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:57:50Z D! [outputs.influxdb] Wrote batch of 110 metrics in 7.75368ms
2021-09-02T07:57:50Z D! [outputs.influxdb] Buffer fullness: 2 / 1000000 metrics
2021-09-02T07:57:51Z D! [outputs.influxdb] Wrote batch of 101 metrics in 9.284954ms
2021-09-02T07:57:51Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:57:52Z D! [outputs.influxdb] Wrote batch of 9 metrics in 7.527304ms
2021-09-02T07:57:52Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:57:53Z D! [outputs.influxdb] Wrote batch of 105 metrics in 10.428176ms
2021-09-02T07:57:53Z D! [outputs.influxdb] Buffer fullness: 4 / 1000000 metrics
2021-09-02T07:57:54Z D! [outputs.influxdb] Wrote batch of 106 metrics in 7.684382ms
2021-09-02T07:57:54Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:57:55Z D! [outputs.influxdb] Wrote batch of 6 metrics in 5.569883ms
2021-09-02T07:57:55Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:57:56Z D! [outputs.influxdb] Wrote batch of 102 metrics in 10.259555ms
2021-09-02T07:57:56Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:57:57Z D! [outputs.influxdb] Wrote batch of 112 metrics in 8.495584ms
2021-09-02T07:57:57Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:57:58Z D! [outputs.influxdb] Wrote batch of 3 metrics in 7.109232ms
2021-09-02T07:57:58Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:57:59Z D! [outputs.influxdb] Wrote batch of 100 metrics in 9.661294ms
2021-09-02T07:57:59Z D! [outputs.influxdb] Buffer fullness: 3 / 1000000 metrics
2021-09-02T07:58:00Z D! [outputs.influxdb] Wrote batch of 117 metrics in 9.47556ms
2021-09-02T07:58:00Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:58:01Z D! [outputs.influxdb] Buffer fullness: 0 / 1000000 metrics
2021-09-02T07:58:02Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:02Z D! [outputs.influxdb] Buffer fullness: 103 / 1000000 metrics
2021-09-02T07:58:02Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:03Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:03Z D! [outputs.influxdb] Buffer fullness: 220 / 1000000 metrics
2021-09-02T07:58:03Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:04Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:04Z D! [outputs.influxdb] Buffer fullness: 220 / 1000000 metrics
2021-09-02T07:58:04Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:05Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:05Z D! [outputs.influxdb] Buffer fullness: 319 / 1000000 metrics
2021-09-02T07:58:05Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:06Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:06Z D! [outputs.influxdb] Buffer fullness: 438 / 1000000 metrics
2021-09-02T07:58:06Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:07Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:07Z D! [outputs.influxdb] Buffer fullness: 440 / 1000000 metrics
2021-09-02T07:58:07Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:08Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:08Z D! [outputs.influxdb] Buffer fullness: 536 / 1000000 metrics
2021-09-02T07:58:08Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:09Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:09Z D! [outputs.influxdb] Buffer fullness: 654 / 1000000 metrics
2021-09-02T07:58:09Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:10Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:10Z D! [outputs.influxdb] Buffer fullness: 660 / 1000000 metrics
2021-09-02T07:58:10Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:11Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:11Z D! [outputs.influxdb] Buffer fullness: 751 / 1000000 metrics
2021-09-02T07:58:11Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:12Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:12Z D! [outputs.influxdb] Buffer fullness: 871 / 1000000 metrics
2021-09-02T07:58:12Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:13Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:13Z D! [outputs.influxdb] Buffer fullness: 880 / 1000000 metrics
2021-09-02T07:58:13Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:14Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:14Z D! [outputs.influxdb] Buffer fullness: 968 / 1000000 metrics
2021-09-02T07:58:14Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:15Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:15Z D! [outputs.influxdb] Buffer fullness: 1000 / 1000000 metrics
2021-09-02T07:58:15Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:16Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:16Z D! [outputs.influxdb] Buffer fullness: 1000 / 1000000 metrics
2021-09-02T07:58:16Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:17Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:17Z D! [outputs.influxdb] Buffer fullness: 1000 / 1000000 metrics
2021-09-02T07:58:17Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:18Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:18Z D! [outputs.influxdb] Buffer fullness: 1000 / 1000000 metrics
2021-09-02T07:58:18Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:19Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:19Z D! [outputs.influxdb] Buffer fullness: 1000 / 1000000 metrics
2021-09-02T07:58:19Z E! [agent] Error writing to outputs.influxdb: could not write any address
2021-09-02T07:58:20Z E! [outputs.influxdb] When writing to [http://c_influxdb:8086]: failed doing req: Post "http://c_influxdb:8086/write?consistency=any&db=telegraf": dial tcp: lookup c_influxdb on 127.0.0.11:53: no such host
2021-09-02T07:58:20Z D! [outputs.influxdb] Buffer fullness: 1000 / 1000000 metrics
2021-09-02T07:58:20Z E! [agent] Error writing to outputs.influxdb: could not write any address

Thanks for your help,
Quentin.

2

That seems like a bug, could you define which version of telegraf you are using? Maybe this should be an issue on GitHub then…

3

I use the latest version 1.19.3.

4

I think best is to create an issue on GitHub

5

I have open an issue
Let’s see if I get answers :slight_smile:

6

Problem solved, it was du to the paramater max_undelivered_messages in the mqtt input plugin which is by default set to 1000