["outputs.influxdb"] did not complete within its flush interval

Telegraf
1

Hi, can anyone help to sort out why there is errors: [“outputs.influxdb”] did not complete within its flush interval. Flush Interval was 10s, I have increased it to 15s but still getting errors.

Nov 30 13:28:57 home-home-grafana-1 systemd[1]: Reloading The plugin-driven server agent for reporting metrics into InfluxDB.
Nov 30 13:28:57 home-home-grafana-1 telegraf[972]: 2020-11-30T11:28:57Z I! Reloading Telegraf config
Nov 30 13:28:57 home-home-grafana-1 telegraf[972]: 2020-11-30T11:28:57Z I! [agent] Hang on, flushing any cached metrics before shutdown
Nov 30 13:28:57 home-home-grafana-1 systemd[1]: Reloaded The plugin-driven server agent for reporting metrics into InfluxDB.
Nov 30 13:28:57 home-home-grafana-1 telegraf[972]: 2020-11-30T11:28:57Z I! Starting Telegraf 1.16.1
Nov 30 13:28:57 home-home-grafana-1 telegraf[972]: 2020-11-30T11:28:57Z I! Loaded inputs: cpu disk diskio kernel mem net netstat processes snmp (13x) swap system
Nov 30 13:28:57 home-home-grafana-1 telegraf[972]: 2020-11-30T11:28:57Z I! Loaded aggregators:
Nov 30 13:28:57 home-home-grafana-1 telegraf[972]: 2020-11-30T11:28:57Z I! Loaded processors:
Nov 30 13:28:57 home-home-grafana-1 telegraf[972]: 2020-11-30T11:28:57Z I! Loaded outputs: influxdb
Nov 30 13:28:57 home-home-grafana-1 telegraf[972]: 2020-11-30T11:28:57Z I! Tags enabled: host=home-home-grafana-1
Nov 30 13:28:57 home-home-grafana-1 telegraf[972]: 2020-11-30T11:28:57Z I! [agent] Config: Interval:5m0s, Quiet:false, Hostname:“home-home-grafana-1”, Flush Interval:15s
Nov 30 13:29:11 home-home-grafana-1 systemd[1]: Reloading The plugin-driven server agent for reporting metrics into InfluxDB.
Nov 30 13:29:11 home-home-grafana-1 telegraf[972]: 2020-11-30T11:29:11Z I! Reloading Telegraf config
Nov 30 13:29:11 home-home-grafana-1 telegraf[972]: 2020-11-30T11:29:11Z I! [agent] Hang on, flushing any cached metrics before shutdown
Nov 30 13:29:11 home-home-grafana-1 systemd[1]: Reloaded The plugin-driven server agent for reporting metrics into InfluxDB.
Nov 30 13:29:11 home-home-grafana-1 telegraf[972]: 2020-11-30T11:29:11Z I! Starting Telegraf 1.16.1
Nov 30 13:29:11 home-home-grafana-1 telegraf[972]: 2020-11-30T11:29:11Z I! Loaded inputs: cpu disk diskio kernel mem net netstat processes snmp (13x) swap system
Nov 30 13:29:11 home-home-grafana-1 telegraf[972]: 2020-11-30T11:29:11Z I! Loaded aggregators:
Nov 30 13:29:11 home-home-grafana-1 telegraf[972]: 2020-11-30T11:29:11Z I! Loaded processors:
Nov 30 13:29:11 home-home-grafana-1 telegraf[972]: 2020-11-30T11:29:11Z I! Loaded outputs: influxdb
Nov 30 13:29:11 home-home-grafana-1 telegraf[972]: 2020-11-30T11:29:11Z I! Tags enabled: home-home-grafana-1
Nov 30 13:29:11 home-home-grafana-1 telegraf[972]: 2020-11-30T11:29:11Z I! [agent] Config: Interval:5m0s, Quiet:false, Hostname:“home-home-grafana-1”, Flush Interval:15s
Nov 30 13:30:56 home-home-grafana-1 telegraf[972]: 2020-11-30T11:30:56Z W! [agent] [“outputs.influxdb”] did not complete within its flush interval
Nov 30 14:30:26 home-home-grafana-1 telegraf[972]: 2020-11-30T12:30:26Z W! [agent] [“outputs.influxdb”] did not complete within its flush interval
Nov 30 15:05:41 home-home-grafana-1 telegraf[972]: 2020-11-30T13:05:41Z W! [agent] [“outputs.influxdb”] did not complete within its flush interval
Nov 30 16:46:11 home-home-grafana-1 telegraf[972]: 2020-11-30T14:46:11Z W! [agent] [“outputs.influxdb”] did not complete within its flush interval
Nov 30 16:56:26 home-home-grafana-1 telegraf[972]: 2020-11-30T14:56:26Z W! [agent] [“outputs.influxdb”] did not complete within its flush interval
Nov 30 19:35:56 home-home-grafana-1 telegraf[972]: 2020-11-30T17:35:56Z W! [agent] [“outputs.influxdb”] did not complete within its flush interval
Nov 30 21:20:26 home-home-grafana-1 telegraf[972]: 2020-11-30T19:20:26Z W! [agent] [“outputs.influxdb”] did not complete within its flush interval
Nov 30 21:26:11 home-home-grafana-1 telegraf[972]: 2020-11-30T19:26:11Z W! [agent] [“outputs.influxdb”] did not complete within its flush interval
Nov 30 21:45:56 home-home-grafana-1 telegraf[972]: 2020-11-30T19:45:56Z W! [agent] [“outputs.influxdb”] did not complete within its flush interval
Nov 30 22:36:26 home-home-grafana-1 telegraf[972]: 2020-11-30T20:36:26Z W! [agent] [“outputs.influxdb”] did not complete within its flush interval

2

The problem is simple, Telegraf is unable to write all the gathered data inside the defined interval.
Can you post your current InfluxDB output and agent configuration sections?
How many points are you gathering per second? (or minute or whatever)

There are some settings that could help you, like the metric_batch_size but you must tune the config based on your scenario

1 Like
3

Stats are collected from 6 nodes (in few months + 4 nodes), every 300 sec. Regarding config:

interval = “300s”
metric_batch_size = 11000
metric_buffer_limit = 12000
collection_jitter = “0s”
flush_interval = “22s”
flush_jitter = “0s”

What could be changed to get rid of “did not complete within its flush interval” warning?

Thanks!

4

Raising the flush_interval should do the trick

From what I see, you fetch data every 5mins but write the gathered data every 22sec, I think this makes sense only if you are overriding the interval in some input plugins to fetch data in a way higher frequency.

I’d set the flush interval to 150s, (or even less like 60s) this means that even if for some reason the first write fails it will (try to) write again (at least once before) new data gets fetched.
I just think it’s pointless to fetch data one time and then write ~25 times in the same interval, you simply don’t have any new data to write.

You can also set the metric_batch_size to a lower value, the suggested one should be 5000. The bigger the batch the more time needed to prepare it, which could be part of the reason for your timeouts.

1 Like