Why writing to inputs.http_listener is much slower then wirting to influxdb?

There is two influxdb output in a telegraf config, one is going to an actual influxdb, one is going to a telegraf with inputs.http_listener (both going over https, content_encoding = "gzip", telegraf 1.4.4 => telegraf 1.5 )
Every second write in the following logs is going to a telegraf (inputs.http_listener) which is exactly where influxdb located.
Do anyone have any idea why is it so much slower ?

2017-12-14T14:01:00Z D! Output [influxdb] buffer fullness: 74 / 10000 metrics.
2017-12-14T14:01:00Z D! Output [influxdb] buffer fullness: 74 / 10000 metrics.
2017-12-14T14:01:01Z D! Output [influxdb] wrote batch of 74 metrics in 1.024278158s
2017-12-14T14:01:22Z D! Output [influxdb] wrote batch of 74 metrics in 22.364114581s
2017-12-14T14:02:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:02:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:02:01Z D! Output [influxdb] wrote batch of 49 metrics in 761.926542ms
2017-12-14T14:02:16Z D! Output [influxdb] wrote batch of 49 metrics in 16.278410488s
2017-12-14T14:03:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:03:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:03:01Z D! Output [influxdb] wrote batch of 49 metrics in 1.011894163s
2017-12-14T14:03:16Z D! Output [influxdb] wrote batch of 49 metrics in 16.600676426s
2017-12-14T14:04:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:04:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:04:03Z D! Output [influxdb] wrote batch of 49 metrics in 2.796344288s
2017-12-14T14:04:16Z D! Output [influxdb] wrote batch of 49 metrics in 16.529340167s
2017-12-14T14:05:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:05:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:05:01Z D! Output [influxdb] wrote batch of 49 metrics in 776.990523ms
2017-12-14T14:05:09Z D! Output [influxdb] wrote batch of 49 metrics in 9.188031617s
2017-12-14T14:06:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:06:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:06:01Z D! Output [influxdb] wrote batch of 48 metrics in 832.753125ms
2017-12-14T14:06:11Z D! Output [influxdb] wrote batch of 48 metrics in 10.751219149s
2017-12-14T14:07:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:07:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:07:01Z D! Output [influxdb] wrote batch of 48 metrics in 944.869263ms
2017-12-14T14:07:10Z D! Output [influxdb] wrote batch of 48 metrics in 10.263134823s
2017-12-14T14:08:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:08:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:08:01Z D! Output [influxdb] wrote batch of 48 metrics in 1.463199081s
2017-12-14T14:08:10Z D! Output [influxdb] wrote batch of 48 metrics in 10.333792858s
2017-12-14T14:09:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:09:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:09:01Z D! Output [influxdb] wrote batch of 49 metrics in 1.09319638s
2017-12-14T14:09:12Z D! Output [influxdb] wrote batch of 49 metrics in 11.798826108s

Thanks in advance

Having more tests, loos like that the timeout increasing depending how much metric that telegraf receiving and how many output it has.

The Telegraf running http_listener is probably having trouble sending its output in a timely fashion. More details about this behavior is on this issue.

1 Like