Why writing to inputs.http_listener is much slower then wirting to influxdb?

1

There is two influxdb output in a telegraf config, one is going to an actual influxdb, one is going to a telegraf with inputs.http_listener (both going over https, content_encoding = "gzip", telegraf 1.4.4 => telegraf 1.5 )
Every second write in the following logs is going to a telegraf (inputs.http_listener) which is exactly where influxdb located.
Do anyone have any idea why is it so much slower ?

2017-12-14T14:01:00Z D! Output [influxdb] buffer fullness: 74 / 10000 metrics.
2017-12-14T14:01:00Z D! Output [influxdb] buffer fullness: 74 / 10000 metrics.
2017-12-14T14:01:01Z D! Output [influxdb] wrote batch of 74 metrics in 1.024278158s
2017-12-14T14:01:22Z D! Output [influxdb] wrote batch of 74 metrics in 22.364114581s
2017-12-14T14:02:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:02:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:02:01Z D! Output [influxdb] wrote batch of 49 metrics in 761.926542ms
2017-12-14T14:02:16Z D! Output [influxdb] wrote batch of 49 metrics in 16.278410488s
2017-12-14T14:03:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:03:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:03:01Z D! Output [influxdb] wrote batch of 49 metrics in 1.011894163s
2017-12-14T14:03:16Z D! Output [influxdb] wrote batch of 49 metrics in 16.600676426s
2017-12-14T14:04:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:04:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:04:03Z D! Output [influxdb] wrote batch of 49 metrics in 2.796344288s
2017-12-14T14:04:16Z D! Output [influxdb] wrote batch of 49 metrics in 16.529340167s
2017-12-14T14:05:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:05:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:05:01Z D! Output [influxdb] wrote batch of 49 metrics in 776.990523ms
2017-12-14T14:05:09Z D! Output [influxdb] wrote batch of 49 metrics in 9.188031617s
2017-12-14T14:06:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:06:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:06:01Z D! Output [influxdb] wrote batch of 48 metrics in 832.753125ms
2017-12-14T14:06:11Z D! Output [influxdb] wrote batch of 48 metrics in 10.751219149s
2017-12-14T14:07:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:07:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:07:01Z D! Output [influxdb] wrote batch of 48 metrics in 944.869263ms
2017-12-14T14:07:10Z D! Output [influxdb] wrote batch of 48 metrics in 10.263134823s
2017-12-14T14:08:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:08:00Z D! Output [influxdb] buffer fullness: 48 / 10000 metrics.
2017-12-14T14:08:01Z D! Output [influxdb] wrote batch of 48 metrics in 1.463199081s
2017-12-14T14:08:10Z D! Output [influxdb] wrote batch of 48 metrics in 10.333792858s
2017-12-14T14:09:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:09:00Z D! Output [influxdb] buffer fullness: 49 / 10000 metrics.
2017-12-14T14:09:01Z D! Output [influxdb] wrote batch of 49 metrics in 1.09319638s
2017-12-14T14:09:12Z D! Output [influxdb] wrote batch of 49 metrics in 11.798826108s

Thanks in advance

2

Having more tests, loos like that the timeout increasing depending how much metric that telegraf receiving and how many output it has.

3

The Telegraf running http_listener is probably having trouble sending its output in a timely fashion. More details about this behavior is on this issue.

1 Like