Telegraf - Default value instead of connection time-outs

Telegraf
, ,
1

Hello!

I was wondering if Telegraf supports a way of filling a set of predefined metric values even when there’s a connection time-out at the remote end-device.

Once telegraf can’t connect, it basically “skips” the input end-point device and leaves influxdb empty.
Thus, querying this data, for example through Grafana, gives me an empty row/no data.
Instead, I’d like for telegraf to fill in predefined values which could later be used to identify as connection errors at the front-end.

I’m using input.http and input.exec (this last one with ‘curl’) to send out get-requests to parse JSON-data in return.

The plugin that comes close is Defaults Processor, but sadly it doesn’t seem to handle connection time-outs.

My only solution was to use input.execd in order to trigger a custom-made console app which would also run a get request, but in turn would handle a connection time-out by returning JSON data with predefined error values to use as troubleshooting.

Any ideas or suggestions are much appreciated.

2

I think your solution is a good one, the other thing I can think of is watching the Telegraf logs for errors.

3

That’s also brilliant! At the moment I’ve built a webAPI instead of a console app, however I’d rather free up my API resources for other stuff instead of hogging it with a bunch of slow servers.

If I were to implement this using the telegraf logfile, I assume this would be inputs.tail? Any idea on how to configure this in order to read the telegraf log format? Should I use grok format for this and somehow “patternize” its parsing algorithm? (If I understood grok correctly)?

This is an example of my logfile with a failing remote server (not necessarily a time-out):

2021-08-10T14:20:03Z D! [agent] Successfully connected to outputs.influxdb_v2
2021-08-10T14:20:03Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2021-08-10T14:20:03Z D! [agent] Successfully connected to outputs.influxdb_v2
2021-08-10T14:20:03Z D! [agent] Starting service inputs
2021-08-10T14:20:13Z E! [inputs.http] Error in plugin: [url=http://localhost:8080/api/v1/parse?ipa=10.10.10.2&port=50071]: Get "http://localhost:8080/api/v1/parse?ipa=10.10.10.2&port=50071": read tcp [::1]:55827->[::1]:8080: wsarecv: An existing connection was forcibly closed by the remote host.
2021-08-10T14:20:13Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2021-08-10T14:20:13Z D! [outputs.influxdb_v2] Wrote batch of 4 metrics in 10.6177ms
2021-08-10T14:20:13Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics