How to determine which plugin failed when using multiple plugins of the same kind

I have a telegraf instance up and running with around 60 modbus input plugins. I’m receiving data every 15 seconds and as far as I’m concerned, everything works fine. However, I realized, that my telegraf instance reports an error on each pull:

...
Oct 01 10:45:15 test telegraf[63782]: 2021-10-01T08:45:15Z E! [inputs.modbus] Error in plugin: modbus: exception '4' (server device failure), function '131'
Oct 01 10:45:30 test telegraf[63782]: 2021-10-01T08:45:30Z E! [inputs.modbus] Error in plugin: modbus: exception '4' (server device failure), function '131'
Oct 01 10:45:45 test telegraf[63782]: 2021-10-01T08:45:45Z E! [inputs.modbus] Error in plugin: modbus: exception '4' (server device failure), function '131'

There seems to be a problem with one of the modbus plugins. Unfortunately, the error message doesn’t include a hint on which of the various modbus plugins fails. Is there a simple way to get this information or do I have to check each device in my config one by one?

Hi Brimstone,
Are you running Telegraf in debug mode with quiet mode set to false? Like so:

[agent]
  ## Default data collection interval for all inputs
  interval = "10s"
  ## Log at debug level.
  debug = true
  ## Log only error level messages.
  quiet = false

I am hoping this will provide us with further context to which input plugin is breaking.

I also suggest you look into adding a suffix / name override to each of your instances. That way we can check what metrics are progressing successfully and narrow down the results.

Hi Jay,
Thanks for your advise, I turned on debug mode. All the successful readings are now also logged. I added suffixes to some instances, but it seems like the measurement name, to which the suffix is added, is not part of the log.

...
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus] got holding@19026[2]: [199 253 119 238]
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus]   field PSum3 with offset 0 with len 4: [199 253 119 238] --> -129775.86
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus] got holding@19026[2]: [70 17 148 116]
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus]   field PSum3 with offset 0 with len 4: [70 17 148 116] --> 9317.113
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus] got holding@19026[2]: [199 252 171 220]
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus]   field PSum3 with offset 0 with len 4: [199 252 171 220] --> -129367.72
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus] got holding@19026[2]: [199 248 204 202]
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus]   field PSum3 with offset 0 with len 4: [199 248 204 202] --> -127385.58
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus] got holding@19026[2]: [68 86 224 251]
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus]   field PSum3 with offset 0 with len 4: [68 86 224 251] --> 859.5153
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z E! [inputs.modbus] Error in plugin: modbus: exception '4' (server device failure), function '131'
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus] got holding@19026[2]: [68 88 26 85]
Oct 01 13:31:00 test telegraf[114045]: 2021-10-01T11:31:00Z D! [inputs.modbus]   field PSum3 with offset 0 with len 4: [68 88 26 85] --> 864.41144
Oct 01 13:31:02 test telegraf[114045]: 2021-10-01T11:31:02Z D! [outputs.influxdb_v2] Wrote batch of 54 metrics in 8.434752ms
Oct 01 13:31:02 test telegraf[114045]: 2021-10-01T11:31:02Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

I can see the altered _measurement name in the Influx UI, but they are not displayed in the logs. Is there another way to uniquely identify the plugins in the logs?

Hi Brimstone,
I chatted with our Telegraf team and yes there is!
If you use the plugin alias setting, log messages that are usually like [outputs.file] become [outputs.file::my_alias]. See for details .

Hey Jay,
I added aliases and managed to identify the Plugin, that reported the error. Thank you!
I will start using aliases for all multi-plugin setups of telegraf in the future.

1 Like

No worries at all, glad it’s sorted.