Process binary input data in Telegraf?

Is there a way to process binary data (raw bytes) in Telegraf in an Input or Processor plugin without having to write a custom plugin for it?

Background: I work a lot with data coming from low-level interfaces or busses (e.g. UART, CAN bus, SPI …) which do not provide pretty text-based strings, but just spit out naked binary bytes.

The “Input data formats” described in the Telegraf documentation seem to be human-readable string-based formats only?

Also with the “value” input data format I miss something like “bytes” or “raw”.

I tried the following test setup:

I generated a simple config file for telegraf and configured an UDP port in the socket_listener plugin.

telegraf --input-filter socket_listener --output-filter file --section-filter agent:global_tags:outputs:inputs config > binary.conf

Then i ran Telegraf with this config:

telegraf --debug --config binary.conf

Then i used the Packet Sender open source utility to send binary data to this UDP port.

But with raw non-ascii bytes, either I get garbage in the output or an error message from the parser:

2021-02-13T22:51:38Z E! [inputs.socket_listener] Unable to parse incoming packet: strconv.Atoi: parsing "\x01\x02\x03": invalid syntax

Currently telegraf doesn’t have a data format that lets you specify a binary format like you describe.

One option to work around not having a binary format would be to use telegraf’s execd input with a custom script. This wouldn’t be a custom plugin and wouldn’t require you to build telegraf. The custom script would receive your data from uart, can, spi, etc, and decode it, then output it in one of telegraf’s existing formats, like csv or line protocol. (see telegraf/DATA_FORMATS_INPUT.md at master · influxdata/telegraf · GitHub) You could use something like python or even powershell to do it. Since execd would run the script, you wouldn’t need to use packetsender and socket_listener to get the data into telegraf. I would recommend trying this approach.

If you’re interested in helping to add a binary data format to telegraf, here is some background:

The difficulty with accepting binary data like this is that to cover everyone’s needs, a telegraf data_format would need a format specification that is expressive enough to describe pretty much any arbitrary format. I would expect it to handle numbers of various sizes and types (like signed/unsigned int and floating point), strings in various encodings, arrays of fixed and variable length, padding, etc.

There are a few open issues related to binary data format:

We also would prefer to use an existing golang library instead of writing something custom for telegraf. The packet library mentioned in 8434 is promising but it’s written for node.js, not golang. Another related library is kaitai.io. It has an expressive format spec but requires the format to be known at compile time.

If we can find a golang library that meets telegraf’s needs, we would be interested in adding a binary data_format plugin.