Telegraf inputs.exec - no data to outputs

Telegraf
1

OS: Ubuntu 22.04.4
telegraf: Telegraf 1.30.0

I’m unable to find additional means of debugging this issue. I could really use some help with this!

I am able to run both inputs.exec commands shown in this the config below as user telegraf and get proper JSON output. These are two wrapper scripts shown below:

  • librespeed-wrapper (aka “librespeed speedtest”)
  • speedtest-wrapper (aka “speedtest.net speedtest”)

I am able to run this config as sudo -u telegraf telegraf --config /etc/telegraf/telegraf.conf --input-filter exec and see expected output on both outputs.file and the udp based outputs.influxdb (re: Troubleshooting Telegraf: Capture output)

However, when I run this under systemd systemctl start telegraf I see only the metrics from the “speedtest.net speedtest” (see reference/comment in telegraf.conf below).

So it’s the “librespeed speedtest” that is failing when I run this under systemd…

However, when I copy the JSON blob to a file with this one modification: commands = ["cat /tmp/f.json"], and run again under systemd I see the expected data in both outputs.file and the udp based outputs.influxdb.

I know. Crazy but true. And it’s been happening all day. I crafted this reproduction case in hopes that one or more awesome people here can help me figure this out!

/etc/telegraf/telegraf.conf

[global_tags]
[agent]
    interval = "10s"
    debug = true
    hostname = "proxy01"
[[outputs.influxdb]]
    urls = ["https://influxdb:8086"]
    username = "influx_telegraf"
    password = "hello!"
    insecure_skip_verify = true
    database = "telegraf"
[[outputs.file]]
  files = ["stdout"]
[[outputs.influxdb]]
  urls = ["udp://localhost:8089"]
##
## librespeed speedtest - https://github.com/librespeed/speedtest-cli
##
[[inputs.exec]]
    interval = "2m"
    commands = ["/usr/local/bin/librespeed-wrapper"]
    # commands = ["cat /tmp/f.json"]
    timeout = "900s"
    data_format = "json"
    name_suffix = "_speedtest_librespeed_test"
##
## speedtest.net speedtest - https://github.com/sivel/speedtest-cli/
##
[[inputs.exec]]
    interval = "4m"
    commands = ["/usr/local/bin/speedtest-wrapper"]
    timeout = "900s"
    data_format = "json"
    name_suffix = "_speedtest_test"

/usr/local/bin/librespeed-wrapper

#!/bin/bash

export http_proxy="http://localhost:8008"

# Use of jq simply extracts first array element. Telegraf also does this:
#  https://github.com/influxdata/telegraf/tree/master/plugins/parsers/json#arrays
#
# In either case, the results from the test case I'm describing is the same.
#
# /usr/local/bin/librespeed-cli --json | jq -c .[0]

/usr/local/bin/librespeed-cli --json
exit 0

sudo -u telegraf /usr/local/bin/librespeed-wrapper

[{"timestamp":"2024-03-15T05:24:18.735745485-07:00","server":{"name":"Los Angeles, United States (1) (Clouvider)","url":"http://la.speedtest.clouvider.net/backend"},"client":{"ip":"","hostname":"","city":"","region":"","country    ":"","loc":"","org":"","postal":"","timezone":""},"bytes_sent":253820928,"bytes_received":148403072,"ping":21,"jitter":0.95,"upload":130.15,"download":76.09,"share":""}]

/usr/local/bin/speedtest-wrapper

#!/bin/bash

export http_proxy="http://localhost:8008"
/bin/speedtest-cli --json
exit 0

sudo -u telegraf /usr/local/bin/speedtest-wrapper

{"download": 80339263.36987075, "upload": 13508783.165418645, "ping": 54.425, "server": {"url": "http://dal.speedtest.clouvider.net:8080/speedtest/upload.php", "lat": "32.7766", "lon": "-96.7969", "name": "Dallas, TX", "country": "United States", "cc": "US", "sponsor": "Clouvider Ltd", "id": "57114", "host": "dal.speedtest.clouvider.net:8080", "d": 8.013596789978571, "latency": 54.425}, "timestamp": "2024-03-15T13:36:01.273319Z", "bytes_sent": 22544384, "bytes_received": 100697982, "share": null, "client": {"ip": "198.55.124.115", "lat": "32.8137", "lon": "-96.8704", "isp": "QuadraNet", "isprating": "3.7", "rating": "0", "ispdlavg": "0", "ispulavg": "0", "loggedin": "0", "country": "US"}}

sudo -u telegraf telegraf --config /etc/telegraf/telegraf.conf --input-filter exec

2024-03-15T12:25:52Z I! Starting Telegraf 1.25.1
2024-03-15T12:25:52Z I! Available plugins: 228 inputs, 9 aggregators, 26 processors, 21 parsers, 57 outputs, 2 secret-stores
2024-03-15T12:25:52Z I! Loaded inputs: exec (2x)
2024-03-15T12:25:52Z I! Loaded aggregators:
2024-03-15T12:25:52Z I! Loaded processors:
2024-03-15T12:25:52Z I! Loaded secretstores:
2024-03-15T12:25:52Z I! Loaded outputs: file influxdb (2x)
2024-03-15T12:25:52Z I! Tags enabled: host=proxy01
2024-03-15T12:25:52Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"proxy01", Flush Interval:10s
2024-03-15T12:25:52Z D! [agent] Initializing plugins
2024-03-15T12:25:52Z D! [agent] Connecting outputs
2024-03-15T12:25:52Z D! [agent] Attempting connection to [outputs.file]
2024-03-15T12:25:52Z D! [agent] Successfully connected to outputs.file
2024-03-15T12:25:52Z D! [agent] Attempting connection to [outputs.influxdb]
2024-03-15T12:25:52Z W! [outputs.influxdb] When writing to [https://influxdb.sfio.win:8086]: database "telegraf" creation failed: 403 Forbidden
2024-03-15T12:25:52Z D! [agent] Successfully connected to outputs.influxdb
2024-03-15T12:25:52Z D! [agent] Attempting connection to [outputs.influxdb]
2024-03-15T12:25:52Z D! [agent] Successfully connected to outputs.influxdb
2024-03-15T12:25:52Z D! [agent] Starting service inputs
2024-03-15T12:26:02Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:26:02Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
[ ... ]
2024-03-15T12:26:52Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:27:02Z D! [outputs.influxdb] Wrote batch of 1 metrics in 344.373µs
2024-03-15T12:27:02Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
exec_speedtest_librespeed_test,host=proxy01 ping=23,jitter=0.21,download=77.44,bytes_sent=277151744,bytes_received=151040888,upload=142.11 1710505616000000000
2024-03-15T12:27:02Z D! [outputs.file] Wrote batch of 1 metrics in 53.699µs
2024-03-15T12:27:02Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:27:02Z D! [outputs.influxdb] Wrote batch of 1 metrics in 5.952032ms
2024-03-15T12:27:02Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
[ ... ]
2024-03-15T12:28:22Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
exec_speedtest_test,host=proxy01 ping=110.394,download=72811536.7860282,bytes_sent=21495808,server_latency=110.394,upload=12957249.561208352,bytes_received=91561442,server_d=459.8101514502958 1710505708000000000
2024-03-15T12:28:32Z D! [outputs.file] Wrote batch of 1 metrics in 95.955µs
2024-03-15T12:28:32Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:28:32Z D! [outputs.influxdb] Wrote batch of 1 metrics in 112.707µs
2024-03-15T12:28:32Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:28:32Z D! [outputs.influxdb] Wrote batch of 1 metrics in 5.661785ms
2024-03-15T12:28:32Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
[ ... ]
2024-03-15T12:28:52Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:29:02Z D! [outputs.influxdb] Wrote batch of 1 metrics in 139.51µs
2024-03-15T12:29:02Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
exec_speedtest_librespeed_test,host=proxy01 bytes_received=80303808,ping=160,download=41.18,bytes_sent=257130496,jitter=54.68,upload=131.85 1710505737000000000
2024-03-15T12:29:02Z D! [outputs.file] Wrote batch of 1 metrics in 41.284µs
2024-03-15T12:29:02Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:29:02Z D! [outputs.influxdb] Wrote batch of 1 metrics in 5.26551ms
2024-03-15T12:29:02Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:29:12Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
[ ... ]
2024-03-15T12:30:52Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:30:52Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:30:52Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
exec_speedtest_librespeed_test,host=proxy01 upload=131.27,download=17.88,ping=33,jitter=3.01,bytes_received=34874240,bytes_sent=256016384 1710505857000000000
2024-03-15T12:31:02Z D! [outputs.file] Wrote batch of 1 metrics in 57.689µs
2024-03-15T12:31:02Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:31:02Z D! [outputs.influxdb] Wrote batch of 1 metrics in 89.106µs
2024-03-15T12:31:02Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:31:02Z D! [outputs.influxdb] Wrote batch of 1 metrics in 5.278421ms
2024-03-15T12:31:02Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
[ ... ]
2024-03-15T12:32:22Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
exec_speedtest_test,host=proxy01 upload=9078047.978239167,bytes_received=42219555,server_d=368.2138085867896,bytes_sent=16252928,download=33657241.02449668,ping=82.934,server_latency=82.934 1710505948000000000
2024-03-15T12:32:32Z D! [outputs.file] Wrote batch of 1 metrics in 55.694µs
2024-03-15T12:32:32Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:32:32Z D! [outputs.influxdb] Wrote batch of 1 metrics in 573.187µs
2024-03-15T12:32:32Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:32:32Z D! [outputs.influxdb] Wrote batch of 1 metrics in 5.287226ms
2024-03-15T12:32:32Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
[ ... ]
2024-03-15T12:32:52Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:33:02Z D! [outputs.influxdb] Wrote batch of 1 metrics in 163.942µs
2024-03-15T12:33:02Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
exec_speedtest_librespeed_test,host=proxy01 bytes_sent=275775488,jitter=52.21,bytes_received=43003968,ping=109,download=22.05,upload=141.4 1710505977000000000
2024-03-15T12:33:02Z D! [outputs.file] Wrote batch of 1 metrics in 90.344µs
2024-03-15T12:33:02Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:33:02Z D! [outputs.influxdb] Wrote batch of 1 metrics in 7.152644ms
2024-03-15T12:33:02Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
[ ... ]
2024-03-15T12:33:22Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
^C2024-03-15T12:33:31Z D! [agent] Stopping service inputs
2024-03-15T12:33:31Z D! [agent] Input channel closed
2024-03-15T12:33:31Z I! [agent] Hang on, flushing any cached metrics before shutdown
2024-03-15T12:33:31Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:33:31Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:33:31Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2024-03-15T12:33:31Z I! [agent] Stopping running outputs
2

@me2017 first of all one question, why don’t you use the internet_speed input plugin which is based on speedtest.net instead of your speedtest-wrapper?

For why you don’t see data: What is the raw JSON output of your wrappers if you call them?

3

Hello @srebhan! Thanks for the suggestion. I had a look at inputs.internet_speed but unfortunately (and surprisingly) it doesn’t include proxy support. That is, I’m unable to define a specific proxy for this input. It’s critical that I run these metrics from localhost:8008. This is why I’m using wrappers with the inputs I’ve shown above. It allows me to define the proxy like so: export http_proxy="http://localhost:8008"

I’m going to edit my original post to reflect the JSON output of the wrappers and also include some general logging from telegraf.

4

Are you sure that exporting the proxy in Telegraf’s enviroment doesn’t work?

5

@me2017 if it doesn’t work with exporting the variable, please open a feature request for proxy support!

6

I’m sure it would work with the OS defined proxy but for this particular use case I need to define a specific proxy for this input.

7

The solution for this particular issue is embarrassingly simple but difficult to find. After redirecting stderr/stdout in the wrapper I found librespeed-cli was making an initial request to an https endpoint and failing. I then realized there was no definition for https_proxy in the wrapper script. Therefore:

export http_proxy="http://localhost:8008"
export https_proxy="http://localhost:8008"

/usr/local/bin/librespeed-cli --json
exit 0
8

@me2017 as I said, if I were you I would create a feature request for the plugin… :wink: