Completely stuck trying to get data imported to influxdb and displayed on grafana


I’ll start off by describing what I want to achieve, I want to write a python script which calls the API of, then that data must be input into influxdb using telegraf and then displayed on grafana. I got the idea from Mining | LiteFoote , however I am now at a point where I am completely stuck and even after hours of googling I do not know how to continue.

I have influx,grafana and telegraf installed on an ec2 AWS instance. I set up influx, grafana and telegraf, will put my telegraf config below. I have my config stored at the default location /etc/telegraf/telegraf.conf (deleted what was inside the file and replaced it with my own text).
I tried testing the config like they did on the litefoote website however I get permission errors.
I tried running telegraf --config /etc/telegraf/telegraf.conf --input-filter exec --debug or with --test , and I get the error [inputs.exec] Error in plugin: exec: fork/exec /home/ubuntu/ permission denied for command ‘/home/ubuntu/’:
So I tried googling and thus I tried running
sudo chown telegraf:telegraf /home/ubuntu/ and chown telegraf
, but none worked so I am not sure what to attempt now.
Here is a screenshot of the litefoot website

I tried ignoring the permission error and just went to setting up grafana, but I do not get any data displayed(I may have set up the from field wrong as I think my setup is slightly different to the litefoot website, but I am not sure)

# Configuration for telegraf agent
  ## Default data collection interval for all inputs
  interval = "10s"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  ## Telegraf will send metrics to outputs in batches of at most
  ## metric_batch_size metrics.
  ## This controls the size of writes that Telegraf sends to output plugins.
  metric_batch_size = 1000

  ## For failed writes, telegraf will cache metric_buffer_limit metrics for each
  ## output, and will flush this buffer on a successful write. Oldest metrics
  ## are dropped first when this buffer fills.
  ## This buffer only fills when writes fail to output plugin(s).
  metric_buffer_limit = 10000

  ## Collection jitter is used to jitter the collection by a random amount.
  ## Each plugin will sleep for a random time within jitter before collecting.
  ## This can be used to avoid many plugins querying things like sysfs at the
  ## same time, which can have a measurable effect on the system.
  collection_jitter = "0s"

  ## Default flushing interval for all outputs. Maximum flush_interval will be
  ## flush_interval + flush_jitter
  flush_interval = "10s"
  ## Jitter the flush interval by a random amount. This is primarily to avoid
  ## large write spikes for users running a large number of telegraf instances.
  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "0s"

  ## By default or when set to "0s", precision will be set to the same
  ## timestamp order as the collection interval, with the maximum being 1s.
  ##   ie, when interval = "10s", precision will be "1s"
  ##       when interval = "250ms", precision will be "1ms"
  ## Precision will NOT be used for service inputs. It is up to each individual
  ## service input to set the timestamp at the appropriate precision.
  ## Valid time units are "ns", "us" (or "µs"), "ms", "s".
  precision = ""

  ## Logging configuration:
  ## Run telegraf with debug log messages.
  debug = false
  ## Run telegraf in quiet mode (error log messages only).
  quiet = false
  ## Specify the log file name. The empty string means to log to stderr.
  logfile = ""

  ## Override default hostname, if empty use os.Hostname()
  hostname = ""
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = false
  ## The URLs of the InfluxDB cluster nodes.
  ## Multiple URLs can be specified for a single cluster, only ONE of the
  ## urls will be written to each interval.
  ## urls exp:
  urls = [""]

  ## Token for authentication.
  token = "$INFLUX_TOKEN"

  ## Organization is the name of the organization you wish to write to; must exist.
  organization = "altus"

  ## Destination bucket to write into.
  bucket = "altus"
  ## Whether to report per-cpu stats or not
  percpu = true
  ## Whether to report total system cpu stats or not
  totalcpu = true
  ## If true, collect raw CPU time metrics.
  collect_cpu_time = false
  ## If true, compute and report the sum of all non-idle CPU states.
  report_active = false
  ## By default stats will be gathered for all mount points.
  ## Set mount_points will restrict the stats to only the specified mount points.
  # mount_points = ["/"]
  ## Ignore mount points by filesystem type.
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "overlay", "aufs", "squashfs"]

commands = ["/home/ubuntu/"]
data_format = "json"
interval = "120s"
name_suffix = "-ethermine"

Home directories are only permissioned for the user that owns them. There are ways around that, but it’s simply not worth it because it breaks the design.

Solution is to move your script outside your ubuntu user’s home directory to somewhere that is shared by everyone. Eg /usr/local/bin/

Disclaimer: if you’re not sure on any of this, make backups, and get a Linux admin to advise you. I am not responsible if you break your server.

sudo mv -v yourScriptNameHere /usr/local/bin/
sudo chmod 755 /usr/local/bin/yourScriptNameHere

Run the script manually and confirm it actually works from the new location.

Then update the path in the telegraf config. And restart the telegraf service.

I’ll give that a shot and report back on how it goes, by running the script manually do you mean something like python /usr/local/bin/ ?

Yep. Just to verify that works at least before moving on to getting telegraf to run it.

Okay made some progress, moved the file successfully and ran it manually and it works. Updated the config file and restarted telegraf, tested it with telegraf --config /etc/telegraf/telegraf.conf --input-filter exec --debug ,but I get a new error now: [inputs.exec] Error in plugin: exec: fork/exec /usr/local/bin/ exec format error for command ‘/usr/local/bin/’:
Here is the inputs exec part of my config…not sure what part of the format is wrong?

Try running just the above , even as your ubuntu user first. Does that provide valid output? Are there any extra lines, formatting, python warnings buried in the Json output? If yes, that’ll be what telegraf is unhappy about

Sorry I am completely new to linux, to run it is a user do you mean say python3 -u ubuntu /usr/local/bin/ ?

Nws. From a terminal window, just copy and paste the command I quoted. The point is to run and test it exactly as you have configured it to run in the telegraf config which is just that one script , nothing else.

If it fails as you (ubuntu user) it’s not going to work as telegraf .

Okay so is this an issue with my python script or what? Because it runs and outputs fine if I run the python script on my windows machine

At a guess, the default python on your linux server is probably python2, and your script is probably formatted with windows line endings “\r\n” , Linux uses just linefeed “\n”

To fix the second, you need to convert the script . Backup your script first! Then try running

dos2unix /usr/local/bin/

If this doesn’t work / dos2unix hasn’t been installed , there are other ways to solve it. If you have notepad++ installed, it has a way to convert line endings buried someone under the edit menu.

To fix the first add “python3” to the front of the script like to

python3 /usr/local/bin/

Running python3 /usr/local/bin/ seems to have fixed both issues, it now runs just like it would on my windows machine. Now to figure out why telegraf is unhappy

1 Like

Put that command into where you have your script in the telegraf config, and it should just work.

Do you mean like:
commands = [“python3 /usr/local/bin/”]

1 Like

Did not have time to work on it yesterday, but I got it working now once I added the python 3 to the config. When running telegraf --config /etc/telegraf/telegraf.conf --input-filter exec --test I get the same output as what I get in python on my windows machine. Now I want to see the data in influxdb, but I don’t see where to select the measurement?
trying to search for my_metric in influxdb I don’t see it, unless I am going about this wrong I’m not sure if it is actually working :sweat_smile:

Nice job. Now take a look at the influxdb log. Did it accept the input record, or throw some error?

Not relevant in your setup with executing python script inside telegraf

On the python side, you can test for success/failure by looking at the http return code. 200 is success and from memory everything else is a failure


You might want to run influxdb in debug mode

If the log says it was successfully accepted, make sure in the explorer view your time range you’re searching on is wide enough to cover the record’s timestamp (I assume your letting influxdb handle that as I don’t see a timestamp in your screenshot)

Edit : removed bit about testing http return code. Not relevant when running script inside telegraf

Not sure how to do any of what you just described…will have google when I get some time. Have not used python, linux or any of influxes programs before so this is all pretty new…a lot of learning to do

Okay I checked the influx logs with sudo journalctl -u influxdb.service and the majority of the lines seem to be an authorization error. On the python side of things I checked the response code of the API URL I am querying and got response code 200 back so that seems all good.

1 Like

Sorry i confused you, the bit about testing http return code isn’t relevant with the setup you’re running now.

The debug link I posted takes you to the docs on how to reconfigure influxdb to run with debugging enabled. Definitely worth doing or you’ll be somewhat blind as to what’s happening on that end

Okay, I configured it like so: influxd --log-level=debug, not sure if the above screenshot helps at all, but I assume this debug mode will store a log file somewhere now?

Take a look at the influxdb2 output config in telegraf, is a valid token configured ?