Missing Prometheus metrics in InfluxDB

Telegraf
, , ,
1

Hello, I’m facing an issue with my Docker setup that includes a Blackbox Exporter, which exports metrics to Prometheus. From there, the data is supposed to be written to InfluxDB using Telegraf. However, I’ve noticed that the probe_success metric is nowhere to be found in InfluxDB.

To be honest, I can find many metrics in the prometheus measurement but not any of those that are being scraped and that I can see in the Prometheus web interface.

I can find metrics like prometheus_http_requests_total but I can’t find that in the Prometheus web interface…

If any of you have encountered a similar issue or have some insights to share, I’d appreciate your help.

2

Maybe someone with specific Blackbox Exporter experience will chime in. However, from the Telegraf side here are some thoughts:

  1. Try using the [[outputs.file]] output plugin to see what metrics are getting collected and ensure it is somehow not getting filtered by your config. The file output will print the metrics to stdout.
  2. If you navigate to the prometheus endpoint itself do you see the metric? What does it look like? Is it a string?
  3. Logs and configs are really nice to see to ensure we are understanding the flow of your config. It also good to include the entire config, so we can see if any processors or other filtering is involved as well.
3

Hey, thanks for your response!

I just checked if the metrics displayed at http://localhost:9090/metrics match those in the Telegraf output, and they do. That’s good news on one hand, but on the other hand, I’m now puzzled as to why the metrics I see through the graphical web interface at http://localhost:9090 are not being displayed there. Do you have any idea about this?

To be honest, I don’t need any of the metrics I can see at the /metrics endpoint, I only need the probe_success ones…

4

My prometheus.yml looks like this:

---
global:
  scrape_interval: 15s
  evaluation_interval: 1m

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: "ha-control-targets"
    metrics_path: /probe
    params:
      module: [http_2xx]
    file_sd_configs:
      - files:
          - /etc/prometheus/targets/ha-control-targets.yml
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

My input plugin in Telegraf looks like this:

[[inputs.prometheus]]
  urls = ["http://localhost:9090/metrics"]
  metric_version = 2
5

Okay, I think I got it… Wow, sometimes I am amazed by my own stupidity…

As you can see in the in the [[inputs.prometheus]] configuration, I’m scraping the Prometheus metrics endpoint that offers metrics about Prometheus… But I needed to scrape the endpoint of the blackbox exporter ( http://localhost:9115/probe?...)… :person_facepalming:

But still, is it possible to keep the urls array small and organized? I got many targets and many modules with whom I’m scraping and the list would become really really long, if I entered them like this:

urls = [
    "http://blackboxexporter:9115/probe?target=apple.com&module=http_2xx",
    "http://blackboxexporter:9115/probe?target=google.com&module=http_2xx",
    "http://blackboxexporter:9115/probe?...",
    ...
]
6

Glad you got this working better!

As far as the URLs, I briefly looked at the config options. Would the static_configs option provide you with a list of targets that you could set in your config and then you get all the results on your metrics endpoint?