Telegraf protobuf for http_listener_v2 throws Number of configs: 0

Telegraf
1

I am trying to setup telegraf to listen for metrics from Apache Airflow v2.7.1
Using the input module http_listener_v2. I am getting one log message when I enable debugging which is “Number of Configs: 0”

Here is my full telegraf configuration

[agent]
  interval = "5s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "5s"
  flush_jitter = "0s"
  precision = "0s"
  debug = true
  quiet = false
  logtarget = "file"
  logfile = ""
  omit_hostname = false


[[inputs.http_listener_v2]]
  service_address = ":8080"
  paths = ["/v1/metrics"]
  methods = ["POST", "PUT"]
  data_source = "body"
  read_timeout = "20s"
  write_timeout = "20s"
  data_format = "xpath_protobuf"
  xpath_protobuf_import_paths = [".", "opentelemetry/proto/metrics/v1"]
  xpath_protobuf_file = "opentelemetry/proto/metrics/v1/metrics.proto"
  xpath_protobuf_type = "opentelemetry.proto.metrics.v1.MetricsData"


[[outputs.file]]
  files = ["stdout"]

I downloaded the opentelemetry proto files from https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto
and moved it to the same directory and I am sure telegraf can read it because it was giving parsing error before I do that.

I am running both telegraf and apache airflow on localhost without using docker
As far as I can tell, airflow is producing the metrics in the exact format as the .proto files specify.

here is a metric produced by airflow represented in json format for readability

{
  "resource_metrics": [
    {
      "resource": {
        "attributes": {
          "service.name": "Airflow"
        },
        "schema_url": ""
      },
      "scope_metrics": [
        {
          "scope": {
            "name": "airflow.metrics.otel_logger",
            "version": null,
            "schema_url": ""
          },
          "metrics": [
            {
              "name": "airflow-otel.serde.load_serializers",
              "description": "",
              "unit": "",
              "data": {
                "data_points": [
                  {
                    "attributes": {},
                    "start_time_unix_nano": 0,
                    "time_unix_nano": 1708089023039230000,
                    "value": 0.005870708031579852
                  }
                ]
              }
            }
          ],
          "schema_url": ""
        }
      ],
      "schema_url": ""
    }
  ]
}

I am not sure how to debug this further, any clues?

2

Hi,

Is that your entire config? If so you need to tell Telegraf what fields and tags to create based on your data as well. Telegraf does not know how to generate a metric based on your data, so you need to tell it, hey I want a metric called foo with these values used as tags and fields.

Also when using the xpath parser I like to use the xpath_print_document option to ensure I know what data I am getting.

3

Here is the output after adding the xpath_print_document = true option

2024-02-17T11:00:15Z D! [parsers.xpath_protobuf::http_listener_v2] XML document equivalent: "<?xml version=\"1.0\"?><resource_metrics><resource><attributes><value><string_value>Airflow</string_value></value><key>service.name</key></attributes></resource><scope_metrics><scope><name>airflow.metrics.otel_logger</name></scope><metrics><name>airflow-otel.serde.load_serializers</name><gauge><data_points><time_unix_nano>1708091636170202000</time_unix_nano><as_double>0.0015034580137580633</as_double></data_points></gauge></metrics></scope_metrics></resource_metrics>"
2024-02-17T11:00:15Z D! [parsers.xpath_protobuf::http_listener_v2] Number of configs: 0
2024-02-17T11:00:18Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics

Can you give a config example or reference to how to tell telegraf which fields and tags to create.

4

thanks to @jpowers hint, I figured out that being novice to telegraf I didn’t specify some needed parameters to xpath to tell it where to look for metrics.

I added another bit to the config file that seems to get things working. Here is the updated config in case someone comes by the same issue

[[inputs.http_listener_v2]]
  service_address = ":8080"
  paths = ["/v1/metrics"]
  methods = ["POST", "PUT"]
  data_source = "body"
  read_timeout = "30s"
  write_timeout = "30s"
  data_format = "xpath_protobuf"
  xpath_protobuf_import_paths = ["."]
  xpath_protobuf_file = "opentelemetry/proto/metrics/v1/metrics.proto"
  xpath_protobuf_type = "opentelemetry.proto.metrics.v1.MetricsData"
  xpath_print_document = true
  xpath_native_types = true

  [[inputs.http_listener_v2.xpath]]
    field_selection = "//resource_metrics/descendant::*"
    field_name_expansion = false
    timestamp = "//time_unix_nano"
    timestamp_format = "unix_ns"