Telegraf config for parsing Juniper UDP / Protobuf (dial-out) data

Telegraf
,
1

I have read through the docs, blogs and other forums and have to be close on my telegraf.conf file here. If someone has asked/answered this before please point me to it.

I have streaming telemetry data coming into a host from a Juniper virtual switch in UDP / protobuf format. I can capture the data using nc and decode it using protoc:

protoc --decode TelemetryStream port.proto -I /usr/include/protos -I /usr/include/google/protobuf < /tmp/data.gpb > /tmp/data.json
cat /tmp/data.json
system_id: "ST-Justin:10.55.60.125"
component_id: 0
sub_component_id: 0
sensor_name: "ROUTER-INF:/junos/system/linecard/interface/:/junos/system/linecard/interface/:PFE"
sequence_number: 103837
timestamp: 1719150965216
version_major: 1
version_minor: 1
enterprise {
  [juniperNetworks] {
    [port.jnpr_interface_ext] {
      interface_stats {
        if_name: "ge-0/0/0"
        init_time: 1718625895
        snmp_if_index: 527
        egress_queue_info {
          queue_number: 0
          packets: 0
          bytes: 0
          tail_drop_packets: 0
          rl_drop_packets: 0
          rl_drop_bytes: 0
          red_drop_packets: 0
          red_drop_bytes: 0
          avg_buffer_occupancy: 0
          cur_buffer_occupancy: 0
          peak_buffer_occupancy: 0
          allocated_buffer_size: 0
        }
<snip>

Here is my current telegraf.conf file:

# ===============================================================
[global_tags]

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  debug = true
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = true
# ===============================================================

[[inputs.socket_listener]]
  service_address = "udp://:2000"
  data_format = "xpath_protobuf"
  xpath_protobuf_file = "port.proto"
  xpath_protobuf_type = "TelemetryStream"
  
  xpath_print_document = true
  xpath_native_types = false 
  xpath_protobuf_import_paths = ["/usr/include/protos","/usr/include/google/protobuf/"]

The resulting XML doc that is dumped by debug is:

grep xml /var/log/syslog
Jun 26 05:22:30 Juniper-ST-Dial-Out telegraf[201640]: 2024-06-26T15:22:30Z D! [parsers.xpath_protobuf::file]  XML document equivalent: "<?xml version=\"1.0\"?><system_id>ST-Justin:10.55.60.125</system_id><component_id>0</component_id><version_major>1</version_major><version_minor>1</version_minor><sub_component_id>0</sub_component_id><sensor_name>ROUTER-INF:/junos/system/linecard/interface/:/junos/system/linecard/interface/:PFE</sensor_name><sequence_number>103837</sequence_number><timestamp>1719150965216</timestamp><enterprise></enterprise>"

For some reason, I am not seeing all the data in the enterprise part of the tree in that output like I do when I use nc and protoc to grab the data.

2

hmm looks like everything is missing in the enterprise section.

total guess: is it possible that the protobuf file you are using imports or uses other files to decode that part of the message? It could be that protoc imports files more easily?

3

I guess it’s possible. My protoc command is using the same directories so I would think both have access to the same list of proto files. Is there any way to test this theory? I’d have hoped that telegraf would just grab all the data and shove it into the <enterprise> block even if it couldn’t decode it and make sense of it instead of just silently dropping all that data.

4

I would start by opening the port.proto protobuf file and read through its import statements. If it is importing any other modules then where are those files located? Specifically is the enterprise definitions there or elsewhere?

I too thought it should be able to read it, but the fact that the entire section is missing makes me think something is missing to be able to decode it.

5

@jryburn could you please open an issue and provide the proto file(s) as well as a (redacted) data sample so we can reproduce the issue?

6

TelemetryStream is a reference to this proto file which is where the current output is being decoded. The rest of the message should be in the port.proto file which is here.

All of these files are in the directory specified.

 xpath_protobuf_import_paths = ["/usr/include/protos","/usr/include/google/protobuf/"]
7

Done. Telegraf config for parsing Juniper UDP / Protobuf (dial-out) data · Issue #15571 · influxdata/telegraf · GitHub