telegraf: dataformat splunkmetric is skipping non-numeric metrics

Telegraf
1

telegraf: dataformat splunkmetric is skipping non-numeric metrics

Hi all,

We are streaming Cisco MDT to telegraf and forwarding those data to Splunk HEC endoint

telegraf.conf

[[inputs.cisco_telemetry_mdt]]
 transport = "grpc"
 service_address = ":57000"
 max_msg_size = 4000000

[[outputs.http]]
  url = "http://spunkhec:8188/services/collector/raw"
  data_format = "splunkmetric"
[outputs.http.headers]
  Content-Type = "application/json"
  Authorization = "Splunk TOKENGOESHERE"

We are able to use moste of the metrics, but not all. E.g. we are missing “oper_status”, “ipv4”,“negotiated_port_speed”. We wrote splunkmetric and json data format to files and compared those.

data_format=json generaged event

{
    "fields": {
        --> very log list <---
        "admin_status": "if-state-up",
        "auto_downstream_bandwidth": 0,
        "auto_upstream_bandwidth": 0,
        "bia_address": "50:f7:22:0c:6c:81",
        "description": "*** testdescription ***",
        "ether_state/auto_negotiate": false,
        "ether_state/enable_flow_control": false,
        "ether_state/media_type": "ether-media-type-sfp-plus",
        "ether_state/negotiated_duplex_mode": "unknown-duplex",
        "ether_state/negotiated_port_speed": "speed-auto",
        --> very log list <---

    },
    "name": "Cisco-IOS-XE-interfaces-oper:interfaces/interface",
    "tags": {
        "host": "f72966db2605",
        "index": "cisco_mdt_test",
        "name": "TenGigabitEthernet0/0/1",
        "path": "Cisco-IOS-XE-interfaces-oper:interfaces/interface",
        "source": "oursource",
        "subscription": "103",
        "user": "${USER}"
    },
    "timestamp": 1688409749
}

data_format=splunkmetric generated event:

{
    "_value": 0,
    "metric_name": "Cisco-IOS-XE-interfaces-oper:interfaces/interface.v6_protocol_stats/out_forwarded_octets",
    "name": "ourname",
    "path": "Cisco-IOS-XE-interfaces-oper:interfaces/interface",
    "subscription": "103",
    "time": 1688473926.2150002,
    "user": "${USER}"
}{
    "_value": 0,
    "metric_name": "Cisco-IOS-XE-interfaces-oper:interfaces/interface.v6_protocol_stats/out_discarded_pkts",
    "name": "ourname",
    "path": "Cisco-IOS-XE-interfaces-oper:interfaces/interface",
    "subscription": "103",
    "time": 1688473926.2150002,
    "user": "${USER}"
}

For us it looks like non numeric metrics are skipped by splunkmetric data format. I guess it would an idea to add those non-numeric values as dimensions to the metrics. This would be the same like fields “name”, “path”, “subscription”.

best regards,

Andreas

2

Hey @schose,

For us it looks like non numeric metrics are skipped by splunkmetric data format

Isn’t this exactly what the documentation says? It even provides a solution. :wink:

TL;DR:Splunk only supports numeric values so you need to convert the non-numeric values to a number using e.g. the enum processor or the starlark processors. While this probably works for oper-status and maybe the IP (by converting it to a large number) the path will be challenging…

I’m not a Splunk expert so I cannot recommend any workaround…

3

Hi @srebhan ,

Thanks for your answer! Sorry that i didn’t found the docs.
Our workaround is not to ingest splunk metrics but events.

we are using a configuration like this, which works fine for us.

[[outputs.http]]
   url = "http://127.0.0.1:8088/services/collector/raw"
   data_format = "json"
   use_batch_format = false

    [outputs.http.headers]
      Content-Type = "application/json"
      Authorization = "Splunk TOKENGOESHERE"
      X-Splunk-Request-Channel = "TOKENGOESHERE"

Best regards,

Andreas

1 Like
4

@schose thanks for letting the community know!