How to create a tag from a field name with Telegraf?

[mohsin106]

Hi,

My optic sensor data coming from a network device looks like this:

 "fields": {
    "Ethernet13/laneStats/0/rxPowerHiAlarm/changes": 0,
    "Ethernet13/laneStats/0/rxPowerHiAlarm/current_value": 0,
    "Ethernet13/laneStats/0/rxPowerHiAlarm/lastChange": 0,
    "Ethernet13/laneStats/0/rxPowerHiWarn/changes": 0,
    "Ethernet13/laneStats/0/rxPowerHiWarn/current_value": 0,
    "Ethernet13/laneStats/0/rxPowerHiWarn/lastChange": 0,
    "Ethernet13/laneStats/0/rxPowerLoAlarm/changes": 0,
    "Ethernet13/laneStats/0/rxPowerLoAlarm/current_value": 0,
    "Ethernet13/laneStats/0/rxPowerLoAlarm/lastChange": 0,
    "Ethernet13/laneStats/0/rxPowerLoWarn/changes": 0,
    "Ethernet13/laneStats/0/rxPowerLoWarn/current_value": 0,
    "Ethernet13/laneStats/0/rxPowerLoWarn/lastChange": 0,
    "Ethernet13/laneStats/0/txPowerHiAlarm/changes": 0,
    "Ethernet13/laneStats/0/txPowerHiAlarm/current_value": 0,
    "Ethernet13/laneStats/0/txPowerHiAlarm/lastChange": 0,
    "Ethernet13/laneStats/0/txPowerHiWarn/changes": 0,
    "Ethernet13/laneStats/0/txPowerHiWarn/current_value": 0,
    "Ethernet13/laneStats/0/txPowerHiWarn/lastChange": 0,
    "Ethernet13/laneStats/0/txPowerLoAlarm/changes": 0,
    "Ethernet13/laneStats/0/txPowerLoAlarm/current_value": 0,
    "Ethernet13/laneStats/0/txPowerLoAlarm/lastChange": 0,
    "Ethernet13/laneStats/0/txPowerLoWarn/changes": 0,
    "Ethernet13/laneStats/0/txPowerLoWarn/current_value": 0,
    "Ethernet13/laneStats/0/txPowerLoWarn/lastChange": 0
  }

To make this data useable I want to remove the first part of the field name (Ethernet13) and store it as a separate tag. Currently, the interface name doesn’t come in as a tag.

I would like the fields to be stored like this:

"fields": {
    "/laneStats/0/rxPowerHiAlarm/changes": 0,
    "/laneStats/0/rxPowerHiAlarm/current_value": 0,
    "/laneStats/0/rxPowerHiAlarm/lastChange": 0,
    "/laneStats/0/rxPowerHiWarn/changes": 0,
    "/laneStats/0/rxPowerHiWarn/current_value": 0,
    "/laneStats/0/rxPowerHiWarn/lastChange": 0,
    "/laneStats/0/rxPowerLoAlarm/changes": 0,
    "/laneStats/0/rxPowerLoAlarm/current_value": 0,
    "/laneStats/0/rxPowerLoAlarm/lastChange": 0,
    "/laneStats/0/rxPowerLoWarn/changes": 0,
    "/laneStats/0/rxPowerLoWarn/current_value": 0,
    "/laneStats/0/rxPowerLoWarn/lastChange": 0,
    "/laneStats/0/txPowerHiAlarm/changes": 0,
    "/laneStats/0/txPowerHiAlarm/current_value": 0,
    "/laneStats/0/txPowerHiAlarm/lastChange": 0,
    "/laneStats/0/txPowerHiWarn/changes": 0,
    "/laneStats/0/txPowerHiWarn/current_value": 0,
    "/laneStats/0/txPowerHiWarn/lastChange": 0,
    "/laneStats/0/txPowerLoAlarm/changes": 0,
    "/laneStats/0/txPowerLoAlarm/current_value": 0,
    "/laneStats/0/txPowerLoAlarm/lastChange": 0,
    "/laneStats/0/txPowerLoWarn/changes": 0,
    "/laneStats/0/txPowerLoWarn/current_value": 0,
    "/laneStats/0/txPowerLoWarn/lastChange": 0
  }

and the tags to be stored like this:

"tags": {
    "host": "69c9bbdbf5-clnkc-telegraf-agent",
    "source": "nep6aara101.mgt.net",
    "interface-name": "Ethernet13"
  }

I tried using the [[processors.regex]] plugin but I don’t think this is possible since I don’t see a way to store the result from the pattern match as a tag. I can’t change the way the data is coming from the device, so any suggestions or ideas would be greatly appreciated. Thank you!

[Franky1]

Is the payload exactly as shown above in the first text block?
Please show us your Telegraf config.

@mohsin106

I tried using the processors.regex plugin but I don’t think this is possible since I don’t see a way to store the result from the pattern match as a tag.

Sure, it is possible

[mohsin106]

@Franky1

Is the payload exactly as shown above in the first text block?

Yes.

Here is my conf file:

Telegraf Conf

[global_tags]
[agent]
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 100000
  collection_jitter = "5s"
  flush_interval = "1m"
  flush_jitter = "5s"
  debug = true
  quiet = false
  hostname = "$containerName-telegraf-agent"
  omit_hostname = false


[[inputs.gnmi]]
  addresses = ["nep6aara101.mgt.net:50051"]
  username = "$routerUser"
  password = "$routerPass"
  encoding = "proto"
  redial = "10s"
  enable_tls = true
  tls_ca = "/etc/telegraf/router_ca.pem"
  insecure_skip_verify = false
  fieldpass = ["neighbors/neighbor/state/management_address",
							 "neighbors/neighbor/state/management_address",
							 "neighbors/neighbor/state/port_description",
							 "neighbors/neighbor/state/port_id",
							 "neighbors/neighbor/state/system_description",
							 "neighbors/neighbor/state/system_name",
							 "avg",
							 "instant",
							 "interval",
							 "max",
							 "max_time",
							 "min",
							 "min_time",
							 "state/admin_status",
							 "ethernet/config/aggregate_id",
							 "state/description",
							 "state/counters/in_broadcast_pkts",
							 "state/counters/in_discards",
							 "state/counters/in_errors",
							 "state/counters/in_fcs_errors",
							 "state/counters/in_multicast_pkts",
							 "state/counters/in_octets",
							 "state/counters/in_unicast_pkts",
							 "aggregation/state/lag_speed",
							 "aggregation/state/lag_type",
							 "state/last_change",
							 "aggregation/config/min_links",
							 "aggregation/config/mlag",
							 "state/oper_status",
							 "state/counters/out_broadcast_pkts",
							 "state/counters/out_discards",
							 "state/counters/out_errors",
							 "state/counters/out_multicast_pkts",
							 "state/counters/out_octets",
							 "state/counters/out_unicast_pkts",
							 "config/interval",
							 "members/member/state/activity",
							 "members/member/state/collecting",
							 "members/member/state/distributing",
							 "members/member/state/synchronization",
							 "members/member/state/timeout",
							 "neighbors/neighbor/state/management_address",
							 "neighbors/neighbor/state/port_description",
							 "neighbors/neighbor/state/port_id",
							 "neighbors/neighbor/state/system_description",
							 "neighbors/neighbor/state/system_name",
							 "available",
							 "utilized"]
  [[inputs.gnmi.subscription]]
    name = "lldp"
    path = "/lldp/interfaces/interface"
    subscription_mode = "sample"
    origin="openconfig"
    sample_interval = "60s"
    suppress_redundant = false    
    heartbeat_interval = "30s"
  [[inputs.gnmi.subscription]]
    name = "bgp"
    path = "/network-instances/network-instance/protocols/protocol/bgp"
    subscription_mode = "sample"
    origin="openconfig"
    sample_interval = "60s"
    suppress_redundant = false    
    heartbeat_interval = "30s"
  [[inputs.gnmi.subscription]]
    name = "mem"
    path = "/components/component/state/memory"
    subscription_mode = "sample"
    origin="openconfig"
    sample_interval = "60s"
    suppress_redundant = false    
    heartbeat_interval = "30s"
  [[inputs.gnmi.subscription]]
    name = "interfaces"
    path = "/interfaces/interface"
    subscription_mode = "sample"
    origin="openconfig"
    sample_interval = "60s"
    suppress_redundant = false    
    heartbeat_interval = "30s"
  [[inputs.gnmi.subscription]]
    name = "lacp"
    path = "/lacp/interfaces/interface"
    subscription_mode = "sample"
    origin="openconfig"
    sample_interval = "60s"
    suppress_redundant = false    
    heartbeat_interval = "30s"
  [[inputs.gnmi.subscription]]
    name = "cpu"
    path = "/components/component/cpu/utilization/state"
    subscription_mode = "sample"
    origin="openconfig"
    sample_interval = "60s"
    suppress_redundant = false    
    heartbeat_interval = "30s"

[[inputs.gnmi]]
  addresses = ["nep6aara101.mgt.cox.net:50051"]
  username = "$routerUser"
  password = "$routerPass"
  encoding = "proto"
  redial = "10s"
  enable_tls = true
  tls_ca = "/etc/telegraf/router_ca.pem"
  insecure_skip_verify = false
  fieldpass = ["*goVendorInfo*",
							 "*rxPower*",
							 "*txPower*",
							 "*lane1TxPower",
							 "*lane3OpticalRxPower",
							 "*lane4TxPower",
							 "*lane2OpticalRxPower",
							 "*lane2TxPower",
							 "*lane1OpticalRxPower",
							 "*lane3TxPower",
							 "*lane4OpticalRxPower"]
  [[inputs.gnmi.subscription]]
    name = "na_optics"
    path="/Sysdb/hardware/archer/xcvr/status/all"
    subscription_mode = "sample"
    origin="eos_native"
    sample_interval = "60s"
    suppress_redundant = false



[[processors.rename]]
  namepass = ["lldp"]
  [[processors.rename.replace]]
    field = "state/counters/tlv_discard"
    dest = "state-counters-tlv-discard"
  [[processors.rename.replace]]
    field = "state/counters/frame_error_in"
    dest = "state-counters-frame-error-in"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/port_description"
    dest = "port-description"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/last_update_time"
    dest = "last-update-time"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/port_id_type"
    dest = "port-id-type"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/management_address_type"
    dest = "management-address-type"
  [[processors.rename.replace]]
    field = "state/name"
    dest = "interface-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/custom_tlvs/tlv/state/value"
    dest = "custom-tlvs-tlv-state-value"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/management_address"
    dest = "management-address"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/system_name"
    dest = "system-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/custom_tlvs/tlv/state/oui"
    dest = "custom-tlvs-tlv-state-oui"
  [[processors.rename.replace]]
    field = "state/counters/tlv_unknown"
    dest = "tlv-unknown"
  [[processors.rename.replace]]
    field = "config/enabled"
    dest = "enabled"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/chassis_id"
    dest = "chassis-id"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/custom_tlvs/tlv/oui"
    dest = "custom-tlvs-tlv-oui"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/custom_tlvs/tlv/type"
    dest = "custom-tlvs-tlv-type"
  [[processors.rename.replace]]
    field = "state/counters/frame_discard"
    dest = "state-counters-frame-discard"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/id"
    dest = "id"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/system_description"
    dest = "system-description"
  [[processors.rename.replace]]
    field = "state/enabled"
    dest = "enabled"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/id"
    dest = "id"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/custom_tlvs/tlv/state/oui_subtype"
    dest = "custom-tlvs-tlv-state-oui-subtype"
  [[processors.rename.replace]]
    field = "name"
    dest = "interface-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/port_id"
    dest = "port-id"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/custom_tlvs/tlv/oui_subtype"
    dest = "custom-tlvs-tlv-oui-subtype"
  [[processors.rename.replace]]
    field = "config/name"
    dest = "interface-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/chassis_id_type"
    dest = "chassis-id-type"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/custom_tlvs/tlv/state/type"
    dest = "custom-tlvs-tlv-state-type"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/registration_time"
    dest = "registration-time"
  [[processors.rename.replace]]
    field = "state/counters/frame_in"
    dest = "frame-in"
  [[processors.rename.replace]]
    field = "state/counters/frame_out"
    dest = "frame-out"
  [[processors.rename.replace]]
    tag = "source"
    dest = "device"
  [[processors.rename.replace]]
    tag = "name"
    dest = "interface-name"

[[processors.rename]]
  namepass = ["bgp"]
  [[processors.rename.replace]]
    field = "neighbors/neighbor/afi_safis/afi_safi/state/enabled"
    dest = "afi-safi-enabled"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/config/peer_as"
    dest = "peer-as"
  [[processors.rename.replace]]
    field = "global/use_multiple_paths/state/enabled"
    dest = "use-multiple-paths-enabled"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/afi_safis/afi_safi/state/prefixes/best_paths"
    dest = "prefixes-best-paths"
  [[processors.rename.replace]]
    field = "global/afi_safis/afi_safi/afi_safi_name"
    dest = "afi-safi-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/neighbor_address"
    dest = "address"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/afi_safis/afi_safi/config/enabled"
    dest = "afi-safi-enabled"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/afi_safis/afi_safi/config/afi_safi_name"
    dest = "peer-group-afi-safi-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/config/send_community"
    dest = "send-community"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/peer_group_name"
    dest = "peer-group-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/config/enabled"
    dest = "enabled"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/config/peer_group_name"
    dest = "peer-group-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/ebgp_multihop/config/multihop_ttl"
    dest = "ebgp-multihop-multihop-ttl"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/config/peer_as"
    dest = "peer-as"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/ebgp_multihop/state/multihop_ttl"
    dest = "ebgp-multihop-multihop-ttl"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/afi_safis/afi_safi/state/prefixes/sent"
    dest = "prefixes-sent"
  [[processors.rename.replace]]
    field = "global/use_multiple_paths/config/enabled"
    dest = "use-multiple-paths-enabled"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/neighbor_address"
    dest = "address"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/ebgp_multihop/state/enabled"
    dest = "peer-group-ebgp-multihop-enabled"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/as_path_options/config/replace_peer_as"
    dest = "replace-peer-as"
  [[processors.rename.replace]]
    field = "global/config/router_id"
    dest = "router-id"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/peer_group"
    dest = "peer-group"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/ebgp_multihop/config/multihop_ttl"
    dest = "peer-group-ebgp-multihop-multihop-ttl"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/afi_safis/afi_safi/afi_safi_name"
    dest = "afi-safi-name"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/transport/state/mtu_discovery"
    dest = "peer-group-mtu-discovery"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/as_path_options/config/replace_peer_as"
    dest = "peer-group-replace-peer-as"
  [[processors.rename.replace]]
    field = "global/afi_safis/afi_safi/state/afi_safi_name"
    dest = "afi-safi-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/messages/sent/UPDATE"
    dest = "sent-update"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/as_path_options/state/replace_peer_as"
    dest = "replace-peer-as"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/afi_safis/afi_safi/state/prefixes/installed"
    dest = "prefixes-installed"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/session_state"
    dest = "session-state"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/afi_safis/afi_safi/state/prefixes/best_ecmp_paths"
    dest = "prefixes-best-ecmp-paths"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/afi_safis/afi_safi/afi_safi_name"
    dest = "peer-group-afi-safi-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/ebgp_multihop/state/enabled"
    dest = "ebgp-multihop-enabled"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/state/peer_as"
    dest = "peer-as"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/afi_safis/afi_safi/state/prefixes/received"
    dest = "prefixes-received"
  [[processors.rename.replace]]
    field = "global/afi_safis/afi_safi/config/afi_safi_name"
    dest = "afi-safi-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/send_community"
    dest = "send-community"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/enabled"
    dest = "enabled"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/ebgp_multihop/state/multihop_ttl"
    dest = "peer-group-ebgp-multihop-multihop-ttl"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/ebgp_multihop/config/enabled"
    dest = "peer-group-ebgp-multihop-enabled"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/afi_safis/afi_safi/state/afi_safi_name"
    dest = "afi-safi-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/description"
    dest = "description"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/transport/state/mtu_discovery"
    dest = "mtu-discovery"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/transport/state/remote_port"
    dest = "remote-port"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/transport/config/mtu_discovery"
    dest = "peer-group-mtu-discovery"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/state/send_community"
    dest = "peer-group-send-community"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/established_transitions"
    dest = "established-transitions"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/config/neighbor_address"
    dest = "address"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/as_path_options/state/replace_peer_as"
    dest = "peer-group-replace-peer-as"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/peer_as"
    dest = "peer-as"
  [[processors.rename.replace]]
    field = "global/state/router_id"
    dest = "router-id"
  [[processors.rename.replace]]
    field = "global/config/as"
    dest = "as"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/afi_safis/afi_safi/state/afi_safi_name"
    dest = "peer-group-afi-safi-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/config/peer_group"
    dest = "peer-queue"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/transport/state/remote_address"
    dest = "remote-address"
  [[processors.rename.replace]]
    field = "global/state/as"
    dest = "as"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/transport/config/mtu_discovery"
    dest = "mtu-discovery"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/ebgp_multihop/config/enabled"
    dest = "enabledebgp-multihop-enabled"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/last_established"
    dest = "last-established"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/afi_safis/afi_safi/config/afi_safi_name"
    dest = "afi-safi-name"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/state/messages/received/UPDATE"
    dest = "received-update"
  [[processors.rename.replace]]
    field = "neighbors/neighbor/config/description"
    dest = "description"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/config/send_community"
    dest = "peer-group-send-community"
  [[processors.rename.replace]]
    field = "peer_groups/peer_group/state/peer_group_name"
    dest = "peer-group-name"
  [[processors.rename.replace]]
    tag = "afi_safi_name"
    dest = "afi-safi-name"
  [[processors.rename.replace]]
    tag = "/network-instances/network-instance/protocols/protocol/name"
    dest = "protocol-name"
  [[processors.rename.replace]]
    tag = "peer_group_name"
    dest = "peer-group-name"
  [[processors.rename.replace]]
    tag = "name"
    dest = "name-tag"
  [[processors.rename.replace]]
    tag = "source"
    dest = "device"
  [[processors.rename.replace]]
    tag = "neighbor_address"
    dest = "address"

[[processors.rename]]
  namepass = ["mem"]
  [[processors.rename.replace]]
    field = "cpu_utilization"
    dest = "cpu-utilization"
  [[processors.rename.replace]]
    field = "memory_usage"
    dest = "memory-usage"
  [[processors.rename.replace]]
    field = "start_time"
    dest = "start-time"
  [[processors.rename.replace]]
    field = "cpu_usage_system"
    dest = "cpu-usage-system"
  [[processors.rename.replace]]
    field = "cpu_usage_user"
    dest = "cpu-usage-user"
  [[processors.rename.replace]]
    field = "name"
    dest = "process-name"
  [[processors.rename.replace]]
    tag = "source"
    dest = "device"
  [[processors.rename.replace]]
    tag = "name"
    dest = "chassis-name"

[[processors.rename]]
  namepass = ["optics"]
  [[processors.rename.replace]]
    field = "date_code"
    dest = "date-code"
  [[processors.rename.replace]]
    field = "serial_no"
    dest = "serial-number"
  [[processors.rename.replace]]
    field = "vendor_rev"
    dest = "vendor-rev"
  [[processors.rename.replace]]
    field = "vendor_part"
    dest = "vendor-part"
  [[processors.rename.replace]]
    tag = "name"
    dest = "interface-name"

[[processors.rename]]
  namepass = ["interfaces"]
  [[processors.rename.replace]]
    field = "config/loopback_mode"
    dest = "loopback-mode"
  [[processors.rename.replace]]
    field = "state/counters/out_discards"
    dest = "out-discards"
  [[processors.rename.replace]]
    field = "ethernet/config/auto_negotiate"
    dest = "auto-negotiate"
  [[processors.rename.replace]]
    field = "hold_time/config/up"
    dest = "hold-time-up"
  [[processors.rename.replace]]
    field = "ethernet/pfc/priorities/priority/state/out_frames"
    dest = "out-frames"
  [[processors.rename.replace]]
    field = "state/name"
    dest = "interface-name"
  [[processors.rename.replace]]
    field = "out_octets"
    dest = "out-octets"
  [[processors.rename.replace]]
    field = "state/counters/in_unicast_pkts"
    dest = "in-unicast-pkts"
  [[processors.rename.replace]]
    field = "config/tpid"
    dest = "tpid"
  [[processors.rename.replace]]
    field = "state/mtu"
    dest = "mtu"
  [[processors.rename.replace]]
    field = "out_errors"
    dest = "out-errors"
  [[processors.rename.replace]]
    field = "aggregation/state/min_links"
    dest = "min-links"
  [[processors.rename.replace]]
    field = "state/counters/out_multicast_pkts"
    dest = "out-multicast-pkts"
  [[processors.rename.replace]]
    field = "state/description"
    dest = "description"
  [[processors.rename.replace]]
    field = "ethernet/state/counters/out_mac_control_frames"
    dest = "out-mac-control-frames"
  [[processors.rename.replace]]
    field = "ethernet/state/mac_address"
    dest = "mac-address"
  [[processors.rename.replace]]
    field = "ethernet/config/mac_address"
    dest = "mac-address"
  [[processors.rename.replace]]
    field = "out_multicast_pkts"
    dest = "out-multicast-pkts"
  [[processors.rename.replace]]
    field = "config/load_interval"
    dest = "load-interval"
  [[processors.rename.replace]]
    field = "in_octets"
    dest = "in-octets"
  [[processors.rename.replace]]
    field = "ethernet/state/counters/in_mac_pause_frames"
    dest = "in-mac-pause-frames"
  [[processors.rename.replace]]
    field = "name"
    dest = "interface-name"
  [[processors.rename.replace]]
    field = "state/counters/in_multicast_pkts"
    dest = "in-multicast-pkts"
  [[processors.rename.replace]]
    field = "config/description"
    dest = "description"
  [[processors.rename.replace]]
    field = "state/counters/in_broadcast_pkts"
    dest = "in-broadcast-pkts"
  [[processors.rename.replace]]
    field = "ethernet/state/negotiated_port_speed"
    dest = "negotiated-port-speed"
  [[processors.rename.replace]]
    field = "ethernet/state/counters/in_jabber_frames"
    dest = "in-jabber-frames"
  [[processors.rename.replace]]
    field = "ethernet/state/counters/out_mac_pause_frames"
    dest = "out-mac-pause-frames"
  [[processors.rename.replace]]
    field = "aggregation/switched_vlan/config/native_vlan"
    dest = "native-vlan"
  [[processors.rename.replace]]
    field = "out_unicast_pkts"
    dest = "out-unicast-pkts"
  [[processors.rename.replace]]
    field = "ethernet/state/counters/in_fragment_frames"
    dest = "in-fragment-frames"
  [[processors.rename.replace]]
    field = "ethernet/pfc/priorities/priority/state/index"
    dest = "index"
  [[processors.rename.replace]]
    field = "ethernet/config/port_speed"
    dest = "port-speed"
  [[processors.rename.replace]]
    field = "ethernet/config/fec_encoding/reed_solomon"
    dest = "reed-solomon"
  [[processors.rename.replace]]
    field = "state/counters/in_errors"
    dest = "in-errors"
  [[processors.rename.replace]]
    field = "aggregation/state/lag_speed"
    dest = "lag-speed"
  [[processors.rename.replace]]
    field = "ethernet/config/sfp_1000base_t"
    dest = "sfp-1000base-t"
  [[processors.rename.replace]]
    field = "state/counters/out_errors"
    dest = "out-errors"
  [[processors.rename.replace]]
    field = "in_broadcast_pkts"
    dest = "in-broadcast-pkts"
  [[processors.rename.replace]]
    field = "in_errors"
    dest = "in-errors"
  [[processors.rename.replace]]
    field = "state/inactive"
    dest = "inactive"
  [[processors.rename.replace]]
    field = "in_multicast_pkts"
    dest = "in-multicast-pkts"
  [[processors.rename.replace]]
    field = "aggregation/config/fallback_timeout"
    dest = "fallback-timeout"
  [[processors.rename.replace]]
    field = "state/counters/out_octets"
    dest = "out-octets"
  [[processors.rename.replace]]
    field = "ethernet/state/port_speed"
    dest = "port-speed"
  [[processors.rename.replace]]
    field = "ethernet/state/enable_flow_control"
    dest = "enable-flow-control"
  [[processors.rename.replace]]
    field = "aggregation/config/fallback"
    dest = "fallback"
  [[processors.rename.replace]]
    field = "ethernet/state/counters/in_oversize_frames"
    dest = "in-oversize-frames"
  [[processors.rename.replace]]
    field = "aggregation/config/mlag"
    dest = "mlag"
  [[processors.rename.replace]]
    field = "ethernet/state/hw_mac_address"
    dest = "hw-mac-address"
  [[processors.rename.replace]]
    field = "hold_time/config/down"
    dest = "hold-time-down"
  [[processors.rename.replace]]
    field = "aggregation/switched_vlan/config/interface_mode"
    dest = "interface-mode"
  [[processors.rename.replace]]
    field = "ethernet/config/fec_encoding/disabled"
    dest = "disabled"
  [[processors.rename.replace]]
    field = "config/mtu"
    dest = "mtu"
  [[processors.rename.replace]]
    field = "state/last_change"
    dest = "last-change"
  [[processors.rename.replace]]
    field = "state/ifindex"
    dest = "ifindex"
  [[processors.rename.replace]]
    field = "ethernet/state/counters/in_crc_errors"
    dest = "in-crc-errors"
  [[processors.rename.replace]]
    field = "out_discards"
    dest = "out-discards"
  [[processors.rename.replace]]
    field = "state/admin_status"
    dest = "admin-status"
  [[processors.rename.replace]]
    field = "state/loopback_mode"
    dest = "loopback-mode"
  [[processors.rename.replace]]
    field = "state/oper_status"
    dest = "oper-status"
  [[processors.rename.replace]]
    field = "config/enabled"
    dest = "enabled"
  [[processors.rename.replace]]
    field = "ethernet/config/fec_encoding/fire_code"
    dest = "fire-code"
  [[processors.rename.replace]]
    field = "aggregation/config/lag_type"
    dest = "lag-type"
  [[processors.rename.replace]]
    field = "state/hardware_port"
    dest = "hardware-port"
  [[processors.rename.replace]]
    field = "ethernet/state/negotiated_duplex_mode"
    dest = "negotiated-duplex-mode"
  [[processors.rename.replace]]
    field = "state/type"
    dest = "type"
  [[processors.rename.replace]]
    field = "ethernet/config/duplex_mode"
    dest = "duplex-mode"
  [[processors.rename.replace]]
    field = "ethernet/config/fec_encoding/reed_solomon544"
    dest = "reed-solomon544"
  [[processors.rename.replace]]
    field = "state/counters/out_broadcast_pkts"
    dest = "out-broadcast-pkts"
  [[processors.rename.replace]]
    field = "ethernet/state/auto_negotiate"
    dest = "auto-negotiate"
  [[processors.rename.replace]]
    field = "out_broadcast_pkts"
    dest = "out-broadcast-pkts"
  [[processors.rename.replace]]
    field = "state/enabled"
    dest = "enabled"
  [[processors.rename.replace]]
    field = "state/tpid"
    dest = "tpid"
  [[processors.rename.replace]]
    field = "state/counters/in_discards"
    dest = "in-discards"
  [[processors.rename.replace]]
    field = "ethernet/pfc/priorities/priority/state/in_frames"
    dest = "in-frames"
  [[processors.rename.replace]]
    field = "ethernet/pfc/priorities/priority/index"
    dest = "index"
  [[processors.rename.replace]]
    field = "config/type"
    dest = "type"
  [[processors.rename.replace]]
    field = "state/counters/in_octets"
    dest = "in-octets"
  [[processors.rename.replace]]
    field = "in_unicast_pkts"
    dest = "in-unicast-pkts"
  [[processors.rename.replace]]
    field = "state/counters/in_fcs_errors"
    dest = "in-fcs-errors"
  [[processors.rename.replace]]
    field = "aggregation/state/lag_type"
    dest = "lag-type"
  [[processors.rename.replace]]
    field = "in_discards"
    dest = "in-discards"
  [[processors.rename.replace]]
    field = "ethernet/config/forwarding_viable"
    dest = "forwarding-viable"
  [[processors.rename.replace]]
    field = "config/name"
    dest = "config-name"
  [[processors.rename.replace]]
    field = "in_fcs_errors"
    dest = "in-fcs-errors"
  [[processors.rename.replace]]
    field = "ethernet/state/forwarding_viable"
    dest = "forwarding-viable"
  [[processors.rename.replace]]
    field = "subinterfaces/subinterface/config/description"
    dest = "description"
  [[processors.rename.replace]]
    field = "hold_time/state/down"
    dest = "hold-time-down"
  [[processors.rename.replace]]
    field = "state/counters/out_unicast_pkts"
    dest = "out-unicast-pkts"
  [[processors.rename.replace]]
    field = "aggregation/state/fallback"
    dest = "fallback"

[[processors.rename]]
  namepass = ["cpu"]
  [[processors.rename.replace]]
    field = "min_time"
    dest = "min-time"
  [[processors.rename.replace]]
    field = "max_time"
    dest = "max-time"
  [[processors.rename.replace]]
    tag = "source"
    dest = "device"
  [[processors.rename.replace]]
    tag = "name"
    dest = "cpu-name"

[[processors.rename]]
  namepass = ["lacp"]
  [[processors.rename.replace]]
    field = "members/member/state/counters/lacp_out_pkts"
    dest = "lacp-out-pkts"
  [[processors.rename.replace]]
    field = "members/member/state/oper_key"
    dest = "oper-key"
  [[processors.rename.replace]]
    field = "members/member/state/timeout"
    dest = "timeout"
  [[processors.rename.replace]]
    field = "member/state/activity"
    dest = "activity"
  [[processors.rename.replace]]
    field = "member/interface"
    dest = "interface"
  [[processors.rename.replace]]
    field = "name"
    dest = "bundle"
  [[processors.rename.replace]]
    field = "member/state/aggregatable"
    dest = "aggregatable"
  [[processors.rename.replace]]
    field = "members/member/state/synchronization"
    dest = "synchronization"
  [[processors.rename.replace]]
    field = "members/member/state/partner_key"
    dest = "partner-key"
  [[processors.rename.replace]]
    tag = "source"
    dest = "device"
  [[processors.rename.replace]]
    tag = "name"
    dest = "bundle"

[[processors.rename]]
  namepass = ["na_optics"]
  [[processors.rename.replace]]
    tag = "name"
    dest = "interface-name"

[[processors.rename]]
[[processors.rename.replace]]
    measurement = "na_optics"
    dest = "optics"

[[processors.rename]]
  namepass = ["optics1"]
  [[processors.rename.replace]]
    tag = "name"
    dest = "interface-name"

[[processors.rename]]
  namepass = ["optics3"]
  [[processors.rename.replace]]
    tag = "name"
    dest = "interface-name"

[[processors.rename]]
  namepass = ["optics2"]
  [[processors.rename.replace]]
    tag = "name"
    dest = "interface-name"

[[processors.rename]]
  namepass = ["optics5"]
  [[processors.rename.replace]]
    tag = "name"
    dest = "interface-name"

[[processors.rename]]
  namepass = ["optics4"]
  [[processors.rename.replace]]
    tag = "name"
    dest = "interface-name"

[[processors.rename]]
  namepass = ["optics7"]
  [[processors.rename.replace]]
    tag = "name"
    dest = "interface-name"

[[processors.rename]]
  namepass = ["optics6"]
  [[processors.rename.replace]]
    tag = "name"
    dest = "interface-name"

[[processors.regex]]
namepass = ["na_optics"]
  [[processors.regex.field_rename]]
  pattern = "^(Ethernet)(\\d+)\\W(\\w+\\W){1,3}(.*)"
  replacement = "${1}${2}-${4}"
  result_key = "overwrite"

[[processors.starlark]]
namepass = ["lldp", "bgp", "mem", "interfaces", "lacp", "cpu"]
  source = '''
load('time.star', 'time')
def apply(metric):
  metric.time = time.now().unix_nano
  return metric
'''

[[processors.starlark]]
namepass = ["bgp"]
  source = '''
load("time.star", "time")
load("logging.star", "log")
def apply(metric):
  for k, v in metric.fields.items():
    # log.debug("This is k: {}".format(k))
    if k == "neighbors/neighbor/state/last_established":
      # new_date = time.from_timestamp(int(v / 1e9))
      metric.fields.pop(k)
      metric.fields["last-established"] = int(v / 1e6)
      # log.debug("This is last-established: {}".format(metric.fields.get("last-established")))
  return [metric]
'''

[[processors.starlark]]
namepass = ["interfaces"]
  source = '''
load("time.star", "time")
load("logging.star", "log")
def apply(metric):
  for k, v in metric.fields.items():
    #log.debug("This is k: {}".format(k))
    if k == "state/last_change":
      #new_date = time.from_timestamp(int(v / 1e9))
      metric.fields.pop(k)
      metric.fields["last-change"] = int(v / 1e6)
      #log.debug("This is last-change: {}".format(metric.fields.get("last-change")))
  return [metric]
'''

[[outputs.file]]
  ## Files to write to, "stdout" is a specially handled file.
  files = ["stdout"]

I’ve been testing the regex plugin and have been able to shorten the field names like this:

namepass = ["na_optics"]
  [[processors.regex.field_rename]]
  pattern = "^(Ethernet)(\\d+)\\W(\\w+\\W){1,3}(.*)"
  replacement = "${1}${2}-${4}"
  result_key = "overwrite"

So now the fields are coming through like this:

"fields": {
    "Ethernet49-changes": 0,
    "Ethernet49-current_value": 0,
    "Ethernet49-lastChange": 0
  }

I want to now pop off the “Ethernet49” and store it as a tag, and make the new field names look like this:

"fields": {
    "changes": 0,
    "current_value": 0,
    "lastChange": 0
  }

[Franky1]

@mohsin106

Here is my conf file:

Phew, kind of a config hell…
Are you aware, that the order parameter matters if you have more than one processors plugin?
Also, I am sure that you could summarise many processors.

@mohsin106

So now the fields are coming through like this:

This kind of presentation confuses me because Telegraf does not work with json, the stdout output is in the influx line protocol.

@mohsin106

I want to now pop off the “Ethernet49” and store it as a tag, and make the new field names look like this:

Here an idea how it could work, not tested, change it to your needs:

[[processors.regex]]
  namepass = ["na_optics"]
  [[processors.regex.field_rename]]
    pattern = "^(Ethernet)(\\d+)\\W(\\w+\\W){1,3}(.*)"
    replacement = "${1}${2}-${4}"
    result_key = "overwrite"
  [[processors.regex.tags]]  # pulls out the EthernetXY as new tag:
    key = "overwrite"
    pattern = '^(\w+)-.+'
    replacement = "${1}"
    result_key = "interface-name"
  [[processors.regex.fields]]  # removes the EthernetXY- from field:
    key = "overwrite"
    pattern =  '^\w+-(.+)'
    replacement = "${1}"

[mohsin106]

@Franky1

Are you aware, that the order parameter matters if you have more than one processors plugin?

I wasn’t aware. Thank you for letting me know. I can now control my data transformations more efficiently!

The regex plugin didn’t work for me. But I was able to get Starlark to do what I need.
This is how I got it to work with Starlark:

[[processors.starlark]]
order = 1
namepass = ["na_optics"]
  source = '''
def apply(metric):
  for k, v in metric.fields.items():
    if k.startswith("Ethernet"):
      tmp = k.split("/")
      metric.tags["interface-name"] = tmp[0]
      tmp.pop(0)
      tmp2 = "-".join(tmp)
      metric.fields[str(tmp2)] = v
      metric.fields.pop(k)
  return [metric]
'''