Using Telegraf to get Data from TTN via MQTT - retrieving all members from arrays

Telegraf
, ,
1

Hi,
I’m using Telegraf to collect sensor data from TTN via MQTT. All in all it works fine.
But there is one thing I do not manage.

In the data provided by TTN there is information about the gateway that has received the sensor data. There can be the situation, that more than one gateway receives the sensor data. TTN handles that and adds data for all receiving gateways in an array with gateway information. Here is an example:

...
"rx_metadata": [
          {
            "gateway_ids": {
              "gateway_id": "eui-acxxxxxxxxxcb663",
              "eui": "ACxxxxxxxxxCB663"
            },
            "time": "2024-04-30T06:11:11.383873939Z",
            "timestamp": 3868520174,
            "rssi": -123,
            "channel_rssi": -123,
            "snr": -8.25,
            "location": {
              "latitude": 52.0917353400241,
              "longitude": 9.36717710439505,
              "source": "SOURCE_REGISTRY"
            },
            "uplink_token": "xxxxxxxxx",
            "received_at": "2024-04-30T06:11:11.824075916Z"
          },
          {
            "gateway_ids": {
              "gateway_id": "eui-a8xxxxxxxxx3a3c4",
              "eui": "A8xxxxxxxxx3A3C4"
            },
            "time": "2024-04-30T06:11:11.822201Z",
            "timestamp": 80545373,
            "rssi": -37,
            "channel_rssi": -37,
            "snr": 13.5,
            "frequency_offset": "-7155",
            "location": {
              "latitude": 52.1017853556478,
              "longitude": 9.37492448586261,
              "altitude": 60,
              "source": "SOURCE_REGISTRY"
            },
            "uplink_token": "xxxxxxxxx",
            "received_at": "2024-04-30T06:11:11.846167537Z"
          }
        ],

As you can see “rx_metadata” is an array that has 2 members with data for both receiving gateways.
I’m using this Telegraf configuration do move the data to an InfluxDB:

[[outputs.influxdb_v2]]
  alias = "ttn_vhs_bucket_loracon"
  namepass = ["ttn_vhs_loracon"]
...

[[inputs.mqtt_consumer]]
  alias = "ttn_consumer_vhs_loracon"
  name_override = "ttn_vhs_loracon"
  servers = ["tcp://eu1.cloud.thethings.network:1883"]
  topics = ["#"]
  max_undelivered_messages = 1
...
[[inputs.mqtt_consumer.json_v2.object]]
    	path = "uplink_message.rx_metadata"
    	disable_prepend_keys = false
    	excluded_keys = ["time", "timestamp"]

As a result I get the “rx_metadata” correctly, but only the last member of that array. More precisely it is the 2nd member, since I hadn’t had the case that more than 2 gateway are receiving the data. But I thing is safe to assume, that it is the last array entry.

Do you have an advice how to handle data from all array entries in Telegraf / InfluxDB?

2

Using this data:

{
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "eui-acxxxxxxxxxcb663",
          "eui": "ACxxxxxxxxxCB663"
        },
        "time": "2024-04-30T06:11:11.383873939Z",
        "timestamp": 3868520174,
        "rssi": -123,
        "channel_rssi": -123,
        "snr": -8.25,
        "location": {
          "latitude": 52.0917353400241,
          "longitude": 9.36717710439505,
          "source": "SOURCE_REGISTRY"
        },
        "uplink_token": "xxxxxxxxx",
        "received_at": "2024-04-30T06:11:11.824075916Z"
      },
      {
        "gateway_ids": {
          "gateway_id": "eui-a8xxxxxxxxx3a3c4",
          "eui": "A8xxxxxxxxx3A3C4"
        },
        "time": "2024-04-30T06:11:11.822201Z",
        "timestamp": 80545373,
        "rssi": -37,
        "channel_rssi": -37,
        "snr": 13.5,
        "frequency_offset": "-7155",
        "location": {
          "latitude": 52.1017853556478,
          "longitude": 9.37492448586261,
          "altitude": 60,
          "source": "SOURCE_REGISTRY"
        },
        "uplink_token": "xxxxxxxxx",
        "received_at": "2024-04-30T06:11:11.846167537Z"
      }
    ]
  }

and this config:

[[inputs.file]]
  files = ["file2.json"]
  data_format = "json_v2"
  [[inputs.file.json_v2]]
  [[inputs.file.json_v2.object]]
    path = "rx_metadata"

get’s me two different metrics:

file gateway_ids_gateway_id="eui-acxxxxxxxxxcb663",gateway_ids_eui="ACxxxxxxxxxCB663",time="2024-04-30T06:11:11.383873939Z",timestamp=3868520174,rssi=-123,channel_rssi=-123,snr=-8.25,location_latitude=52.0917353400241,location_longitude=9.36717710439505,location_source="SOURCE_REGISTRY",uplink_token="xxxxxxxxx",received_at="2024-04-30T06:11:11.824075916Z" 1714507257000000000
file gateway_ids_gateway_id="eui-a8xxxxxxxxx3a3c4",gateway_ids_eui="A8xxxxxxxxx3A3C4",time="2024-04-30T06:11:11.822201Z",timestamp=80545373,rssi=-37,channel_rssi=-37,snr=13.5,frequency_offset="-7155",location_latitude=52.1017853556478,location_longitude=9.37492448586261,location_altitude=60,location_source="SOURCE_REGISTRY",uplink_token="xxxxxxxxx",received_at="2024-04-30T06:11:11.846167537Z" 1714507257000000000

However, there are no tags to differentiate between the two different gateways, you need to specify either a different timestamp on each or better yet, set the gateway_id and/or eui as a tag.

3

Thank you very much for your answer!

Hmm, doesn’t look too much different from mine. :thinking:. Ok, maybe I have to dig deeper in the documentation.

The idea to create tags from the gateway IDs sound interesting. Not sure how to do that, but I guess I will find something in the docs.

4

Take a look at adding something like:

[[inputs.file.json_v2.object.tag]]
path = “gateway_id”

You can also use the converter processor to change a field to a tag.

1 Like