Unable to View Twin Id and Other Details from Kafka in InfluxDB

Telegraf
, , ,
1

I’m facing an issue while using InfluxDB to store data from IoT devices (Eclipse Ditto) through Kafka using Telegraf. The data I’m receiving is in JSON format, containing information about various attributes and features of the devices. However, I’m only able to see the “features” data in InfluxDB, and information like the “thingId” and other attributes are missing.

Here’s an example of the JSON data I’m working with:

{
  "thingId": "org.acme:thing001",
  "attributes": {
    "location": "Device created for the demo"
  },  
  "features": {
    "temperature": {
      "properties": {
        "value": 30
      }
    },
    "humidity": {
      "properties": {
        "value": 40
      }
    },
    "pressure": {
      "properties": {
        "value": 26
      }
    }    
  }
}

My Telegraf configuration for consuming this data from Kafka and writing it to InfluxDB is as follows:

[[outputs.influxdb_v2]]
  urls = ["EC2 IP & Port"]
  token = "Token"
  organization = "Org"
  bucket = "Ditto"

[[inputs.kafka_consumer]]
  brokers = ["EC2 IP & Port"]
  topics = ["Kafka Target"]
  max_message_len = 1000000
  data_format = "json"

I suspect that there might be an issue in my Telegraf configuration or possibly with the InfluxDB write process that’s causing the “thingId” and attributes to not be properly stored or displayed in InfluxDB.

Has anyone encountered a similar issue? How can I ensure that the entire JSON data, including the “thingId” and attributes, is visible and queryable in InfluxDB? Any guidance or suggestions would be greatly appreciated. Thank you!

2

Hello @karthick_G,
Hmm that’s odd:

It should handle the nested json…

I’d try:

 data_format = "json"
  tag_keys = ["thingID"]
  json_string_fields = ["temperature.properties.value", "humidity.properties.value", "pressure.properties.value"]
  json_query = "features"

Alternatively, you might want to use the JSON v2 formatter instead:

3

@Anaisdg - I appreciate the solution you recommended; following your advice worked perfectly!

 data_format = "json"
  tag_keys = ["thingId"]
  json_string_fields = ["temperature.properties.value", "humidity.properties.value", "pressure.properties.value"]
  json_query = "features"

Once again, thank you so much for your help:))

4

@Anaisdg - I wanted to add the attributes(Strings) also to be stored inside the influxDB. I’m only able to see the “features” data in InfluxDB but other attributes are missing(Strings).

Here’s an example of the JSON data I’m working with:

{
  "thingId": "org.acme:thing001",
  "attributes": {
    "location": "Device created for the demo",
    "manufacturer": "Acme Corporation",
    "model": "SmartThing-5000",
    "serialNumber": "SN123456789",
    "firmwareVersion": "v1.2.3",
    "status": "active"
  },  
  "features": {
    "temperature": {
      "properties": {
        "value": 30
      }
    },
    "humidity": {
      "properties": {
        "value": 40
      }
    },
    "pressure": {
      "properties": {
        "value": 26
      }
    }    
  }
}

I tried this

data_format = "json"
  tag_keys = ["thingID"]
  json_string_fields = ["attributes.manufacturer",  "attributes.model","attributes.status"]
  json_query = "value"