Telegraf parsing/processing JSON data (dynamic field name)

Telegraf
1

Hello :slight_smile:

I got some issues storing the following data into InfluxDB.
All I need is basically 2 values.
Which isn’t an issue to get but to transform the values into the structure I want.

I tried working with tags, processor->rename, starlark etc. but nothing was successful.

I am requesting multiple sensor data using “inputs.http” plugin.

components[0]->status->value
components[0]->config->name

{
  "components": [
    {
      "key": "bthomesensor:201",
      "status": {
        "id": 201,
        "value": 34,
        "last_updated_ts": 1777738048
      },
      "config": {
        "id": 201,
        "addr": "...",
        "name": "humidity_kitchen",
        "obj_id": 46,
        "idx": 0,
        "meta": {
          "ui": {
            "icon": null
          }
        }
      }
    }
  ],
  "cfg_rev": 50,
  "offset": 0,
  "total": 2
}

[[inputs.http]]
  name_override = "sensors"
  urls = [
    'http://<kitchen>/rpc/Shelly.GetComponents?keys=["bthomesensor:201"]',
    'http://<other>/rpc/Shelly.GetComponents?keys=["bthomesensor:201"]',
    'http://<outside>/rpc/Shelly.GetComponents?keys=["bthomesensor:201"]'
  ]
  data_format = "json"
  tagexclude = ["url"]

The structure I want to archive:

_measurement  -> sensors
_field        -> humidity
 tag          -> room          -> humidity_kitchen

Additionally the value of the tag room should be processed, trimming it’s prefix “humidity_”

_measurement | _field   | _value | room
------------------------------------------
sensors      | humidity | 34     | kitchen
sensors      | humidity | 38     | other
sensors      | humidity | 44     | outside

Additionally I need to get the data of a few other sensor, too.
They follow the same structure but with a different config name.

$ http://<kitchen>/rpc/Shelly.GetComponents?keys=["bthomesensor:202"]

{
  "components": [
    {
      "key": "bthomesensor:202",
      "status": {
        "id": 202,
        "value": 18,
        "last_updated_ts": 1777745372
      },
      "config": {
        "id": 202,
        "addr": "...",
        "name": "temperature_kitchen",
        "obj_id": 69,
        "idx": 0,
        "meta": {
          "ui": {
            "icon": null
          }
        }
      }
    }
  ],
  "cfg_rev": 50,
  "offset": 0,
  "total": 1
}

Parsed data to store in InfluxDB

_measurement | _field      | _value | room
------------------------------------------
sensors      | temperature | 18     | kitchen
sensors      | temperature | 22     | other
sensors      | temperature | 16     | outside

I guess I need to use multiple “inputs.http” configs, to process different data.

The goal is to store humidity, temperature, window and door (opened 1, closed 0).

**Appreciate any help…**

This is driving my crazy for multiple days now.

Or if possible parsing the data even more dynamic using the value of:

components[0]->config->name

temperature_kitchen
temperature_other
humidity_outside
window_other
…

Parsed maybe via. regex (divider _ ):

FIELD-NAME _ ROOM

_measurement | _field      | _value | room
------------------------------------------
sensors      | temperature | 18     | kitchen
sensors      | temperature | 22     | other
sensors      | humidity    | 16     | outside
sensors      | window      | 1      | other
2

I believe you’re on the right track with trying Starlark. I’d recommend using the json_v2 input plugin instead of json for your data format. By specifying included_keys to the fields you care about, you should be able to extract what you need, then use Starlark to split on the underscore in the humidity_kitchen value to make sure that the single value can get written to two separate columns, with something like:

def apply(metric):
    name = metric.fields.get("components_config_name")
    parts = name.split("_", 1)

    field = parts[0]
    room = parts[1]

    metric.tags["field"] = field
    metric.tags["room"] = room

The rest of it should be pretty easy from there.

You’ll definitely need to do different instances of the input plugin (and processing) for different data models, though.

3

Thank you very much for pointing me into the right direction.

I literally didn’t know where to start.
Reading docs and examples, inlcuding yours, to starlark, enabling debug logging and trying, trying, trying… finally solved it.

[[inputs.http]]
  name_override = "sensors"
  urls = [
    'http://<kitchen>/rpc/Shelly.GetComponents?keys=["bthomesensor:201"]',
    'http://<kitchen>/rpc/Shelly.GetComponents?keys=["bthomesensor:202"]',
    'http://<sleeping>/rpc/Shelly.GetComponents?keys=["bthomesensor:201"]',
    'http://<sleeping>/rpc/Shelly.GetComponents?keys=["bthomesensor:202"]'
  ]
  data_format = "json"
  json_string_fields = ["components_0_status_value", "components_0_config_name"]
  fieldinclude = ["components_0_status_value", "components_0_config_name"]
  tagexclude = ["url", "host"]


[[processors.starlark]]
  namepass = ["sensors"]

  source = '''
load("logging.star", "log")

def apply(metric):
  log.debug("{}".format(metric))

  name = metric.fields.get("components_0_config_name")
  status = metric.fields.get("components_0_status_value")
  parts = name.split("_", 1)

  type = parts[0]
  room = parts[1]

  result = Metric(metric.name)
  result.fields[type] = float(status)
  result.tags["room"] = room

  metric = []
  metric.append(result)

  return metric
'''

> sensors,room=kitchen humidity=59
> sensors,room=kitchen temperature=18.2
> sensors,room=sleeping humidity=51

“json_string_fields” is very important to set, as the “config_name” is a string ans “status_value” can be a boolean value.

Shelly BLU devices (windows/doors) return their state as boolean (true/false) which converts to 1/0 using “float(status)”.

If only Telegraf would support digest auth in their http plugin. :disappointed_face:

One problem solved, one ahead.