Parsing strange JSON payload

1

Hi,
I need to import a quiet strange mqtt JSON payload, generated by a wallbox.
It looks like
…/meters/0/value_ids = [1,2,3,13,17,21,39]
…/meters/0/values = [241.11,242.22,239.33,0,0,5.39,0]
with the left part before the equal sign being the mqtt topic and the right part after being the value.

Now, it needs to be imported into influxDB, with an import format mapping the “value_ids” and the corresponding “value” into seperate import lines, like
measurement,topic=…/meters/0/1 value=241.11
measurement,topic=…/meters/0/2 value=242.22
measurement,topic=…/meters/0/3 value=239.33
measurement,topic=…/meters/0/13 value=0
measurement,topic=…/meters/0/17 value=0
measurement,topic=…/meters/0/21 value=5.39
measurement,topic=…/meters/0/39 value=0

I’m at a complete loss, how to tackle the problem, even how to start, despite I have an already working telegraf config to import more conventional mqtt data.

Any hint ist appreciated.
Cheers.

2

Hello @universal-dilettant,
I’d suggest using the execd processing plugin to parse with a python script.
Something like:

import json

def process_mqtt_payload(payload):
    data = json.loads(payload)
    value_ids = data.get("value_ids", [])
    values = data.get("values", [])
    
    for vid, value in zip(value_ids, values):
        topic = f"…/meters/0/{vid}"
        print(f"measurement,topic={topic} value={value}")

# Example input
mqtt_payload = '{"value_ids": [1,2,3,13,17,21,39], "values": [241.11,242.22,239.33,0,0,5.39,0]}'
process_mqtt_payload(mqtt_payload)

Or node-red for preprocessing.

Or starlark I think you might be able to use starlark for that.
Something like

def apply(metric):
    value_ids = metric.fields["value_ids"]
    values = metric.fields["values"]
    
    for i in range(len(value_ids)):
        new_metric = metric.clone()
        new_metric.tags["topic"] = metric.tags["topic"] + "/" + str(value_ids[i])
        new_metric.fields["value"] = values[i]
        yield new_metric

@srebhan might have a more clever idea though.

3

Hello @Anaisdg,
I guess the difficulty with starlark might be that the 2 semantically connected mqtt topics are temporally independant from each other.
value_ids does usually appear once after the source (a wallbox) has rebooted and connected to the mqtt broker (eclipse), while values is updated every other second. As such I need to keep the state = value_ids for the whole runtime of the process. IMHO that speaks for execd - despite I’m a bit reluctant to give up so easily on the “standard” features of telegraf.
@srebhan: you’ve got another approach beside execd or a completely independant preprocessor like node-red et.al.??

Thx in adv

4

@universal-dilettant if you can ensure that the IDs are available you can use starlark and “store” the information in state like in the starplug example. This state will be kept for the runtime of Telegraf and you can then look-up the information for every metric…