Converting assignments in JSON before sending to influx

Hi all

I’m trying to learn how to operate Telegraf and I am having problems understanding the many config options and how they work together as input, output, processor/aggregator.
Mainly, what and how to write the telegraf config to route input to a specific output.
(i.e. system metrics to bucket1, and http listener v2 to bucket2)

I have a webhook that will send JSON data to a telegraf/http listener v2.
However, the assigned values in the JSON data is wrapped in quotes and I think Telegraf isn’t able to directly push this data to influx before some processing which I have no idea how to read the documentation within the site and github tickets. I tried, but got confused as to why these configurations would work without a reference to why they would work at all.
If I send post data manually with JSON assignments as integers, i see the influx database updated with values.

This is a sample webhook data:
{
“gatewayMessage”:{
“gatewayID”:“972221” ,
“gatewayName”:“EGW-PWRGen - 972221” ,
“accountID”:“42540”,
“networkID”:“69923” ,
“messageType”:“0” ,
“power”:“0”,
“batteryLevel”: “101” ,
“date”: “2021-03-09 13:56:19”,
“count”:“2”,
“signalStrength”: “0”,
“pendingChange”: “False”
},
“sensorMessages”:[
{
“sensorID”:“730956” ,
“sensorName”:“Button - 730956”,
“applicationID”:“11”,
“networkID”:“69923”,
“dataMessageGUID”:“4192709f-81e3-44d1-8090-d1491d95b34a”,
“state”: “0”,
“messageDate”: “2021-03-09 13:55:49”,
“rawData”:“False”,
“dataType”: “ButtonPressed”,
“dataValue”: “False”,
“plotValues”: “0”,
“plotLabels”: “Pressed”,
“batteryLevel”: “0”,
“signalStrength”: “100”,
“pendingChange”: “True”,
“voltage”: “1.94”
},
{
“sensorID”:“731000” ,
“sensorName”:“Voltage Meter - 200 VDC - 731000”,
“applicationID”:“113”,
“networkID”:“69923”,
“dataMessageGUID”:“0e1ad617-6a06-4aa2-b25a-070074cc307d”,
“state”: “2”,
“messageDate”: “2021-03-09 13:56:18”,
“rawData”:“0”,
“dataType”: “Voltage0To5”,
“dataValue”: “0”,
“plotValues”: “0”,
“plotLabels”: “Volts”,
“batteryLevel”: “100”,
“signalStrength”: “100”,
“pendingChange”: “True”,
“voltage”: “3.02”
}
]
}

My Telegraf config has two outputs, each to a different bucket, say bucket1 and bucket2.
Bucket1 should get the system metrics like cpu, ram, disk, etc…
Bucket2 should get the webhook data that Telegraf picks up from the http listener v2 addon.

upon reading some notes on git and influx site, some people use tags = …
some use tagpass = …

I keep trying options but I feel i am so lost because none work out of the box with multiple outputs.
HTTP listener v2 should pick up the webhook json data, then convert wrapped values of “100” of the batterylevel to 100 as an integer. then send to bucket2 of influx output.

I’m using the latest releases, on a linux debian AWS instance for testing.
More importantly, I wish to learn how to solve this and future problems on my own, any thoughts on how this works and what and where to look for answers would be greatly appreciated.

This is the config for the listener:

[[inputs.http_listener_v2]]
service_address = “:8080”
path = “/telegraf”
methods = [“POST”]
data_source = “body”
data_format = “json”

I’m not sure of the tag format or how it should be applied.

tags = { influx_routing = “bucket2”}

This is the processor config:

Convert values to another metric value type

I’m also not sure if this is correct at all.

[[processors.converter]]
[processors.converter.tags]
integer = [“batteryLevel”]

This is the config for the bucket2 output:

[[outputs.influxdb_v2]]
tags = “monnitTOinfluxdb”
urls = [“http://127.0.0.1:8086”]
token = “the token generated from the portal goes here”
organization = “myorg”
bucket = “bucket2”
tagpass = { influx_routing = “bucket2”}

This is the config for the bucket1 output
[[outputs.influxdb_v2]]
tags = “testvpcmetrics”
urls = [“http://127.0.0.1:8086”]
token = “token generated from the portal goes here”
organization = “myorg”
bucket = “mybucket”
tagpass = [“testvpcmetrics”]

Somehow several questions are mixed together here?
I’m only going to deal with the splitting on the two buckets now.
Measurement filtering is the keyword:

https://docs.influxdata.com/telegraf/v1.17/administration/configuration/#measurement-filtering

Splitting to different buckets would work for example as sketched below (with the measurement name).
But it also works with tags.

[[inputs.http_listener_v2]]
  # other settings of this input plugin here
  name_override = "webhook"

[[outputs.influxdb_v2]]
  # other settings of this output plugin here
  bucket = "bucket1"
  namedrop = ["webhook"]

[[outputs.influxdb_v2]]
  # other settings of this output plugin here
  bucket = "bucket2"
  namepass = ["webhook"]

Btw, please post your Telegraf config snippets in Markdown format here in the future:

```toml
put the config code snippets here
```
1 Like

Thanks @Franky1 that worked like a charm. I can’t believe it was that simple.
I’ve been really trying hard to crack my head against this.
I’m curious as to why the brackets only go around namedrop and namepass assignments but not name_override.
Also, if I can impose again, what’s the difference between tags and name_override in this scenario with their complementary tagpass and namepass? Would have tags worked the same as name_override?

I have not tested the example below, but this is how it should work in principle with tags. Depending on what you want to achieve, you can also filter much more granularly with measurement filtering.

[[inputs.http_listener_v2]]
  # other settings of this input plugin here
  [inputs.http_listener_v2.tags]
    tag1 = "foo"  # add custom tag

[[outputs.influxdb_v2]]
  # other settings of this output plugin here
  bucket = "bucket1"
  [outputs.influxdb_v2.tagdrop] # exclude
    tag1 = ["foo"]

[[outputs.influxdb_v2]]
  # other settings of this output plugin here
  bucket = "bucket2"
  [outputs.influxdb_v2.tagpass] # include
    tag1 = ["foo"]
1 Like

Thanks @Franky1 I can better understand that link now with those examples. I also have a better appreciation of Telegraf due to your effort. Thank you again.

How are the tags passed on the http string, e.g. via curl?

@wicadmin
https://docs.influxdata.com/influxdb/v2.0/write-data/developer-tools/api/

1 Like