JSON values wrapped with quotes

Telegraf
,
1

Hello all

Have a webhook that posts data to Telegraf HTTP Listener v2; However, the JSON webhook contains quotes around numeric values, I wasn’t able to apply the processors.converter plugin to reformat or inform telegraf what to do with those values before sending to influxdb.

Here is my telegraf.conf file:

[agent]

interval = “10s”
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = “0s”
flush_interval = “10s”
flush_jitter = “0s”
precision = “”
debug = true
quiet = false
logfile = “”
hostname = “”

[[outputs.influxdb_v2]]
urls = [“http://127.0.0.1:8086”]
token = “SomeTokenGoesHere”
organization = “myorg”
bucket = “mybucket”
namedrop = [“webhook”]

[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = false
[[inputs.disk]]
ignore_fs = [“tmpfs”, “devtmpfs”, “devfs”, “overlay”, “aufs”, “squashfs”]
[[inputs.diskio]]
[[inputs.mem]]
[[inputs.net]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]

[[outputs.influxdb_v2]]
urls = [“http://127.0.0.1:8086”]
token = “AnotherTokenGoesHere”
organization = “myorg”
bucket = “monnit”
namepass = [“webhook”]

[[inputs.http_listener_v2]]
service_address = “:8080”
path = “/telegraf”
methods = [“POST”]
data_source = “body”
data_format = “json”
name_override = “webhook”

[[processors.converter]]
[processors.converter.fields]
integer = [“sensorMessages.voltage”]

Here is the JSON body of the webhook post request to Telegraf:

{
“gatewayMessage”:{
“gatewayID”:“972221” ,
“gatewayName”:“EGW-PWRGen - 972221” ,
“accountID”:“42540”,
“networkID”:“69923” ,
“messageType”:“0” ,
“power”:“0”,
“batteryLevel”: “101” ,
“date”: “2021-03-13 15:16:40”,
“count”:“1”,
“signalStrength”: “0”,
“pendingChange”: “False”

},
“sensorMessages”:[
{
“sensorID”:“730958” ,
“sensorName”:“Voltage Meter - 500 VAC - 730958”,
“applicationID”:“122”,
“networkID”:“69923”,
“dataMessageGUID”:“1544784a-4ede-4f42-8ed9-7afa42a5cd98”,
“state”: “2”,
“messageDate”: “2021-03-13 15:16:39”,
“rawData”:“0”,
“dataType”: “Voltage0To500”,
“dataValue”: “0”,
“plotValues”: “0”,
“plotLabels”: “V”,
“batteryLevel”: “100”,
“signalStrength”: “100”,
“pendingChange”: “True”,
“voltage”: “2.63”

 }

]

}

I have been trying for a couple of days to get some data through using Putty as test.
Figured out that if I manually remove the quotes on the values like voltage and signal strength, then those values go through.

If processors.converter won’t convert the values wrapped in quotes, then my only solution is to use a python script to format those JSON messages.

Any resources I can rely on? I have already looked into the Telegraf configuration page:
Telegraf Plugins

Has anyone used the HTTP listener v2 plugin and managed to fix the JSON data in a processors plugin?
Any idea please, I’d appreciate it.

2

What I notice quickly: Voltage is a float, not an integer

If that doesn’t help, you can post a sample of the influx line protocol here - without the processor plugin enabled. Add this to your config:

[[outputs.file]] # only for debugging
  namepass = ["webhook"]
  files = ["webhook.out"]
  influx_sort_fields = true

And please post the result here in markdown format:

```log
put the influx line protocol sample here
```
3

Hi @Franky1

I set the file location as /tmp/webhook.out and restarted the service.
Waited a while. Output file was completely blank.

I can see the posted data from the webhook side, but nothing lands in the output file.
Could it be the listener?

4

So, I tried to test a dummy webhook to be sent from Putty.

POST /telegraf HTTP/1.1
Host: 192.168.1.100
Version: HTTP/1.1
User-Agent: Mozilla
Accept: text/html
Content-Type: application/json
Connection: close
Content-Length: 33

{
“value1”:42,
“value2”:44
}

And the POST response was fine.

HTTP/1.1 204 No Content
Date: Sun, 14 Mar 2021 10:06:29 GMT
Connection: close

The webhook.file stored this:

webhook,host=ip-172-31-6-236 value1=42,value2=44 1615716389915366667

However, if I wrap the number fields with quotations:

POST /telegraf HTTP/1.1
Host: 192.168.1.100
Version: HTTP/1.1
User-Agent: Mozilla
Accept: text/html
Content-Type: application/json
Connection: close
Content-Length: 37

{
“value1”:“42”,
“value2”:“44”
}

POST Response remains 204:

HTTP/1.1 204 No Content
Date: Sun, 14 Mar 2021 10:10:26 GMT
Connection: close

But webhook.out has no new entries.

5

Well, with some success, managed to fix the input and processor plugins to absorb json webhooks.
However, there seems to be some parameters dropped due to pipes in the json_name_keys and json_string_fields that correspond to:

“plotValues”: “0|0|0|0|0|0|0|0|0|0|0|0|0”,
“plotLabels”: “Ah|Phase1Average|Phase1Max|Phase1Min|Phase1Duty|Phase2Average|Phase2Max|Phase2Min|Phase2Duty|Phase3Average|Phase3Max|Phase3Min|Phase3Duty”

. I didn’t know how to deal with those. Though Telegraf presents a means to an end on minor things, I think there should be a json processor rather than json data format in the input plugin. This way some ordering can be made on the parsing of the webhook post message.

Here is the latest telegraf config for those who still have issues with telegraf and http_listener_v2, you can see how I fixed it to the requirements of my webhook json jargon, and tailor it to your needs:

Configuration for telegraf agent

[agent]
interval = “10s”
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = “0s”
flush_interval = “10s”
flush_jitter = “0s”
precision = “”
debug = true
quiet = false
logfile = “”
hostname = “”

[[outputs.influxdb_v2]]
urls = [“http://127.0.0.1:8086”]
token = “YourTokenGoesHere”
organization = “myorg”
bucket = “mybucket”
namedrop = [“webhook”]

[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = false
[[inputs.disk]]
ignore_fs = [“tmpfs”, “devtmpfs”, “devfs”, “overlay”, “aufs”, “squashfs”]
[[inputs.diskio]]
[[inputs.mem]]
[[inputs.net]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]

[[outputs.influxdb_v2]]
urls = [“http://127.0.0.1:8086”]
token = “YourOtherTokenGoesHere”
organization = “myorg”
bucket = “OtherBucket”
namepass = [“webhook”]

[[inputs.http_listener_v2]]
service_address = “:8080”
path = “/telegraf”
methods = [“POST”]
data_source = “body”
data_format = “json”
tag_keys = [“sensorName”]
json_string_fields = [“plotValues”, “voltage”, “signalStrength”]
json_name_key = “plotLabels”
json_query = “sensorMessages”
name_override = “webhook”

[[processors.converter]]
namepass = [“webhook”]
[processors.converter.fields]
float = [“plotValues”, “voltage”, “signalStrength”]

[[outputs.file]] # only for debugging
namepass = [“webhook”]
files = [“/tmp/webhook.out”]
influx_sort_fields = true

1 Like
6

Just a few quick comments:

  1. measurement name
json_name_key = "plotLabels"

is useless in my opinion because it is overwritten by:

name_override = "webhook"
  1. pipe symbols

If you have pipes | in your json plotValues values, the processors.converter of course cannot convert this to float. For such unusual payloads you have to write your own processor, for example with processors.starlark or processors.regex or processors.execd

7

Hi @Franky1 , the override was useful to split the bucket loads.
I’ll look the regex up and see how that would work.