Isolate metrics of different data types in InfluxDB buckets via Telegraf

Telegraf
1

Good day,

I have the following Telegraf script that is collecting data from a MQTT broker and placing it in a Influx bucket. I want to isolate the topic via its type i.e. int. You can see an example of the script below.

[[inputs.mqtt_consumer]
servers = [“server”]
topics = [my_topics]
connection_timeout = “30s”
data_format = “json_v2”

[[inputs.mqtt_consumer.json_v2]]

[[inputs.mqtt_consumer.json_v2.tag]]
path = "x.#[type=\"Int32\"]#.y"

[[inputs.mqtt_consumer.json_v2.field]]
path = "x.#[type=\"Int32\"]#.y"

[[inputs.mqtt_consumer.json_v2]]
timestamp_path = "x.#[type=\"Int32\"]#.y"
timestamp_format = "unix"

[[outputs.influxdb_v2]]

urls = ["http://localhost:8086"]
token = "$INFLUX_TOKEN"
organization = "z"
bucket = "my_bucket"

In my influxdb bucket I am seeing all the topics as well as all the tags. However, the values I believe are being mapped incorrectly or the value of one tag is written to all the other tags.

Where am I going wrong with my script? Are there any links that will help me solve this issue? Any help or directions are much appreciated.

I hope I have been clear in my explanation. Thank you.

2

Hi,

It might help if you provide 1) an example of the JSON parsing and 2) the metric you want to produce as well.

The path option defines the location of that field, tag or timestamp you want to generate. Based on what you have above, it looks like you are creating a tag, field, and timestamp based on the same JSON item? Is that what you want?

3

Hi @jpowers

Thank you for your reply.

An example of the JSON would be {r:1, s:p, m:{type:INT32,name:a,value:b,timestamp:t}}. I want to a metric which will have the name as a tag, the value as a field and of course the associated timestamp. The type can be int, float or boolean, but I want to isolate the integers. So yes, that is indeed what I want, but unfortunately the values does not seem to be mapped correctly.

For example, the name could be apples, bananas or pears, so I would expect all the rows in the data table that belongs to the tag apples to have the associated value of apples, however the value of the apples tag seems to have values associated with the bananas name in the JSON parsing.

I hope this makes sense.

4

So let’s use some real, valid JSON:

{
    "r": 1,
    "s": "p",
    "m": {
      "type": "INT32",
      "name": "a",
      "value": "b",
      "timestamp": "1990-12-31T23:59:00Z"
    }
}

I am going to use the file input plugin, but the parser config applies to your mqtt consumer messages as well:

[[inputs.file]]
  files = ["input.json"]
  data_format = "json_v2"

  [[inputs.file.json_v2]]
  [[inputs.file.json_v2.object]]
    path = "m"
    timestamp_key = "timestamp"
    timestamp_format = "2006-01-02T15:04:03Z"

  [[inputs.file.json_v2.object.tag]]
    path = "name"
  [[inputs.file.json_v2.object.field]]
    path = "value"

Would produce:

file,host=ryzen,name=a value="b" 662605140000000000

The kicker is if your data types are changing. InfluxDB expects a data type for a field to be consistent once it is created. So you cannot change the value from a string to an integer between messages. You would want to specify the data type you want for the field and ignore others, or use a string to try to capture many types.

Does that help?

5

@jpowers

How would you isolate via datatype in this case?

6

You could start by saving everything as a string value:

  [[inputs.file.json_v2.object.field]]
    path = "value"
    type = "string"

Then you could use a processor to modify or sort that into something else.

7

Hey @jpowers

I don’t think this will work for me.

To give a much more realistic JSON:

{
“x”: {“first”: “Tom”, “last”: “Anderson”},
“age”:37,
“children”: [“Sara”,“Alex”,“Jack”],
“fav.movie”: “Deer Hunter”,
“metric”: [
{“type”: “Int32”, “name”: “metricA”, “value”: 1000, “timestamp”: 1621443400},
{“type”: “Float”, “name”: “metricB”, “value”: 2000, “timestamp”: 1621443405},
{“type”: “Int32”, “name”: “metricC”, “value”: 3000, “timestamp”: 1621443410}
]
}

The path you suggested will not work. According to GitHub - tidwall/gjson: Get JSON values quickly - JSON parser for Go I can access the path as such

path = “metric.#[type=="Int32"]#.name”
path = “metric.#[type=="Int32"]#.value”

With this I get the name of each metric that is of type integer. However, the values does not seem to be mapped correctly.

Based on the syntax, I believe “metric.#[type=="Int32"]#.value” will give me

[1000,3000]

But both metricA and metricC will have the value of 3000, instead of having 1000 and 3000, respectively.

I hope this gives a bit more clarity on the problem I’m dealing with. Thanks!

8

I don’t think this will work for me.

This is why it is important to provide an actual example :wink:

Based on the syntax, I believe “metric.#[type==“Int32”]#.value” will give me

Have you tried that on the gjson playground? I did not see the type specifier work. metric.#.name works but adding the type no longer returned anyting.

https://gjson.dev/

9

@jpowers

I played around a bit and here is a comparison

image
image1341×734 42 KB

image
image1276×684 37.2 KB

The first image, shows the edited JSON and the second image shows the default JSON. I am using the same syntax for each path, and am getting a result with the default JSON. I’m not sure why my edited JSON is returning an empty list.