Json processor problem

Telegraf
1

I’ve been trying to get some json data from my electricity meter to be ingested by telegraf but I keep getting stuck. The problem is that most data floats are stored as strings. I tried solving this with processors but I keep doing something wrong. Here are my settings

telegraf.conf:

[agent]
    interval = "1s"
    debug = false
    round_interval = true
    flush_interval = "1s"
    flush_jitter = "0s"
    collection_jitter = "0s"
    metric_batch_size = 1000
    metric_buffer_limit = 10000
    quiet = false
    logfile = "/var/log/telegraf/conf.log"
    omit_hostname = true

[[outputs.influxdb]]
  urls = ["http://127.0.0.1:8086"]
  database = "telegraf"
  username = "user"
  password = "HIDDEN"
  [outputs.influxdb.tagdrop]
    influxdb_database = ["*"]

[[outputs.influxdb]]
  urls = ["http://127.0.0.1:8086"]
  database = "p1"
  username = "user"
  password = "HIDDEN"

  tagexclude = ["influxdb_database"]
  [outputs.influxdb.tagpass]
    influxdb_database = ["p1"]

p1.conf

[[inputs.http]]
urls = ["MY_URLl"]
method = "GET"
data_format = "json"
json_time_key = "Timestamp"
json_time_format = "06010215045W"
[inputs.http.tags]
influxdb_database = "p1"

[[processors.converter]]
[processors.converter.fields]
float = ["Energy_Delivered_Tariff2"]

My Json

{"Timestamp":"191228185905W","Energy_Delivered":"10470.947","Energy_Returned":0.048,"Energy_Delivered_Tariff1":"6064.464","Energy_Delivered_Tariff2":"4406.484"}

Whatever I try I can not get this to work. I tried adding json_name_keys but this resulted in data getting stored as strings which is of no use. If I set up a mock page with the output of the json without quotes around the values it does work. The problem, in my opinion, seems to be somewhere in the usage of the processor.

Any help would be greatly appreciated.

2

What’s the current output of this conf? does it return values?
can you post the result of the telegraf test command ?
(set debug = true and logfile “” in the agent settings in order to see the whole output in the console)

telegraf --config file.conf --test

My theory is that the fields are discarded before reaching the processor plugin
see: Json Parser

JSON String are ignored unless specified in the tag_key or json_string_fields options.

Try to set the following in the conf file

[[inputs.http]]
urls = ["MY_URLl"]
method = "GET"
data_format = "json"
json_time_key = "Timestamp"
json_time_format = "06010215045W"
json_string_fields = ["Energy_Delivered","Energy_Returned","Energy_Delivered_Tariff1","Energy_Delivered_Tariff2"]
[inputs.http.tags]
influxdb_database = "p1"

[[processors.converter]]
[processors.converter.fields]
float = ["Energy_Delivered","Energy_Returned","Energy_Delivered_Tariff1","Energy_Delivered_Tariff2"]
2 Likes
3

After several hours more work I’ve found out the solution. I backed up all my configs and temporarily wrote one just for this service to try and make it work. Eventually I came up with the following which works exactly how I wanted:

[agent]
    interval = "1s"
    debug = false
    round_interval = true
    flush_interval = "1s"
    flush_jitter = "0s"
    collection_jitter = "0s"
    metric_batch_size = 1000
    metric_buffer_limit = 10000
    quiet = false
    logfile = "/var/log/telegraf/conf.log"
    omit_hostname = true

[[outputs.influxdb]]
  urls = ["http://127.0.0.1:8086"]
  database = "p1"
  username = "user"
  password = "HIDDEN"

[[inputs.http]]
   urls = ["MY_URL"]
   timeout = "3s"
   data_format = "json"
   json_string_fields = ["Energy*","Power*","Gas*","Current*","Voltage*"]

[[processors.converter]]
   [processors.converter.fields]
      float = ["Energy*","Power*","Gas*","Current*","Voltage*"]

The main differences are as following:

  • Deleted the method GET
  • Deleted the timestamp field as it is not actually needed
  • Changed json_string_fields to use all data except for timestamp (note: Not the entire json output was show in my original post)
  • Used the same selector for my processors.
  • Recreate database from scratch before reloading.

After this worked I split the code back into several files and re-enabled my other configs. I added a name_override to my inputs.http and a namepass to my processors.converter so only data from this input will be processed by my processor.

While typing this I received a reply to my original post by @Giovanni_Luisotto who seems to point out the exact solution to my problem (Thanks!!). I’ll mark his post as solution as he is correct but still wrote this post with extra information for anyone else that might be struggling with the same thing in the hope it makes things more clear.

1 Like
4

Thank you very much. I was just writing a post detailing how I solved it when you replied but I did not fully understand why it worked the way it did. Your post clarifies a lot!