Telegraf http plugin with json file trouble

Telegraf
21

Yes, goal is to have the timestamp from the JSON “time-tag” otherwise it won’t be of much use if every line has the same timestamp. It will end up as a single DB entry in InfluxDB with the content of the last entry in the JSON.

Telegraf automatically adds the current time to each data record it sends to InfluxDB at runtime. This is not the intended behavior for my use case. Every single data record needs to have its unique time preserved from the source JSON.

22

@HB9VQQ

If you add this to your conf file

[[processors.starlark]]
  ## Source of the Starlark script.
  source = '''

def apply(metric):

  for k, v in metric.fields.items():
    if k == "time-tag":
      metric.fields["time-tag"] = metric.fields["time-tag"] + "-01"
  return metric
'''

then the resulting output is

SWPC time-tag="2040-04-01",predicted_ssn=0.1,high_ssn=10.1,low_ssn=0,predicted_f10.7=67.8,high_f10.7=76.8,low_f10.7=67.7 1658150680000000000
SWPC time-tag="2040-05-01",predicted_ssn=0.1,high_ssn=10.1,low_ssn=0,predicted_f10.7=67.8,high_f10.7=76.8,low_f10.7=67.7 1658150680000000000
SWPC low_f10.7=67.7,time-tag="2040-06-01",predicted_ssn=0.1,high_ssn=10.1,low_ssn=0,predicted_f10.7=67.8,high_f10.7=76.8 1658150680000000000
SWPC predicted_ssn=0,high_ssn=9,low_ssn=0,predicted_f10.7=67.73,high_f10.7=8,low_f10.7=67.7,time-tag="2040-07-01" 1658150680000000000
SWPC time-tag="2040-08-01",predicted_ssn=0,high_ssn=9,low_ssn=0,predicted_f10.7=67.73,high_f10.7=8,low_f10.7=67.7 1658150680000000000
SWPC low_ssn=0,predicted_f10.7=67.73,high_f10.7=8,low_f10.7=67.7,time-tag="2040-09-01",predicted_ssn=0,high_ssn=9 1658150680000000000
SWPC predicted_ssn=0,high_ssn=9,low_ssn=0,predicted_f10.7=67.73,high_f10.7=8,low_f10.7=67.7,time-tag="2040-10-01" 1658150680000000000
SWPC high_f10.7=8,low_f10.7=67.7,time-tag="2040-11-01",predicted_ssn=0,high_ssn=9,low_ssn=0,predicted_f10.7=67.73 1658150680000000000
SWPC low_f10.7=67.7,time-tag="2040-12-01",predicted_ssn=0,high_ssn=9,low_ssn=0,predicted_f10.7=67.73,high_f10.7=8 1658150680000000000

You could filter on time-tag in InfluxDB or, with a little more code in the processor plug-in, you could replace the timestamp with an RFC3339 version of time-tag.

23

Only the last line of the output will be stored in the InfluxDB → every line has the same timestamp (1658150680000000000), last one wins.

In the older JSON (V1) plugin there was a setting for “json_time_key” which appears to do exactly what I am looking for.

24

I have it working by using the old JSON (V1) plugin thanks to the “json_time_key”

[[inputs.file]]
  ## Files to parse each interval.  Accept standard unix glob matching rules,
  ## as well as ** to match recursive files and directories.
  files = ["./predicted-solar-cycle.json"]
  data_format = "json"
  name_override = "SWPC_predicted"
  json_time_key = "time-tag"
  json_time_format = "2006-01-02"

JSON (edited) source file. Appended “-01” manually

[{
        "time-tag": "2020-01-01",
        "predicted_ssn": 0.0,
        "high_ssn": 9.0,
        "low_ssn": 0.0,
        "predicted_f10.7": 69,
        "high_f10.7": 8.0,
        "low_f10.7": 67.7
    }, {
        "time-tag": "2020-02-01",
        "predicted_ssn": 0.0,
        "high_ssn": 9.0,
        "low_ssn": 0.0,
        "predicted_f10.7": 100,
        "high_f10.7": 8.0,
        "low_f10.7": 67.7
    }
]

Output to InfluxDB

SWPC_predicted high_f10.7=8,high_ssn=9,low_f10.7=67.7,low_ssn=0,predicted_f10.7=69,predicted_ssn=0 1577836800000000000
SWPC_predicted high_f10.7=8,high_ssn=9,low_f10.7=67.7,low_ssn=0,predicted_f10.7=100,predicted_ssn=0 1580515200000000000

Pls note the timestamps at the end of both lines

How it looks in InfluxDB

image
image1237×1054 90.7 KB

It would be great if the newer json_v2 plugin would also provide the “json_time_key”

25

@HB9VQQ Great detective work!! Glad you got it figured out and thanks for letting me know the solution.

1 Like