Combine measurements with different time of measurement into one sequential measurement / graph

Hi!
I have a JSON feed from a HTTP endpoint that provides measurements per 30 min. Two 30 min measurements are bundled togheter and sent every our. These measurements are in their own array element. Array element 0 is the first 30 minutes of an hour, and array element 1 is the last 30 minutes of an hour.

These array elements have their own timestamp in the JSON feed, but they have the same timestamp in Influx due to beein sent at the same time. How can I combine these two measurements into one sequential measurement?

The API: POST Forwarding of Data - Sofar API

Hi @WANders -
If you want to break out each of these measurements and use their JSON timestamps to override InFluxDB, you might try Telegraf’s [[inputs.http]] plugin with the JSON_v1 parser. This post has an example of how that’s done.

1 Like

There is no way to do this with the JSON_v2 parser?

@WANders Not at this time. That functionality was not brought forward into v2. And you’re not the first person to ask for it so I think the urgency is growing.

I get this error:

2022-07-29T09:27:30.765281508Z 2022-07-29T09:27:30Z E! [inputs.http] Error in plugin: [url=https://api.sofarocean.com/api/latest-data?spotterId=*****&token=******]: JSON time key could not be found

The plugin I have so far looks like this:

[[inputs.http]]
interval = "10s"
name_override = "Spotter"

urls = [
   "https://api.sofarocean.com/api/latest-data?spotterId=****&token=****"
]

data_format = "json"
json_time_key = "timestamp"
json_time_format = "2017-11-08T07:06:57.000Z"

Any idea what I’m doing wrong? - NVM. I Added json_query and the data is nested in data.waves.

But now I get a Time format error:

2022-07-29T11:42:21Z E! [inputs.http] Error in plugin: [url=https://api.sofarocean.com/api/latest-data?spotterId=*****&token=*****]: parsing time "2022-07-29T10:30:31.000Z": month out of range

Maybe the data format 2017-11-08T07:06:57.000Z is dificult to parse? I have no way of editing the raw data.

@WANders Check the indentation. Should be

[[inputs.http]]
  interval = "10s"
  name_override = "Spotter"

  urls = [
   "https://api.sofarocean.com/api/latest-data?spotterId=****&token=****"
  ]
etc . . .

and the timestamp’s path is waves.timestamp, right? Then try json_time_key = "waves.timestamp".

Here’s the documentation for what you want to do

and this post is very similar to your situation.

1 Like

It seems to find the time tag now, but the time format is wrong. The closest I get is “month out of range”.

With unix_ms:

2022-07-29T11:58:50Z E! [inputs.http] Error in plugin: [url=https://api.sofarocean.com/api/latest-data?spotterId=****&token=****]: strconv.ParseInt: parsing "2022-07-29T10:30:31": invalid syntax

with unix:

2022-07-29T12:00:00Z E! [inputs.http] Error in plugin: [url=https://api.sofarocean.com/api/latest-data?spotterId=****&token=****]: strconv.ParseInt: parsing "2022-07-29T10:30:31": invalid syntax

With the GO reference:

2022-07-29T12:40:11Z E! [inputs.http] Error in plugin: [url=https://api.sofarocean.com/api/latest-data?spotterId=****&token=****]: parsing time "2022-07-29T11:30:31.000Z" as "Mon Jan 2 15:04:05 MST 2006": cannot parse "2022-07-29T11:30:31.000Z" as "Mon"

With the “format” of the timestamp in the raw data:

2022-07-29T12:34:30Z E! [inputs.http] Error in plugin: [url=https://api.sofarocean.com/api/latest-data?spotterId=****&token=****]: parsing time "2022-07-29T11:30:31.000Z": month out of range

Month out of range - perhaps the data needs rearranging? But I can’t alter the raw data.

@WANders You are plowing new ground as far as my knowledge goes. The way I read the GO case is to put the reference time into whatever format your timestamp corresponds to, which would be something like this

json_time_format =  "2006-Jan-02T15:04 MST"

In other words, try your last case (month out-of-range) but use the GO reference time put into your timestamp format. (Fingers crossed. :slightly_smiling_face:)

It sort of works - some data seems to come through into InfluxDB, but I do get this error in Telegraf:


2022-08-10T06:11:35Z E! [inputs.http] Error in plugin: [url=https://api.sofarocean.com/api/latest-data?spotterId=SPOT-*****&token=***********]: Get "https://api.sofarocean.com/api/latest-data?spotterId=SPOT-*****&token=*******: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I had to add the Go reference time - but with 5 seconds as well. - That’s important for some reason.

######### JSON_v1 Parser ##########
[[inputs.http]]
  #interval = "10s"
  name_override = "Spotter-waves"
  urls = [
   "https://api.sofarocean.com/api/latest-data?spotterId=SPOT-*****&token=********************"  
  ]

data_format = "json"
json_query = "data.waves"
json_time_key = "timestamp"
json_time_format = "2006-01-02T15:04:05.000Z"