I am trying to parse these values into an influxdb bucket using telegraf so I’ve got the following input in the conf file. I’m capturing the amount as a string because that’s how it is output in the json and I know from reading the documentation that it would be ignored unless explicitly added as a string field (unless I’m mistaken)
The idea is to collect the value of the currencies and use the time obtained from the first url to set the _time in the bucket. However, no data is being written to the bucket.
I enabled debug=true for telegraf and I get the following:
> 2021-06-12T19:19:30Z E! [inputs.http] Error in plugin: [url=https://api.coinbase.com/v2/prices/ETH-USD/sell]: JSON time key could not be found
> 2021-06-12T19:19:30Z E! [inputs.http] Error in plugin: [url=https://api.coinbase.com/v2/prices/MATIC-USD/sell]: JSON time key could not be found
> 2021-06-12T19:19:30Z E! [inputs.http] Error in plugin: [url=https://api.coinbase.com/v2/prices/ETC-USD/sell]: JSON time key could not be found
> 2021-06-12T19:19:30Z E! [inputs.http] Error in plugin: [url=https://api.coinbase.com/v2/time]: JSON time key could not be found
> 2021-06-12T19:19:38Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
I would expect the sell urls to error out about the time key, but the time one should be fine.
Thanks for your help @Hipska
I changed json_time_key to “data.epoch” and “data” to no avail.
I have also commented out the json_time_key and json_time_format fields altogether but still no data is being extracted and stored. How do I debug the http input plugin? I’ve enabled debug=true in the telegraf.conf file.
Hi all,
So I’ve semi-fixed the problem by restarting the system. No idea what happened there.
Here’s the thing though, it’s only logging “data_epoch” as a field so not processing the other URLs in the list.
Am I using the plugin correctly? Is it supposed to be able to process multiple URLs in the same plugin because it looks like it’s picking the first URL and processing it but not the rest.
@Franky1 This works perfectly. Thank you very much indeed. Is it the time URL that’s confusing things then?
Is there a way to perhaps run a separate http input and use the timestamp from that input?
The problem is, that the api endpoints /time and /sell respond with different json payloads. The json parser cannot handle this in that way you want it.
Yes you could use a second http plugin just to query the /time endpoint.
But then you have to merge these two data streams from /time and /sell somehow with an additional plugin.
Thanks. I implemented a similar thing but I was wondering if there was some sort of processor that collected the outcome from both input plugins and combined them then output them into the file/influxdb bucket
@t481
As suggested by @Anaisdg, you could use an processors.execd plugin.
Here is a working solution with Python, the Telegraf config and the Python script:
from influxdb_client import Point
from line_protocol_parser import parse_line
dataset = dict()
while True:
try:
input_line = input() # read from stdin
except EOFError: # catch EOF error
break
except KeyboardInterrupt: # catch KeyboardInterrupt
break
else:
data = parse_line(input_line) # parse input line
amount = data['fields'].get('amount')
if amount:
data['fields'].update({'amount': float(amount)}) # update field type
# collect all metrics
measurement = data['measurement']
if measurement == "sell":
base = data['tags'].get('base')
dataset[base] = data
elif measurement == "time":
dataset["time"] = data
# if all metrics collected, replace timestamp and spit them out
if len(dataset) == 4:
newtime = dataset["time"].get("time")
for key, metric in dataset.items():
if key != "time":
metric['time'] = newtime # assign new time
point = Point.from_dict(metric) # metric from dict
print(point.to_line_protocol()) # write to stdout
dataset.clear() # clear set of metrics for next collection
