Telegraf | prometheus output trouble

Due the rising power prices in germany, i want to collect data from awattar for price comparison between awattar and my local power deliver company.

I have downloaded a python script for collecting the data and due my lack of programming skill its an try and error quest.

what i have done so far

Telegraf Conf

[[inputs.exec]]
commands = ["/usr/bin/python3 /etc/telegraf/scripts/awattar-api.py"]

  timeout = "10s"
  data_format = "prometheus"

python code / source GitHub - Gransi/awattar: Python client for aWATTar API


import sys

from datetime import datetime
date= datetime.utcnow() - datetime(1970, 1, 1)
seconds =(date.total_seconds())
milliseconds = round(seconds*1000)

from awattar import AwattarClient
#unix_timestamp = time.time()  

def main(argv):

#    print ('Connect to aWATTar')
    client = AwattarClient('DE')

#    print ('Get Market data from API')
    data = client.request()
    
#    for item in data:
#        print(f'{item.start_datetime:%s} - {(item.marketprice / 1000):.4f}')

    for item in data:
        print(f'awattar-{item.start_datetime:%H} {milliseconds} {(item.marketprice / 1000):4f}')
    


    
if __name__ == "__main__":
	main(sys.argv[1:])

Script Output:

awattar-16 1669308114149 0.322720
awattar-17 1669308114149 0.299590
awattar-18 1669308114149 0.285910
awattar-19 1669308114149 0.261560
awattar-20 1669308114149 0.229940
awattar-21 1669308114149 0.217100
awattar-22 1669308114149 0.173640
awattar-23 1669308114149 0.187290
awattar-00 1669308114149 0.175190
awattar-01 1669308114149 0.163070
awattar-02 1669308114149 0.151820
awattar-03 1669308114149 0.159150
awattar-04 1669308114149 0.181640
awattar-05 1669308114149 0.230600
awattar-06 1669308114149 0.297070
awattar-07 1669308114149 0.307650
awattar-08 1669308114149 0.289850
awattar-09 1669308114149 0.280450
awattar-10 1669308114149 0.263470
awattar-11 1669308114149 0.256030
awattar-12 1669308114149 0.272430
awattar-13 1669308114149 0.290170
awattar-14 1669308114149 0.318360
awattar-15 1669308114149 0.327360

Telegraf Test:

 telegraf -config /etc/telegraf/telegraf.conf --debug -test > /tmp/test 
2022-11-24T16:42:17Z I! Starting Telegraf 1.24.3
2022-11-24T16:42:17Z I! Available plugins: 222 inputs, 9 aggregators, 26 processors, 20 parsers, 57 outputs
2022-11-24T16:42:17Z I! Loaded inputs: exec http
2022-11-24T16:42:17Z I! Loaded aggregators: 
2022-11-24T16:42:17Z I! Loaded processors: 
2022-11-24T16:42:17Z W! Outputs are not used in testing mode!
2022-11-24T16:42:17Z I! Tags enabled: host=iot-data-collector
2022-11-24T16:42:17Z D! [agent] Initializing plugins
2022-11-24T16:42:17Z D! [agent] Starting service inputs
2022-11-24T16:42:18Z E! [inputs.exec] Error in plugin: reading text format failed: text format parsing error in line 1: spurious string after timestamp: " 0.322720"
2022-11-24T16:42:18Z D! [agent] Stopping service inputs
2022-11-24T16:42:18Z D! [agent] Input channel closed
2022-11-24T16:42:18Z D! [agent] Stopped Successfully
2022-11-24T16:42:18Z E! [telegraf] Error running agent: input plugins recorded 1 errors

Where is my failure? Thanks in advance for looking / helping with my problem

I don’t think this is valid Prometheus format… Check Exposition formats | Prometheus.

One more question, why don’t you collect that data directly using the http input plugin of Telegraf and a JSON parser?

I have tryed it, a few weeks ago. but with no success. i will give it a another try.
Thats now the basis which i want to try to extract the needed data

[[inputs.http]]
  name_override = "awattar"
  urls = [
    "https://api.awattar.de/v1/marketdata"
  ]
  method = "GET"

  data_format = "json"

Currently i get this output:

root@iot-data-collector:/etc/telegraf# telegraf -config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d --test --debug -test | grep awattar
...
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_0_end_timestamp=1669392000000,data_0_marketprice=327.36,data_0_start_timestamp=1669388400000,data_10_end_timestamp=1669428000000,data_10_marketprice=199.13,data_10_start_timestamp=1669424400000,data_11_end_timestamp=1669431600000,data_11_marketprice=200,data_11_start_timestamp=1669428000000,data_12_end_timestamp=1669435200000,data_12_marketprice=197.99,data_12_start_timestamp=1669431600000,data_13_end_timestamp=1669438800000,data_13_marketprice=211.94,data_13_start_timestamp=1669435200000,data_14_end_timestamp=1669442400000,data_14_marketprice=214.94,data_14_start_timestamp=1669438800000,data_15_end_timestamp=1669446000000,data_15_marketprice=228.07,data_15_start_timestamp=1669442400000,data_16_end_timestamp=1669449600000,data_16_marketprice=264.96,data_16_start_timestamp=1669446000000,data_17_end_timestamp=1669453200000,data_17_marketprice=279.9,data_17_start_timestamp=1669449600000,data_18_end_timestamp=1669456800000,data_18_marketprice=257.83,data_18_start_timestamp=1669453200000,data_19_end_timestamp=1669460400000,data_19_marketprice=253.32,data_19_start_timestamp=1669456800000,data_1_end_timestamp=1669395600000,data_1_marketprice=349.9,data_1_start_timestamp=1669392000000,data_20_end_timestamp=1669464000000,data_20_marketprice=249.99,data_20_start_timestamp=1669460400000,data_21_end_timestamp=1669467600000,data_21_marketprice=237.41,data_21_start_timestamp=1669464000000,data_22_end_timestamp=1669471200000,data_22_marketprice=257.61,data_22_start_timestamp=1669467600000,data_23_end_timestamp=1669474800000,data_23_marketprice=264.02,data_23_start_timestamp=1669471200000,data_2_end_timestamp=1669399200000,data_2_marketprice=337.95,data_2_start_timestamp=1669395600000,data_3_end_timestamp=1669402800000,data_3_marketprice=314.37,data_3_start_timestamp=1669399200000,data_4_end_timestamp=1669406400000,data_4_marketprice=274.15,data_4_start_timestamp=1669402800000,data_5_end_timestamp=1669410000000,data_5_marketprice=266.32,data_5_start_timestamp=1669406400000,data_6_end_timestamp=1669413600000,data_6_marketprice=269.92,data_6_start_timestamp=1669410000000,data_7_end_timestamp=1669417200000,data_7_marketprice=226.03,data_7_start_timestamp=1669413600000,data_8_end_timestamp=1669420800000,data_8_marketprice=212.77,data_8_start_timestamp=1669417200000,data_9_end_timestamp=1669424400000,data_9_marketprice=210.7,data_9_start_timestamp=1669420800000 1669389774000000000
...

My guess i have to play with the settings, due the output is a big blob, which is not usable atm

Looks like now im a step ahead with my trail and error, due my lack of english …

[[inputs.http]]
  name_override = "awattar"
  urls = [
    "https://api.awattar.de/v1/marketdata"
  ]
  method = "GET"

  data_format = "json_v2"

[[inputs.http.json_v2]]

  [[inputs.http.json_v2.field]]
       path = "data"
root@iot-data-collector:/etc/telegraf# telegraf -config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d --test --debug -test  | grep awattar 
...
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669392000000,data_marketprice=327.36,data_start_timestamp=1669388400000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669395600000,data_marketprice=349.9,data_start_timestamp=1669392000000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669399200000,data_marketprice=337.95,data_start_timestamp=1669395600000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669402800000,data_marketprice=314.37,data_start_timestamp=1669399200000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669406400000,data_marketprice=274.15,data_start_timestamp=1669402800000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669410000000,data_marketprice=266.32,data_start_timestamp=1669406400000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669413600000,data_marketprice=269.92,data_start_timestamp=1669410000000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669417200000,data_marketprice=226.03,data_start_timestamp=1669413600000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669420800000,data_marketprice=212.77,data_start_timestamp=1669417200000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669424400000,data_marketprice=210.7,data_start_timestamp=1669420800000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669428000000,data_marketprice=199.13,data_start_timestamp=1669424400000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669431600000,data_marketprice=200,data_start_timestamp=1669428000000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669435200000,data_marketprice=197.99,data_start_timestamp=1669431600000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669438800000,data_marketprice=211.94,data_start_timestamp=1669435200000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669442400000,data_marketprice=214.94,data_start_timestamp=1669438800000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669446000000,data_marketprice=228.07,data_start_timestamp=1669442400000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669449600000,data_marketprice=264.96,data_start_timestamp=1669446000000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669453200000,data_marketprice=279.9,data_start_timestamp=1669449600000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669456800000,data_marketprice=257.83,data_start_timestamp=1669453200000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669460400000,data_marketprice=253.32,data_start_timestamp=1669456800000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669464000000,data_marketprice=249.99,data_start_timestamp=1669460400000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669467600000,data_marketprice=237.41,data_start_timestamp=1669464000000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669471200000,data_marketprice=257.61,data_start_timestamp=1669467600000,data_unit="Eur/MWh" 1669391647000000000
> awattar,host=iot-data-collector,url=https://api.awattar.de/v1/marketdata data_end_timestamp=1669474800000,data_marketprice=264.02,data_start_timestamp=1669471200000,data_unit="Eur/MWh" 1669391647000000000

Looks good. Maybe you can use one of the timestamps (start or end) as the metric timestamp…

I have now a another problem, the prometheus output plugin doesnt process it how i need it. it looks like only the last value wins. the value before gets overwritten.

Thats the output from /metrics

awattar_data_end_timestamp{data_unit="Eur/MWh",host="iot-data-collector",url="https://api.awattar.de/v1/marketdata"} 1.6699932e+12
awattar_data_marketprice{data_unit="Eur/MWh",host="iot-data-collector",url="https://api.awattar.de/v1/marketdata"} 288
awattar_data_start_timestamp{data_unit="Eur/MWh",host="iot-data-collector",url="https://api.awattar.de/v1/marketdata"} 1.6699896e+12

I have compered it with the output from the awattar api. 288 € / MWh is the last output currently
Any idea how to solve it?