I’ve got a script which generates some personal finance metrics, which I’m storing in InfluxDB to power a Grafana dashboard. I decided to instead get Prometheus serving those metrics, so I can use PromQL.
So I’ve set up InfluxDB as a read remote for Prometheus:
remote_read:
- name: "finance"
url: "http://localhost:8086/api/v1/prom/read?db=finance"
read_recent: true
And I changed my measurements based on Prometheus endpoints support in InfluxDB | InfluxDB OSS 1.8 Documentation
> select * from finance_count_cleared
name: finance_count_cleared
time __name__ instance job value
---- -------- -------- --- -----
1577836800000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 5
1577923200000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 10
1578009600000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 12
1578182400000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 15
1578268800000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 20
1578355200000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 24
1578441600000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 28
1578528000000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 32
(etc)
(here name, job, and instance are tags, and value is the field)
But I’m not having any luck getting the metrics to show up in Prometheus.
If I make a query to Prometheus for, say, finance_count_cleared, Prometheus makes a request to InfluxDB, but as far as I can tell InfluxDB is returning an empty response.
Here’s a trace with tcpflow:
::1.60546-::1.08086: POST /api/v1/prom/read?db=finance HTTP/1.1
Host: localhost:8086
User-Agent: Prometheus/
Content-Length: 70
Accept-Encoding: snappy
Content-Encoding: snappy
Content-Type: application/x-protobuf
X-Prometheus-Remote-Read-Version: 0.1.0
C.B
A...............!..__name__..finance_count_cleared"........ ......
::1.08086-::1.60546: HTTP/1.1 200 OK
Content-Encoding: snappy
Content-Type: application/x-protobuf
Request-Id: f3bd1d43-34d5-11eb-ad5f-e0d55eac001e
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.8.3
X-Request-Id: f3bd1d43-34d5-11eb-ad5f-e0d55eac001e
Date: Wed, 02 Dec 2020 19:38:28 GMT
Content-Length: 4
..
.
Since Prometheus is making a request, it looks like the problem is on the InfluxDB end. Do I have my measurements structured incorrectly? Is there some configuration I need to do?
Thanks
