InfluxDB 1.83 not returning metrics to Prometheus remote_read

Store
1

I’ve got a script which generates some personal finance metrics, which I’m storing in InfluxDB to power a Grafana dashboard. I decided to instead get Prometheus serving those metrics, so I can use PromQL.

So I’ve set up InfluxDB as a read remote for Prometheus:

remote_read:
- name: "finance"
  url: "http://localhost:8086/api/v1/prom/read?db=finance"
  read_recent: true

And I changed my measurements based on Prometheus endpoints support in InfluxDB | InfluxDB OSS 1.8 Documentation

> select * from finance_count_cleared                                                                 
name: finance_count_cleared                                                                           
time                __name__              instance       job                 value
----                --------              --------       ---                 -----
1577836800000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 5  
1577923200000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 10 
1578009600000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 12 
1578182400000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 15 
1578268800000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 20 
1578355200000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 24 
1578441600000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 28 
1578528000000000000 finance_count_cleared localhost:8086 hledger-to-influxdb 32 
(etc)

(here name, job, and instance are tags, and value is the field)

But I’m not having any luck getting the metrics to show up in Prometheus.

If I make a query to Prometheus for, say, finance_count_cleared, Prometheus makes a request to InfluxDB, but as far as I can tell InfluxDB is returning an empty response.

Here’s a trace with tcpflow:

::1.60546-::1.08086: POST /api/v1/prom/read?db=finance HTTP/1.1
Host: localhost:8086
User-Agent: Prometheus/
Content-Length: 70
Accept-Encoding: snappy
Content-Encoding: snappy
Content-Type: application/x-protobuf
X-Prometheus-Remote-Read-Version: 0.1.0

C.B
A...............!..__name__..finance_count_cleared"........ ......                                                                                                                  
::1.08086-::1.60546: HTTP/1.1 200 OK                                                                                                                                                
Content-Encoding: snappy
Content-Type: application/x-protobuf                                                                                                                                                
Request-Id: f3bd1d43-34d5-11eb-ad5f-e0d55eac001e
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.8.3
X-Request-Id: f3bd1d43-34d5-11eb-ad5f-e0d55eac001e
Date: Wed, 02 Dec 2020 19:38:28 GMT
Content-Length: 4

..
.

Since Prometheus is making a request, it looks like the problem is on the InfluxDB end. Do I have my measurements structured incorrectly? Is there some configuration I need to do?

Thanks

2

@tim.hall can you help here please?