405 when trying to use the Prometheus read end-point

#1

According to this page, it should be possible to scrape metrics from Influxdb using Prometheus.

In my experience however, the remote_write end-point works, but not the remote_read.

Here’s my setup in influxdb:

> CREATE USER readuser WITH PASSWORD 'foo'
> CREATE USER writeuser WITH PASSWORD 'bar'
> CREATE DATABASE prometheus
> GRANT WRITE ON prometheus TO writeuser
> GRANT READ ON prometheus TO readuser

I’ve then successfully configured Prometheus to push metrics using:

remote_write:
  - url: "http://1.2.3.4:8086/api/v1/prom/write?u=writeuser&p=bar&db=prometheus"

I can see those writes appearing in the logs just fine.

When I try to query the database however, I get a 405 error:

$ curl -i "http://1.2.3.4:8086/api/v1/prom/read?u=readuser&p=foo&db=prometheus"
HTTP/1.1 405 Method Not Allowed
Allow: POST
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.4.3
Date: Thu, 05 Apr 2018 17:17:31 GMT
Content-Length: 19

Method Not Allowed

The corresponding stanza I tried with in the Prometheus config file was:

remote_read:
  - url: "http://1.2.3.4:8086/api/v1/prom/read?u=readuser&p=foo&db=prometheus"

Anyone got any ideas?

I also tried a POST, which makes no sense in this context IMHO:

$ curl -i -XPOST "http://1.2.3.4:8086/api/v1/prom/read?u=readuser&p=foo&db=prometheus"
HTTP/1.1 400 Bad Request
Content-Type: application/json
Request-Id: ca5ab3e2-38f7-11e8-9f76-000000000000
X-Influxdb-Build: OSS
X-Influxdb-Error: snappy: corrupt input
X-Influxdb-Version: 1.4.3
X-Request-Id: ca5ab3e2-38f7-11e8-9f76-000000000000
Date: Thu, 05 Apr 2018 17:35:53 GMT
Content-Length: 34

{"error":"snappy: corrupt input"}
#2

I’m actually seeing the same behavior with an influx instance im running in kubernetes, i can hit the prom metrics api at http://influx-instance:8086/metrics but receive method not allowed on the read endpoint (only difference is that i am hitting it without a user/pass)

#3

Yeah my instance is actually running in a local minikube cluster. Didn’t want to go into that to complicate the post further :slight_smile:

#4

It is a bit tricky to read from Influx using the Prometheus interface.
(Taken from https://prometheus.io/docs/prometheus/latest/storage/#overview):

The read and write protocols both use a snappy-compressed protocol buffer encoding over HTTP.

#5

Not sure I get your point, @saminou. I get that it could affect the curl example, but not how it would affect Prometheus.

#6

My understanding is Prometheus can only retrieve from Influx what it stored itself to Influx. And it does both using its own interface, making standard curl requests futile.

#7

Same problem with influxdb2.0.3. There was no problem with influxdb1.8 it appears after I upgrading to influxdb2.