The problem:
I’ve recently migrated hardware to a newer server (all software versions match) and I started having some strange influx errors in the log:
Feb 01 15:44:45 REDACTED influxd-systemd-start.sh[2678]: ts=2023-02-01T15:44:45.286157Z lvl=info msg=Unauthorized log_id=0fkNclAl000 error=“authorization not found”
Feb 01 15:44:51 REDACTED influxd-systemd-start.sh[2678]: ts=2023-02-01T15:44:51.063622Z lvl=info msg=“http: TLS handshake error from 127.0.0.1:48304: EOF” log_id=0fkNclAl000 service=http
Feb 01 15:44:54 REDACTED influxd-systemd-start.sh[2678]: ts=2023-02-01T15:44:54.064662Z lvl=info msg=“http: TLS handshake error from 127.0.0.1:48330: EOF” log_id=0fkNclAl000 service=http
Feb 01 15:44:54 REDACTED influxd-systemd-start.sh[2678]: ts=2023-02-01T15:44:54.079118Z lvl=info msg=“http: TLS handshake error from 127.0.0.1:48322: read tcp 127.0.1.1:8086->127.0.0.1:48322: read: connection reset by peer” log_id=0fkNclAl000 service=http
Feb 01 15:44:54 REDACTED influxd-systemd-start.sh[2678]: ts=2023-02-01T15:44:54.080827Z lvl=info msg=“http: TLS handshake error from 127.0.0.1:48342: EOF” log_id=0fkNclAl000 service=http
Feb 01 15:44:55 REDACTED influxd-systemd-start.sh[2678]: ts=2023-02-01T15:44:55.286243Z lvl=info msg=Unauthorized log_id=0fkNclAl000 error=“authorization not found”
Software versions:
I’m using:
- InfluxDB v2.6.1 (git: 9dcf880fe0) build_date: 2022-12-29T15:53:07Z
- Grafana Version 9.3.6 (commit: 978237e7cb, branch: HEAD)
- Telegraf 1.25.1 (git: HEAD@e1a0d74e)
- Ubuntu 22.04.1 LTS
I’ve enabled SSL in /etc/influxdb/config.toml using self signed certificates:
https-enabled = true
tls-cert = "/etc/ssl/influxdb-selfsigned.crt"
tls-key = "/etc/ssl/influxdb-selfsigned.key"
https-certificate = "/etc/ssl/influxdb-selfsigned.crt"
https-private-key = "/etc/ssl/influxdb-selfsigned.key"
What I’ve tried:
- I’ve double checked configurations for Telegraf and Grafana data sources and everything looks fine.
- I have some LAN clients that use the Influx Python libraries to API write information back to the InfluxDB, but none of these are erroring and data is being submitted Ok. The libraries (and the OS’s) are up to date.
- I’ve also tried regenerating Influx Tokens for everything, whilst double checking that database access is correct.
- I’ve followed this guide: https://docs.influxdata.com/influxdb/v2/admin/security/enable-tls/
And everything is as it should be, except file permissions to the certificates are 0777. - This is my TLS verification which fails for some reason:
curl -v -k https://localhost:8086/api/v2/ping
* Trying 127.0.0.1:8086...
* Connected to localhost (127.0.0.1) port 8086 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=GB; ST=Some-State; O=REDACTED; CN=REDACTED
* start date: Jan 30 15:23:23 2023 GMT
* expire date: Jan 30 15:23:23 2024 GMT
* issuer: C=GB; ST=Some-State; O=REDACTED; CN=REDACTED
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55b54f97dc60)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /api/v2/ping HTTP/2
> Host: localhost:8086
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 401
< content-type: application/json; charset=utf-8
< x-influxdb-build: OSS
< x-influxdb-version: v2.6.1
< x-platform-error-code: unauthorized
< content-length: 55
< date: Wed, 01 Feb 2023 16:04:50 GMT
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host localhost left intact
{"code":"unauthorized","message":"unauthorized access"}
Questions:
Why is my TLS verification failing?
And is there anyway I can find out more information, like which client is trying to communicate and is getting ‘not authorized’ in the logs?
