Hi,
I have problem which Im not able to solve for a few days.
If I could somehow introspect or debug the request sent during the tick_script in InfluxDBOut node I would know what’s going on, but all I get in log is this:
ts=2021-01-27T16:40:44.066Z lvl=error msg="failed to write points to InfluxDB" service=kapacitor task_master=main task=munet_inactive node=influxdb_out13 err="invalid response: code 401: body: {\"code\":\"unauthorized\",\"message\":\"Unauthorized\"}"
I have instances ofr Influxdb 2.0.3, Kapacitor 1.5.7-alpine and Telegraf 1.17.0-alpine in Kubernetes cluster.
Following these guides: Use Kapacitor with InfluxDB OSS | InfluxDB OSS 2.0 Documentation
Upgrade from InfluxDB 1.x to 2.0 | InfluxDB OSS 2.0 Documentation
I tried to connect all three instances to communicate with each other. With my current set up, I can clearly send data from Telegraf to Influx, from Telegraf to Kapacitor and get data from Kapacitor vie inputs.kapacitor plugin.
The only trouble I have is when tick_scripts sends data via InfluxDBOutNode to existing bucket in Influx 2. Logs show 401 unathorized. I have read all about raised levels of security in Influx 2 and how to authorized request from Kapacitor to write there.
These are relevant part of config files from Kapacitor and Influx:
1st is kapacitor.conf in /etc/kapacitor/kapacitor.conf
hostname = "kapacitor-kapacitor"
data_dir = "/var/lib/kapacitor"
skip-config-overrides = false
default-retention-policy = "1440h"
[http]
# HTTP API Server for Kapacitor
# This server is always on,
# it serves both as a write endpoint
# and as the API endpoint for all other
# Kapacitor calls.
bind-address = ":9092"
auth-enabled = false
log-enabled = true
write-tracing = false
pprof-enabled = false
https-enabled = false
# https-certificate = "/etc/ssl/kapacitor.pem"
shutdown-timeout = "10s"
[replay]
dir = "/var/lib/kapacitor/replay"
[storage]
boltdb = "/var/lib/kapacitor/kapacitor.db"
[logging]
file = "STDERR"
level = "DEBUG"
[[influxdb]]
enabled = true
default = true
urls = ["http://influx2-influxdb.influx2-test:8086"]
username = "admin"
password = "${INFLUX_TOKEN}"
timeout = 0
# Absolute path to pem encoded CA file.
# A CA can be provided without a key/cert pair
# ssl-ca = "/etc/kapacitor/ca.pem"
# Absolutes paths to pem encoded key and cert files.
# ssl-cert = "/etc/kapacitor/cert.pem"
# ssl-key = "/etc/kapacitor/key.pem"
# Do not verify the TLS/SSL certificate.var
# This is insecure.
insecure-skip-verify = false
startup-timeout = "5m"
disable-subscriptions = true
kapacitor-hostname = ""
http-port = 0
udp-bind = ""
udp-buffer = 1000
udp-read-buffer = 0
INFLUX_TOKEN is env variable which holds base 64 decoded string attached to admin user. Admin is the initial user created during the influx setup process. This was verified multiple times on multiple occassions. The token is valid and has absolute access all over the db. I also hard copied the value of token into password field to be sure it’s correct.
2nd is Influx 2 config file stored in /root/.influxdbv2/config.toml:
bind_address: 8086
reporting_disabled: false
bolt_path: "/root/.influxdbv2/influxd.bolt"
engine_path: "/root/.influxdbv2/engine"
query_log_enabled: true
query_concurrency: 10
storage_cache_max_memory_size: 1073741824
storage_cache_snapshot_memory_size: 26214400
storage_cache_snapshot_write_cold_duration: 10m0s
storage_compact_full_write_cold_duration: 4h0m0s
storage_compact_throughput_burst: 50331648
max_select_point: 0
max_select_series: 0
max_select_buckets: 0
storage_retention_check_interval: 30m0s
storage_shard_precreator_advance_period: 30m0s
storage_shard_precreator_check_interval: 10m0s
tls_cert: "/etc/ssl/influxdb.pem"
tls_key: ""
Ignore ‘:’ symbol. I just copied over the content of values.yaml file cause I’m using Helm charts.
Running influx auth list shows the token value which is the same in Influx as in Kapacitor .conf file.
Result of influx bucket list
ID Name Retention Organization ID
2411c0548ad7ec27 _monitoring 168h0m0s 1cd6739724a9d246
7cb55a12fafeb15a _tasks 72h0m0s 1cd6739724a9d246
089960edc30e3850 kapacitor_results 0s 1cd6739724a9d246
c1c7cf76a8cdeb85 uhura_metrics 0s 1cd6739724a9d246
I also mapped buckets to databases via influx v1 dbrp create commands. Listing shows this:
06fd0fabc2f5d000 kapacitor_results 089960edc30e3850 0s true 1cd6739724a9d246
06fd42f5ac35d000 uhura_metrics c1c7cf76a8cdeb85 0s true 1cd6739724a9d246
Tick script tried to wrote to both databases. But everytime I received 401.
Relevant part of tick_script:
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
dbrp "uhura_metrics"."0s"
var outputDB = 'uhura_metrics'
var outputRP = '0s'
var outputMeasurement = 'hugoconnector_message_report'
I also tried removing the create() statement under the node and still 401.
I am able to curl Influx pod from Kapacitator pod with write post request or pretty much any other request.
The directory where the DB is have the read/write permissions for root user.
Running Linux 5.8.0-40-generic x86_64 on Ubuntu.
Kubernetes 1.20, Helm 3
If you have any idea what’s going on I would appreciate the help.
Thank you.
