Kapacitor fail to write to Influx 2.0 instance with tick_script InfluxDBOutNode - 401

Hi,
I have problem which Im not able to solve for a few days.
If I could somehow introspect or debug the request sent during the tick_script in InfluxDBOut node I would know what’s going on, but all I get in log is this:

ts=2021-01-27T16:40:44.066Z lvl=error msg="failed to write points to InfluxDB" service=kapacitor task_master=main task=munet_inactive node=influxdb_out13 err="invalid response: code 401: body: {\"code\":\"unauthorized\",\"message\":\"Unauthorized\"}"

I have instances ofr Influxdb 2.0.3, Kapacitor 1.5.7-alpine and Telegraf 1.17.0-alpine in Kubernetes cluster.
Following these guides: Use Kapacitor with InfluxDB OSS | InfluxDB OSS 2.0 Documentation
Upgrade from InfluxDB 1.x to 2.0 | InfluxDB OSS 2.0 Documentation

I tried to connect all three instances to communicate with each other. With my current set up, I can clearly send data from Telegraf to Influx, from Telegraf to Kapacitor and get data from Kapacitor vie inputs.kapacitor plugin.

The only trouble I have is when tick_scripts sends data via InfluxDBOutNode to existing bucket in Influx 2. Logs show 401 unathorized. I have read all about raised levels of security in Influx 2 and how to authorized request from Kapacitor to write there.

These are relevant part of config files from Kapacitor and Influx:
1st is kapacitor.conf in /etc/kapacitor/kapacitor.conf

    hostname = "kapacitor-kapacitor"
    data_dir = "/var/lib/kapacitor"
    skip-config-overrides = false
    default-retention-policy = "1440h"

    [http]
      # HTTP API Server for Kapacitor
      # This server is always on,
      # it serves both as a write endpoint
      # and as the API endpoint for all other
      # Kapacitor calls.
      bind-address = ":9092"
      auth-enabled = false
      log-enabled = true
      write-tracing = false
      pprof-enabled = false
      https-enabled = false
      # https-certificate = "/etc/ssl/kapacitor.pem"
      shutdown-timeout = "10s"

    [replay]
      dir = "/var/lib/kapacitor/replay"

    [storage]
      boltdb = "/var/lib/kapacitor/kapacitor.db"

    [logging]
      file = "STDERR"
      level = "DEBUG"

    [[influxdb]]
      enabled = true
      default = true
      urls = ["http://influx2-influxdb.influx2-test:8086"]
      username = "admin"
      password = "${INFLUX_TOKEN}"
      timeout = 0
      # Absolute path to pem encoded CA file.
      # A CA can be provided without a key/cert pair
      #   ssl-ca = "/etc/kapacitor/ca.pem"
      # Absolutes paths to pem encoded key and cert files.
      #   ssl-cert = "/etc/kapacitor/cert.pem"
      #   ssl-key = "/etc/kapacitor/key.pem"

      # Do not verify the TLS/SSL certificate.var
      # This is insecure.
      insecure-skip-verify = false
      startup-timeout = "5m"
      disable-subscriptions = true
      kapacitor-hostname = ""
      http-port = 0
      udp-bind = ""
      udp-buffer = 1000
      udp-read-buffer = 0

INFLUX_TOKEN is env variable which holds base 64 decoded string attached to admin user. Admin is the initial user created during the influx setup process. This was verified multiple times on multiple occassions. The token is valid and has absolute access all over the db. I also hard copied the value of token into password field to be sure it’s correct.

2nd is Influx 2 config file stored in /root/.influxdbv2/config.toml:

  bind_address: 8086
  reporting_disabled: false
  bolt_path: "/root/.influxdbv2/influxd.bolt"
  engine_path: "/root/.influxdbv2/engine"
  query_log_enabled: true
  query_concurrency: 10
  storage_cache_max_memory_size: 1073741824
  storage_cache_snapshot_memory_size: 26214400
  storage_cache_snapshot_write_cold_duration: 10m0s
  storage_compact_full_write_cold_duration: 4h0m0s
  storage_compact_throughput_burst: 50331648
  max_select_point: 0
  max_select_series: 0
  max_select_buckets: 0
  storage_retention_check_interval: 30m0s
  storage_shard_precreator_advance_period: 30m0s
  storage_shard_precreator_check_interval: 10m0s
  tls_cert: "/etc/ssl/influxdb.pem"
  tls_key: ""

Ignore ‘:’ symbol. I just copied over the content of values.yaml file cause I’m using Helm charts.
Running influx auth list shows the token value which is the same in Influx as in Kapacitor .conf file.

Result of influx bucket list

ID			Name			Retention	Organization ID
2411c0548ad7ec27	_monitoring		168h0m0s	1cd6739724a9d246
7cb55a12fafeb15a	_tasks			72h0m0s		1cd6739724a9d246
089960edc30e3850	kapacitor_results	0s		1cd6739724a9d246
c1c7cf76a8cdeb85	uhura_metrics		0s		1cd6739724a9d246

I also mapped buckets to databases via influx v1 dbrp create commands. Listing shows this:

06fd0fabc2f5d000	kapacitor_results	089960edc30e3850	0s			true	1cd6739724a9d246
06fd42f5ac35d000	uhura_metrics		c1c7cf76a8cdeb85	0s			true	1cd6739724a9d246

Tick script tried to wrote to both databases. But everytime I received 401.
Relevant part of tick_script:

|influxDBOut()
            .create()
            .database(outputDB)
            .retentionPolicy(outputRP)
            .measurement(outputMeasurement)
dbrp "uhura_metrics"."0s"

var outputDB = 'uhura_metrics'

var outputRP = '0s'

var outputMeasurement = 'hugoconnector_message_report'

I also tried removing the create() statement under the node and still 401.
I am able to curl Influx pod from Kapacitator pod with write post request or pretty much any other request.

The directory where the DB is have the read/write permissions for root user.
Running Linux 5.8.0-40-generic x86_64 on Ubuntu.
Kubernetes 1.20, Helm 3

If you have any idea what’s going on I would appreciate the help.

Thank you.

Were you able to fix this issue? I’m getting the same exact one at the moment, with a similar setup.

Hello @rafaelnogueira13 and @ondrej-ivanko,
Are you able to successfully test the compatibility with

curl --request GET http://example.com:8080/query?db=example-db \
  --header "Authorization: Token YourAuthToken" \
  --data-urlencode "q=SELECT used_percent FROM example-db.example-rp.example-measurement WHERE host=host1"

Hi @Anaisdg,

I tried to send request with these values:

curl --request GET http://uhura-office-testinflux-influxdb2:80/query?bucket=uhura_metrics -H "Authorization: <my decoded base64 string>" --data-urlencode "q=SELECT used_percent FROM uh
ura_metrics.0s.cpu WHERE host=uhura-office-testinflux-telegraf"

I received empty dictionary in response {}
This is verbose response from curl:

*   Trying 10.233.1.236:80...
* Connected to uhura-office-testinflux-influxdb2 (10.233.1.236) port 80 (#0)
> GET /query?bucket=uhura_metrics HTTP/1.1
> Host: uhura-office-testinflux-influxdb2
> User-Agent: curl/7.69.1
> Accept: */*
> Authorization: Token <my decoded base64 string>
> Content-Length: 103
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 103 out of 103 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 09 Feb 2021 15:21:38 GMT
< Content-Length: 3
< Content-Type: text/plain; charset=utf-8
< 
{}
* Connection #0 to host uhura-office-testinflux-influxdb2 left intact

To clarify things. The url of the curl request is build from DNS address:port of my kubernetes service serving as proxy for influx 2 pod.
bucket param in query is the name of current bucket/db in my Influx 2.x instance.
In query the FROM statement is built from bucket_name.retention_policy_of_that_bucket.cpu
host is the hostname of telegraf instance in kubernetes.

I could not emulate the query from the curl request in Influx 2.x GUI, because I couldn’t find used_percent and cpu fields.

Hope this helps with resolution to our predicament.

Hello @ondrej-ivanko,

Thank you for the additional info, I’ve submitted your question to our kapacitor expert. Please @ me if he doesn’t get back to you by the end of the week. Thank you for your patience in advance.

Thank you

      username = "admin"
      password = "${INFLUX_TOKEN}"

This will not work. You need to use the influx v1 auth create command to create a v1 user with the permissions you want (such as being able to read/write to which buckets, etc)

2 Likes

Hi @Emrys_Landivar that actually worked. Thank you so much.

Hi @Anaisdg @Emrys_Landivar ,

thank you both for help. I don’t think I’d figure that out myself.

I think one of use should create an issue on github where documentation for influx resides, as this critical part of information regarding v1 auth create is missing from there. It’s mentioning tokens, but it seems to be mentioning tokens applicable for Influx v2, not for v1 when it comes to using Kapacitor.

Would you create this issue, or should I? Thank you for answer.

Hi @rafaelnogueira13, this is working solution. At least for me.

@ondrej-ivanko fyi I just found the following issue that mentions this thread: Misc Kapacitor · Issue #2221 · influxdata/docs-v2 · GitHub