[Solved] Kapacitor Stream tasks fail after changing subscription-mode

Store
,
1

Hello,

Recently, while debugging a problem in kapacitor, I came upon the following in the documentations.

The cluster subscription-mode is planned for future versions of Kapacitor and Kapacitor Enterprise. If used currently, subscription data will not be received.

Since the subscription-mode field in my kapacitor.conf is previously set to cluster, I changed the field to server and restarted kapacitor.

After restarting the server, the following happened:

  1. kapacitor stats ingress show that all previously functional stream tasks are no longer getting data.

    • Only entries showing up in stats ingress are measurements from _kapacitor autogen

  2. stream tasks are no longer reporting data

The following symptoms persisted even after I changed subscription-mode to cluster. As well, here are some observations:

  1. show subscriptions in Influx show that each dbrp pair has 2 subscriptions, one with the server_id of kapacitor, and one with cluster_id. When I delete either of the subscriptions, kapacitor will re-create them at the next subscription sync.

Please help, any hints/ leads for this bug will be appreciated.

System Info:

InfluxDB shell version: 1.7.1
Kapacitor OSS 1.5.1 (git: HEAD 89828ffff6cf5cd4cb2b34bf883e134395f734de)
Linux host 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

InfluxDB and Kapacitor reside on the same host machine, so I’m inclined to rule out connectivity issues.

2

Thank God, solved my issue.
For record’s sake, here’s what I noticed:

After doing a clean install of kapacitor, show subscriptions from influx reveals that the previous kapacitor is creating subscriptions to http://kapacitor:9092 instead of http://localhost:9092

(first two subscriptions are created by old kapacitor, third subscription is created by new kapacitor)

name: telegraf
retention_policy name mode destinations

autogen kapacitor-30179dfe-46d7-4cf5-aaec-bb4ce8c6c28d ANY [http://kapacitor:9092]
autogen kapacitor-284b4697-0723-4910-9f27-e88529dfd51b ANY [http://kapacitor:9092]
autogen kapacitor-8e7ed956-799f-4a68-9d91-78b609b67d67 ANY [http://localhost:9092]

Please note that I used subscription-mode="server", but the default config sets subscription-mode="cluster"