Helm, Kapacitor and Tick scripts

#1

Hello,

I’ve deployed Telegraf, Influx, Kapacitor and Chronograf on Kubernetes using Helm.
I’ve written some Tick scripts using the Chronograf UI to generate alerts on my data.
I’m puzzled how to automate deployment of those Tick scripts.

If I were working with Kubernetes manifests, I could probably add in a config map containing my Tick scripts, however, I’m using ArgoCD to deploy the Helm charts and am constrained by what is supported in github influxdata/helm-charts/tree/master/charts/kapacitor

This seems to be what I’m looking for, but never got merged:

This also seems to do what I want, but isn’t supported via Helm (unless I can use the sidecar option to load the directory structure and pass in the [load] config as environmental variables? Its not clear what the sidecar option is for)

A similar community post gives me the idea that maybe I can log into the Kapacitor pod post deployment and load the scripts that way (user wrote a shell script, unfortunately I’ve hit my link limit)

Would really appreciate some community suggestions how to do this!
Thanks,
Tom

#2

On the sideload option, if I add a config map containing my tick script:

 % cat kapacitor-sideload-tickscript-test.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: kapacitor-sideload-tickscript-test
  labels:
    kapacitor_sideload: "1"
data:
  test.tick: |
    dbrp "telegraf"."autogen"
    var  data = batch
      |query('Select.... etc

and use the sideload option to filter for and load this:

cat values.yaml
kapacitor:
  influxURL: 'http://influxdb:8086'
  sidecar:
    sideload:
      enabled: true
      label: kapacitor_sideload

I can see my tick script in /var/lib/kapacitor/sideload inside the kapacitor pod deployed via helm. However, I still need to run kapacitor commands to add it, i.e “kapacitor define test -tick test.tick” etc.

If I could modify /etc/kapacitor/kapacitor.conf [load] section to point at /var/lib/kapacitor/sideload (or change the sideload folder to /etc/kapacitor/load if [load] is enabled by default) then maybe I could get this working, but I think I’m overusing the sideload node to get my tick scripts auto deployed?

Very confused how I can redeploy kapacitor using helm with all my tick scripts executing out of the box…

Thanks!

#3

Ok, not sure if this is how sideload is intended to be used, but it seems to work:

cat values.yaml 

kapacitor:
  influxURL: 'http://influxdb:8086'
  sidecar:
    sideload:
      enabled: true
      label: kapacitor_sideload
      folder: /var/lib/kapacitor/sideload/tasks 
  envVars:
    KAPACITOR_LOAD_ENABLED: true
    KAPACITOR_LOAD_DIR: /var/lib/kapacitor/sideload

With the tick scripts I want available as config maps, when I deloy kapacitor with helm the tick scripts/tasks are running out of the box!

1 Like
#4

You might find some helpful examples here:

#5

Did you manage to get multiple scripts to load from /sideload/tasks?

I have a ConfigMap which is writing multiple .tick files to the directory via the sidecar but Kapacitor only seems to load the first file it finds, then moves on to processing the handlers folder (which doesn’t exist).

No errors in the logs:

ts=2020-12-14T10:37:10.414Z lvl=debug msg=“loading templates” service=load
ts=2020-12-14T10:37:10.414Z lvl=debug msg=“loading tasks” service=load
ts=2020-12-14T10:37:10.414Z lvl=debug msg=“loading object from file” service=load object=task file=/var/lib/kapacitor/sideload/tasks/a_prod_error_count.tick
ts=2020-12-14T10:37:10.420Z lvl=info msg=“Starting target manager…” service=scraper
ts=2020-12-14T10:37:10.457Z lvl=debug msg=“starting task” service=kapacitor task_master=main task=a_prod_error_count
ts=2020-12-14T10:37:10.457Z lvl=info msg=“started task” service=kapacitor task_master=main task=a_prod_error_count
ts=2020-12-14T10:37:10.457Z lvl=debug msg=“listing dot” service=kapacitor task_master=main dot=“digraph a_prod_error_count {\nstream0 -> from1;\nfrom1 -> window2;\nwindow2 -> sum3;\nsum3 -> alert4;\n}”
ts=2020-12-14T10:37:10.479Z lvl=debug msg=“loading handlers” service=load

#6

Hi Philbo,
It was unreliable. The problem I had was that the tasks didn’t load at all and I put this down to a timing issue; the tasks were being posted before kapacitor was ready.

We have since moved to Influx cloud and while I don’t remember the exact details, I can outline the solution.
It looks like I added the tasks as configmap.
I then had my own kubernetes job to post the tasks.
The crux of the job is an init container to wait for kapacitor to be ready:

initContainers:
      - name: wait-for-kapacitor
        image: darthcabs/tiny-tools:1
        imagePullPolicy: IfNotPresent
        args:
        - /bin/bash
        - -c
        - >
          set -x;
          while [[ "$(curl -s -o /dev/null -w ''%{http_code}'' http://{{ .Release.Namespace }}-kapacitor/v1/tasks?fields=status)" != "200" ]]; do 
            echo '.'
            sleep 15;
          done

and then using the same image, curl each task to the api:

      containers:
      - name: kapacitor-alerts-job
        image: darthcabs/tiny-tools:1 
        imagePullPolicy: IfNotPresent
        command: ['/bin/bash']
        args: ['-c','for file in $(ls -1 files); do curl -X POST http://{{ .Release.Namespace }}-kapacitor:9092/kapacitor/v1/tasks -d @files/$file; sleep 1;done']
        volumeMounts:
        - name: kapacitor-alerts 
          mountPath: "/files"

We deploy using helm, hence the {{ .Release.Namespace }} to build the url.
Hope this helps!
Thanks,
Tom

#7

Hey Tom,

Appreciate you getting back to me.

Strange, I ended up with the same config as you posted on Aug 10, but it consistently only loads one file. I’ve looked through the Kapacitor code and nothing obvious jumped out at me. My thoughts were that it was maybe a timing issue as well.

Thanks for the extra details for the workaround!

Cheers,
Phil

#8

Github issue raised for the problem