Telegraf direct to kapacitor

#1

I’m trying to route a telegraf output directly to kapacitor so that I can massage and filter the data prior to sending it to influx.

I’ve read this older post on the topic https://community.influxdata.com/t/telegraf-kapacitor-influxdb-possible/29/2 but I’m unclear on what to do in my kapacitor script to get the data from telegraf instead of from influx.

My telegraf.conf contains the following:

[[outputs.influxdb]]
  urls = ["udp://127.0.0.1:9092"]
  database = "stage_default"
  retention_policy = "weekly"
  namepass = ["ap_util"]

(If I point my url to my influxdb it logs the data in influx successfully).

In my kapacitor script I want something like:

dbrp "????"."????"   <-- not sure what to put here?

stream
  | from()
    .measurement('ap_util')
  | log()   <-- currently just for testing that I'm receiving the data at all
0 Likes

#2

I haven’t tried it, but maybe you just need to put the values from the Telegraf output?

dbrp "stage_default"."weekly"
0 Likes

#3

I’ve tried that and it just connects to the influx database and retention policy if they exist. And if they don’t exist kapacitor logs an error complaining they don’t exist.

EDIT: I tried that with a batch instead of a stream, and got that result. With stream nothing gets logged to indicate neither problem nor success.

0 Likes

#4

Some additional google searching and I found this blog on the subject. Still to no avail though. I tried putting the dbrp directly in the script as per v1.4. I also tried removing it from the script and specifying it on the command line to define the script.

Going to try going back to kapacitor 1.3 (when the blog was written) to see if that makes a difference.

0 Likes

#5

Hi, I am having the same requirement of Telegraf -> Capacitor.
** HTTP link removed due to new user limitation.

[telegraf.conf]
[[outputs.influxdb]]
urls = ["[httplink]:9092"]
database = “telegraf”
retention_policy = “autogen”

[kapacitor.conf]
[[influxdb]]
enabled = true
default = true
name = “kapacitor”
urls = ["[httplink]:9092"]
username = “”
password = “”
timeout = 0

The above configuration does not seem to work because the Kapacitor logs =>
ts=2019-04-03T14:42:15.642+08:00 lvl=error msg=“failed to connect to InfluxDB, retrying…” service=influxdb cluster=kapacitor err=“Get [httplink]:9092/ping: dial tcp 192.168.0.77:9092: connect: connection refused”
ts=2019-04-03T14:42:16.441+08:00 lvl=error msg=“failed to connect to InfluxDB, retrying…” service=influxdb cluster=kapacitor err=“Get [httplink]:9092/ping: dial tcp 192.168.0.77:9092: connect: connection refused”

Did anyone get Telegraf -> Kapacitor to work properly?

[root@xxx kapacitor]# telegraf -version
Telegraf 1.10.1 (git: HEAD a6778f46)

[root@xxx kapacitor]# rpm -qa | grep kapacitor
kapacitor-1.5.2-1.x86_64

0 Likes

#6

Hi Daniel ,

to get Telegraf -> Kapacitor to work ,
you only need the telegraf configuration you showed ,
then ( assuming the inputs.cpu is present ) you can create a tick script , to test , as follows

dbrp “telegraf”.“autogen”
stream
|from()
.database(‘telegraf’)
.measurement(‘cpu’)
|log()

the error you get is because kapacitor.conf should use port 8086 best regards

0 Likes

#7

Hi @MarcV
Thanks for your reply.

Are you saying that the configuration should look like this?

[telegraf.conf]
[[outputs.influxdb]]
urls = ["[httplink]:9092"]
database = “telegraf”
retention_policy = “autogen”

[kapacitor.conf]
[[influxdb]]
enabled = true
default = true
name = “kapacitor”
urls = ["[httplink]:8086"]
username = “”
password = “”
timeout = 0

What I intended to achieve is to get Telegraf to send data directly to Kapacitor without installing InfluxDB.

Thanks again!

0 Likes

#8

Hi Daniel ,
almost :slight_smile: here is an example … ( ps : I have stopped the influx database before doing this test)

in /etc/telegraf/telegraf.conf I have only the following :

[[inputs.cpu]]

[[outputs.influxdb]]
urls = [“http://localhost:9092”]
database = “telegraf”
retention_policy = “autogen”

my tick script contains :

dbrp “telegraf”.“autogen”
stream
|from()
.database(‘telegraf’)
.measurement(‘cpu’)
|log()

and in /etc/kapacitor/kapacitor.conf I have set

enabled = false

in the [[influxdb]] section .
The rest of the kapacitor.conf file I have not changed.

kapacitor watch teka shows that the data is flowing through kapacitor

ts=2019-04-04T11:37:50.001+02:00 lvl=info msg=point service=kapacitor task_master=main task=teka node=log2 prefix= name=cpu db=telegraf rp=autogen group= tag_cpu=cpu0 tag_host=myhost field_usage_irq=0 field_usage_guest=0 field_usage_idle=99.19919919847132 field_usage_softirq=0 field_usage_user=0.30030030029558424 field_usage_system=0.20020020019553883 field_usage_nice=0 field_usage_iowait=0.20020020020009086 field_usage_steal=0.10010010009776941 field_usage_guest_nice=0 time=2019-04-04T09:37:40Z

you can use telegraf as your database name but be aware that it is the default name
for the database created by telegraf if telegraf connects to influxdb.

0 Likes

#9

Hi @MarcV
Thanks so much for your tips. I made good progress!

I configured Telegraf to use outputs.http to send data over to Kapacitor which is listening on port 9092 =>

[[outputs.http]]

## URL is the address to send metrics to

url = “[httplink]:9092”
method = “POST”
data_format = “influx”

However, it seems code 404 was returned.

[telegraf log]
2019-04-04T10:11:09Z D! [outputs.http] buffer fullness: 285 / 10000 metrics.
2019-04-04T10:11:09Z E! [agent] Error writing to output [http]: when writing to [httplink:9092] received status code: 404

[kapacitor log]
ts=2019-04-04T18:11:37.002+08:00 lvl=info msg=“http request” service=http host=192.168.0.77 username=- start=2019-04-04T18:11:37.001953675+08:00 method=POST uri=/ protocol=HTTP/1.1 status=404 referer=- user-agent=Telegraf/1.10.1 request-id=0809ded8-56c2-11e9-8066-000000000000 duration=193.443µs
ts=2019-04-04T18:11:44.004+08:00 lvl=info msg=“http request” service=http host=192.168.0.77 username=- start=2019-04-04T18:11:44.003624904+08:00 method=POST uri=/ protocol=HTTP/1.1 status=404 referer=- user-agent=Telegraf/1.10.1 request-id=0c363daf-56c2-11e9-8067-000000000000 duration=1.039467ms

Did I miss anything? Thanks again.

0 Likes

#10

Hi Daniel ,
you didn’t miss anything , you were supposed to miss the code 404 :slight_smile:
because you can ignore it …
I have also the 404 but that is expected as explained in the blog ,

Apr 04 11:20:26 myhost telegraf[8454]: 2019-04-04T09:20:26Z W! [outputs.influxdb] when writing to [http://localhost:9092]: database “telegraf” creation failed: 404 Not Found

from the blog a bit higher in this thread :

The database parameter should point to a non-existent database (you can ignore Telegraf’s warning about the database not being found).

0 Likes

#11

Hi @MarcV
Thanks so much. It’s working now!!

1 Like

#12

Hi @MarcV
By the way, is the Telegraf->Kapacitor setup only limited to “stream” mode and not “batch” mode?

Thanks.

0 Likes

#13

Hi @danielyeap,
Looks like my answer is gone :slight_smile:

0 Likes