Kafka consumer json topics

Telegraf
,
1

Hi all,
I’m trying to write a telegraf.conf that receive two topics in json format with same structure, but different data. I’m trying to place each topic into a different bucket. But all keys/values json are been showing in the both buckets.

this is what i’m write following the documentation and topics found here:

[[inputs.kafka_consumer]]
  brokers = ["localhost:9092"]
  topics = ["topic1", "topic2"]
  data_format = "json"

[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = ""
  organization = ""
  bucket = "bucket1"
  [[outputs.influxdb_v2.tagpass]]
    topic = ["topic1"]

[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = ""
  organization = ""
  bucket = "bucket2"
  [[outputs.influxdb_v2.tagpass]]
    topic = ["topic2"]

Type of data that i’m using:
topic1: {“key_A”: int, “key_B”: int}
topic2: {“key_C”: int, “key_D”: int}

as i said, when i explore bucket1 i expect to see only
“key_A”: int and “key_B”: int
but is displayed
“key_A”: int and “key_B”: int and “key_C”: int and “key_D”: int

same for bucket2

what i’m doing wrong with this telegraf.conf???

2

You have not provided any information to telegraf as to how to parse your data. The parsers require configuration for you to tell it what values are tags or fields for example. In your case you need to specify that the topic is a tag, so that you can use it with tagpass later on.

3

| You have not provided any information to telegraf as to how to parse your data.
thats make sense. I’v been studing bore about and I found a good sintax at this topic.
fixing my code i did this:

[[inputs.kafka_consumer]]
  brokers = ["localhost:9092"]
  topics = ["topic1"]
  data_format = "json"
  [inputs.kafka_consumer.tags]
    destination = "bucket1"
  
[[inputs.kafka_consumer]]
  brokers = ["localhost:9092"]
  topics = ["topic2"]
  data_format = "json"
  [inputs.kafka_consumer.tags]
    destination = "bucket2"

[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = ""
  organization = ""
  bucket_tag = "destination"
  exclude_bucket_tag = true

but still not working.
Any tip about what is missing or mistake?

4

but still not working.

I would use the [[outputs.file]] output to ensure your metrics are looking correct and setting the tag as expected then next.

5

Hi jpowers,
I’ve taken a step back in order to explore more the documentation.
As the input i need and also output. Now i’ts clear that Kakfa input uses topic_tag and influxdb_v2 dosen’t have this field. The way is to adatp then. So I did:

[[inputs.kafka_consumer]]
  brokers = ["kafka:9092"]
  topics = ["topic1"]
  data_format = "json"
  topic_tag = ["tag1"]

[[inputs.kafka_consumer]]
  brokers = ["kafka:9092"]
  topics = ["topic2"]
  data_format = "json"
  topic_tag = ["tag2"]

[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]
  token = ""
  organization = ""
  bucket = "bucket1"
  [outputs.influxdb_v2.tagpass]
    topic_tag = ["tag1"]

[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]
  token = ""
  organization = ""
  bucket = "bucket2"
  [outputs.influxdb_v2.tagpass]
    topic_tag = ["tag2"]

But it is still join all data into both buckets. Why???

6

But it is still join all data into both buckets. Why???

Without logs showing the actual data via outputs.file I am unable to help further.

7

hi, I’m trying to use

[[outputs.file]]
  files = ["stdout", "/tmp/metrics.out"]

but nothis is been placed there. So my conclusion is that the service fails when it start.
It’s been hard to find an answer how to fix it. I think that the sintax is not good enought to the service crash fill moments after start.
So, is any of the codes before right?
can you provide me more examples how different inputs can be placed into different buckets?
or is any way to write more than one .conf file and each one do exacly what i want (content from 1 from a kafka topic going to 1 bucket)?

8

Then my suggestion is not to run it as a service, but via the CLI so you can see everything to stdout.

9

It would be so nice is there was a “perfect” example of how to do this online.
I can’t believe one of the influx staff hasn’t put this together yet.
If I knew how I would do it for you - but sadly I don’t. Hopefully someone will do it now . . . .