[[inputs.kafka_consumer]] problem "decoding" avro-formated data

Telegraf
, ,
1

Hello,

I am trying to collect data from Kafka (from a SAN-Switch) using telegrafs [[inputs.kafka_consumer]].
The Kafka-producer uses the avro-format when putting the data into Kafka (which I can not change).

Here is the relevant part of telegraf.conf

[[inputs.kafka_consumer]]
  brokers = ["<hostname>:9093"]
  topics = ["perfmon_metrics_switch"]
  version = "7.3.3"
  insecure_skip_verify = true
  max_message_len = 1000000
  data_format = "avro"
  avro_schema = '''
        {
                "namespace": "com.brocade.streaming",
                "name": "fibrechannel_switch_statistics",
                "type": "record",
                "version": "1",
                "fields": [
                        {"name": "switch_wwn", "type": "string", "doc": "WWN of the Physical Switch."},
                        {"name": "statistics_collection_time", "type": "long", "doc": "Epoch time when statistics is collected."},
                        {"name": "up_time", "type": "long", "doc": "Switch Up Time (in hundredths of a second)"},
                        {"name": "cpu_utilization", "type": "float", "default": 0, "doc": "CPU Utilization in %"},
                        {"name": "memory_utilization", "type": "float", "default": 0, "doc": "Memory Utilization in %"}
                ]
        }
  '''
  avro_measurement = "Switch"
  avro_tags = ["switch_wwn"]
  avro_fields = ["up_time", "cpu_utilization", "memory_utilization"]
  avro_timestamp = "statistics_collection_time"
  avro_timestamp_format = "unix_ms"

[[outputs.file]]
  files = ["stdout", "/tmp/metrics.out"]
  data_format = "influx"

The “raw” data read with kafka-avro-console-consumer looks like this:

{"switch_wwn":"10:00:50:EB:1A:0B:84:3A","statistics_collection_time":1682509200092,"up_time":1166984904,"cpu_utilization":14.0,"memory_utilization":20.0}
{"switch_wwn":"10:00:D8:1F:CC:5B:3E:77","statistics_collection_time":1682509200224,"up_time":164446104,"cpu_utilization":4.0,"memory_utilization":10.0}

I am expecting to get the following output:

Switch,switch_wwn=10:00:50:EB:1A:0B:84:3A up_time=1166984904,cpu_utilization=14.0,memory_utilization=20.0 1682509200092000
Switch,switch_wwn=10:00:D8:1F:CC:5B:3E:77 up_time=164446104,cpu_utilization=4.0,memory_utilization=10.0 1682509200224000

But, what I get is something like the following:

Switch cpu_utilization=0.0000000025322037799924146,memory_utilization=0.0000000006411111641568823,statistics_collection_time=0i,up_time=0i 0
Switch memory_utilization=0.0000000006411111641568823,statistics_collection_time=0i,up_time=0i,cpu_utilization=0.0000000025322037799924146 0
Switch up_time=0i,cpu_utilization=0.0000000025322037799924146,memory_utilization=0.0000000006411111641568823,statistics_collection_time=0i 0
Switch statistics_collection_time=0i,up_time=0i,cpu_utilization=0.0000000025322037799924146,memory_utilization=0.0000000006411111641568823 0
Switch up_time=0i,cpu_utilization=0.0000000025322037799924146,memory_utilization=0.0000000006411111641568823,statistics_collection_time=0i 0
Switch statistics_collection_time=0i,up_time=0i,cpu_utilization=0.0000000025322037799924146,memory_utilization=0.0000000006411111641568823 0

Some version information:

OS: AlmaLinux release 8.7 (Stone Smilodon)
Telegraf: 1.26.1 (git: HEAD@8f421f25)
Kafka: Confluent Kafka 7.3.3-ccs (Commit:37ce202868a57983d7f28fdeb51e9147ba8e3f95)

Any Idea what I am doing wrong and what I can do to correct it?

Regards,
Dan

2

@dan-m-joh is there a way for you to extract the raw binary message data so I can reproduce this issue on my side?

Oh and please file an issue!

3

I can at least try…

kafka.base64.txt is the “raw” output from kafka-console-consumer --topic perfmon_metrics_switch | tee kafka.raw; base64 kafka.raw > kafka.base64.txt and kafka.txt is the output from kafka-avro-console-consumer for the same data.

As I can not upload attachments as a new user I paste it instead…
kafak.txt

{"switch_wwn":"10:00:50:EB:1A:0B:84:3A","statistics_collection_time":1682595600101,"up_time":1175624904,"cpu_utilization":14.0,"memory_utilization":20.0}
{"switch_wwn":"10:00:D8:1F:CC:5B:3E:77","statistics_collection_time":1682595600239,"up_time":173086104,"cpu_utilization":5.0,"memory_utilization":10.0}
{"switch_wwn":"10:00:C4:F5:7C:B7:FE:9A","statistics_collection_time":1682595600479,"up_time":1175738304,"cpu_utilization":13.0,"memory_utilization":20.0}
{"switch_wwn":"10:00:D8:1F:CC:59:2F:0F","statistics_collection_time":1682595600622,"up_time":78017204,"cpu_utilization":5.0,"memory_utilization":10.0}
{"switch_wwn":"10:00:D8:1F:CC:59:62:FF","statistics_collection_time":1682595600366,"up_time":69673904,"cpu_utilization":5.0,"memory_utilization":10.0}
{"switch_wwn":"10:00:C4:F5:7C:C2:AF:42","statistics_collection_time":1682595600723,"up_time":1235864804,"cpu_utilization":13.0,"memory_utilization":20.0}
{"switch_wwn":"10:00:50:EB:1A:0B:85:3A","statistics_collection_time":1682595600867,"up_time":1235745304,"cpu_utilization":13.0,"memory_utilization":20.0}
{"switch_wwn":"10:00:D8:1F:CC:5B:3D:A7","statistics_collection_time":1682595601000,"up_time":173218804,"cpu_utilization":5.0,"memory_utilization":11.0}

and kafka.base64.txt

AAAAAAMuMTA6MDA6NTA6RUI6MUE6MEI6ODQ6M0HK25qo+GGQ85ThCAAAYEEAAKBBCgAAAAADLjEw
OjAwOkQ4OjFGOkNDOjVCOjNFOjc33t2aqPhhsNaIpQEAAKBAAAAgQQoAAAAAAy4xMDowMDpEODox
RjpDQzo1OTo2MjpGRtzfmqj4YeCOuUIAAKBAAAAgQQoAAAAAAy4xMDowMDpDNDpGNTo3QzpCNzpG
RTo5Qb7hmqj4YYDfouEIAABQQQAAoEEKAAAAAAMuMTA6MDA6RDg6MUY6Q0M6NTk6MkY6MEbc45qo
+GHoyrNKAACgQAAAIEEKAAAAAAMuMTA6MDA6QzQ6RjU6N0M6QzI6QUY6NDKm5Zqo+GHIs86aCQAA
UEEAAKBBCgAAAAADLjEwOjAwOjUwOkVCOjFBOjBCOjg1OjNBxueaqPhhsOi/mgkAAFBBAACgQQoA
AAAAAy4xMDowMDpEODoxRjpDQzo1QjozRDpBN9Dpmqj4YejvmKUBAACgQAAAMEEK

If I should capture the “raw” data using some other method then please tell me how.

I have also opened a new issue: #13155

Regards,
Dan

1 Like