We are getting protobuf message, which we are trying to read using telegraf. Few errors we are getting:
a. what value should be there for xpath_protobuf_type. I am trying some options but getting error “Proto” not found.
b. How should i define the tags and fields, since the example given here refers to xml format and cannot find anything for protobuf. telegraf/plugins/parsers/xpath at master · influxdata/telegraf · GitHub
Hi,
Do you know what protobuf type are you using? Do you have a protobuf definition file?
Thanks!
syntax = "proto3";
option java_package = "com.ruckus.cloud.lte.sciproxy.gpb";
option java_outer_classname = "SciMessageProtocol";
We are not sure of protobuf type. Is this information enough for now?
Thanks
I am not familiar with that protobuf. However, you will need to provide the definition file and the type. Once those are resolved you can move on to creating tags and fields with the XPath parser.
Is there a defined set of protobuf types? Because i defined the proto file and provided protobuf type but still was getting same error proto not found.
no, can provide the exact error message, please?
ok and what does your configuration file look like?
[[inputs.mqtt_consumer]]
servers = ["aws_mq_broker_provided"]
qos = 0
topics = [
"Topic_name_provided"
]
client_id = "telegraf"
username = "######"
password = "####"
data_format = "xpath_protobuf"
xpath_protobuf_type = "com.ruckuswireless.smallcell.sciproxy.gpb"
xpath_protobuf_file = "proto_file_location_provided"
xpath_print_document = true
And proto-file is:
syntax = "proto3";
option java_package = "com.ruckuswireless.smallcell.sciproxy.gpb";
message Ap {
double dl = 1;
double ul = 2;
}
message ApNon{
uint32 StartTime = 1;
uint64 SampleDurationSec = 2;
repeated Ap ap = 10;
}
package directive to correctly name your custom proto buffer. Something like this:package myproto;
xpath_protobuf_type value should be set to the class + message type. Using the package name above of myproto + your Ap message type would require:xpath_protobuf_type = "myproto.Ap"
I used this protobuf file:
syntax = "proto3";
package myproto;
message Ap {
double dl = 1;
double ul = 2;
}
message ApNon{
uint32 StartTime = 1;
uint64 SampleDurationSec = 2;
repeated Ap ap = 10;
}
with this config:
[agent]
[[outputs.file]]
[[inputs.file]]
files = ["test.data"]
data_format = "xpath_protobuf"
xpath_protobuf_type = "myproto.Ap"
xpath_protobuf_file = "my.proto"
xpath_print_document = true
And while my file doesn’t have protobuf messages, I got the following error message, which indicates it has loaded the proto type and is attempting to read in data.
2022-03-04T13:58:40Z E! [inputs.file] Error in plugin: proto: cannot parse invalid wire-format data
Give that a shot!
Hi, Thanks for that it’s working, but its working partially. So first problem is we are getting import error for this file:
syntax = "proto3";
package myproto;
import "APnon.proto";
message Ap {
double dl = 1;
double ul = 2;
ApNon apnon = 100;
}
///contents of APnon.proto:
syntax = "proto3";
package myproto;
message Apnoncumulative {
double dl = 1;
double ul = 2;
}
message ApNon{
repeated Apnoncumulative apnoncumulativeobject = 10;
}
So after getting import error. I tried to define telegraf config for each file. But for ApNon.proto file i am getting error “cannot parse invalid wire-format data” for second message but for first message i am able to read contents. Telegraf config for ApNon.proto
xpath_protobuf_type = "package.ApNon" //invalid wire format
xpath_protobuf_type = "package.Apnoncumulative" //working correctly.
Any idea to resolve these 2 issues.
Thanks
Telegraf is not able to read more than one type. It is only going to read a single value from that type option.
Sorry for not being clear. I meant with one telegraf config wire format was giving error and for second telegraf config it was parsing correctly. But i need to use first vonfiguration only.
Can you provide the exact error message and config you are using, please?
Proto file contents:
//ApNon.proto
syntax = "proto3";
package myproto;
message Apnoncumulative {
double dl = 1;
double ul = 2;
}
message ApNon{
repeated Apnoncumulative apnoncumulativeobject = 10;
}
So here is first configuration:
data_format = "xpath_protobuf"
xpath_protobuf_type = "package.ApNon"
xpath_protobuf_file = "APnon.proto"
Telegraf log:
Second telegraf configuration:
data_format = "xpath_protobuf"
xpath_protobuf_type = "package.Apnoncumulative"
xpath_protobuf_file = "APnon.proto"
Telegraf log:
XML document equivalent: "<?xml version=\"1.0\"?><dl>5.5</dl><ul>3.0</ul>"
What would be solution to resolve issue in first telegraf configuration. Because requirement is to read ApNon message.
Thanks
@Pratik_Das_Baghel & @Ravikant_Gautam are you two working on the same project/team?
yes we are working on same project.
This error message comes directly from the protobuf library:
cannot parse invalid wire-format data
It means it was unable to parse the data you received given the type and message you specified. It could be that the data coming in is not the format you expected, your buffer message could be wrong, or some other things. At this point, Telegraf is doing what you told it to do.
I would suggest checking out some of the raw data with Protobuf Code Generator and Parser | protobufnet | Marc Gravell and ensuring it matches the type you are specifying.
yes we are working on same project.
It would be really nice to keep these questions in a single place. We are now up to three places, between this topic, the new topic you created, and the bug you filed. I’m going to let Sven help you out with importing multiple files.
Hi sorry for creating a completely different thread for importing multiple proto file errors.
The issue I raised on GitHub because Sven told me to do the same so he can re-produce the same issue and help me better.
Thanks
Hi. So does the telegraf protobuf parser supports the above protobuf format. With one message calling another message using repeated.
