I’m using influxdb to collect data in two formats - boolean and float. Data of both types are stored under the same root topic name.
Using the documented syntax for mqtt_consumer, it seems I can only set a consumer to read one of these formats at a time, and any data received using the other format will result in an error.
I suppose I could declare two instances of the same consumer to do the writing and put up with lots of errors in both consumers but I would prefer something more elegant and efficient.
My best idea is to create a different parser type - call it ‘universal_value’ which would float or boolean dependent on the contents of the value message.
Does such a parser already exist?
Is there another way of doing this?
Thanks!
Dave
Telegraf can parse a number of input formats. You are correct that if you specify the “value” input format, then you also need to provide a data_type in your configuration.
You write that “Data of both types are stored under the same root topic name”—I assume that this means you have different topics for each bit of data, with the same root? For example, root-topic-name/data-type1 and root-topic-name/data-type2? If that’s the case, you can configure the parser and value for each topic.
Another option would be to send your data using InfluxDB Line Protocol, instead of a value. This would let you send both types of data in the same message, and specify which field you’d like to store the various pieces of data.
Hi Noah.
Thanks for replying.
Yes I have different topics with the same route, exactly as you have described. However ideally I don’t want to have to write/generate a new telegraf config for each of these as these topics can be generated/edited by the user at run time.
I’m not completely clear on how line protocol works but it sounds as if this would require me to redefine the structure of the messages in my mqtt stream which I don’t want to do.
So how about the custom parser idea? it would be fairly simple to write something that looks at the contents of the message and if it’s ‘true’ or ‘false’ returns a bool, otherwise tries to parse a float.
Thanks
Dave
You could write a custom parser, but it seems like a step backward to store different data types under different topics, then ignore this and try to write custom code to infer what the application should do based on the type.
I would strongly recommend you look into configuration automation and infrastructure as code tooling to manage both your MQTT topics and Telegraf configs. Dealing with these manually sounds like a recipe for trouble later down the road, and putting proper tooling in place will alleviate the toil of creating and managing configs.
If you do decide to write a custom parser, keep in mind that you can’t store data of different types in the same field in InfluxDB, so you will need to have logic about which field to store the data in as part of the parser. This is not how we generally use parsers, and would be specific to your application.
Thanks Noah.
We already have an infrastructure which allows users and services to query the data types of the different topics.
Telegraf/influx is simply an add-on we’d like to support. We don’t want to have to rewrite/regenerate a config file every time the data structure changes.
For now a custom parser will do the job. If we continue to use telegraf/influx I guess we’ll look at writing our own mqtt plugin - I’m not over-happy at the limited topic-mapping options in the current plugin anyway.
Cheers
Dave
For others coming to this thread, the recommended approach is to generate and deploy your configuration files based on changes to the data structures.
