Telegraf and MQTT Consumer, string contains XML format

Hi, there.
Is it possible to decode also string in XML format, which I receiving over MQTT consumer?

The string should have this content:
9900001-01-01T00:00:0002021-08-03T13:38:42.656Z0

Well, I need only this the value โ€œ99โ€ from this 99

End of the day, I like to have the value in my influxDB.

Thanks a lot.

If the MQTT payload is proper XML, yes, you have to configure the XML input format:

https://docs.influxdata.com/telegraf/v1.19/data_formats/input/xml/

Hi Franky, I saw this, but I have not a file. It is part of MQTT consumer. Instead JSON I get this XML format.
Is is possible to combine this? Or do I need a own plugin?
Thanks.

Yes, should be possible. XML input format can be configured for all plugins, the inputs.file example is perhaps a bit irritating here.

[[inputs.mqtt_consumer]]
  # do the whole mqtt_consumer config here

  data_format = "xml"
  
  [[inputs.mqtt_consumer.xml]]
    #metric_selection = "/Bus/child::Sensor"
    #metric_name = "string('example')"
    #timestamp = "/Gateway/Timestamp"
    #timestamp_format = "2006-01-02T15:04:05Z"

    ## Tag definitions using the given XPath queries.
    [inputs.mqtt_consumer.xml.tags]
      name   = "substring-after(Sensor/@name, ' ')"
      device = "string('the ultimate sensor')"

    ## Integer field definitions using XPath queries.
    [inputs.mqtt_consumer.xml.fields_int]
      consumers = "Variable/@consumers"

    ## Non-integer field definitions using XPath queries.
    [inputs.mqtt_consumer.xml.fields]

Sorry, Iยด am confused. What I need to do, if I need to catch this value?

16

1600001-01-01T00:00:0002021-08-03T13:38:42.656Z0

Thanks

<DataValue xmlns="http://opcfoundation.org/UA/2008/02/Types.xsd" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Value><Value><UInt16>16</UInt16></Value>9999</Value><StatusCode><Code>0</Code></StatusCode><SourceTimestamp>0001-01-01T00:00:00</SourceTimestamp><SourcePicoseconds>0</SourcePicoseconds><ServerTimestamp>2021-08-03T13:38:42.656Z</ServerTimestamp><ServerPicoseconds>0</ServerPicoseconds></DataValue>

The two consecutive tags look most odd for XML to me.

Antony.

Yes, this will be send by a visualisation software (named VISU+ or Movicon). Normally this is used to send from one device to another device over MQTT Broker (it is xml silmilar). Well. it seems that it is not possible to send universal topics with this kind of software :frowning:

<DataValue xmlns="http://opcfoundation.org/UA/2008/02/Types.xsd" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Value><Value><UInt16>16</UInt16></Value></Value><StatusCode><Code>0</Code></StatusCode><SourceTimestamp>0001-01-01T00:00:00</SourceTimestamp><SourcePicoseconds>0</SourcePicoseconds><ServerTimestamp>2021-08-03T13:38:42.656Z</ServerTimestamp><ServerPicoseconds>0</ServerPicoseconds></DataValue>type or paste code here

@PhoenixAlex

Please post the snippet of your Telegraf config for the mqtt plugin and xml formatting here.

I put this xml snippet in a file and ran this telegraf config, seems to work.
You just have to transfer this config to your mqtt input plugin.

[[inputs.file]]
  name_override = "xml"
  files = ["xmlparser.xml"]
  data_format = "xml"
  [[inputs.file.xml]]
    timestamp = "/DataValue/ServerTimestamp"
    timestamp_format = "2006-01-02T15:04:05.999Z"
    [inputs.file.xml.fields_int]
      value = "/DataValue/Value/Value/UInt16"
      statuscode = "/DataValue/StatusCode/Code"

[[outputs.file]]  # only for debugging
  files = ["xmlparser.out"]
  influx_sort_fields = true

Output:

xml statuscode=0i,value=16i 1627997922656000000

Hi Franky, that sounds good. Thanks,
Now I need to transfer to MQTT plugin and XML parser, right?

Similar like that?


## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "xml"
  
  [[inputs.mqtt_consumer.xml]]
    #metric_selection = "/Bus/child::Sensor"
    #metric_name = "string('example')"
    timestamp = "/DataValue/ServerTimestamp"
    timestamp_format = "2006-01-02T15:04:05.999Z"

    ## Tag definitions using the given XPath queries.
    [inputs.mqtt_consumer.xml.tags]
      name   = "substring-after(Sensor/@name, ' ')"
      device = "string('the ultimate sensor')"

    ## Integer field definitions using XPath queries.
    [inputs.mqtt_consumer.xml.fields_int]
      value = "/DataValue/Value/Value/UInt16"
      statuscode = "/DataValue/StatusCode/Code"
	  
    ## Non-integer field definitions using XPath queries.
    [inputs.mqtt_consumer.xml.fields]

not quite, better this way:

[[inputs.mqtt_consumer]]
  # your mqtt plugin config here

  data_format = "xml"
  [[inputs.mqtt_consumer.xml]]
    timestamp = "/DataValue/ServerTimestamp"
    timestamp_format = "2006-01-02T15:04:05.999Z"
    [inputs.mqtt_consumer.xml.fields_int]
      value = "/DataValue/Value/Value/UInt16"
      statuscode = "/DataValue/StatusCode/Code"

Hi Franky, you are the best. It seems that I can see a value, now :slight_smile:
Thanks very much for that.

But I it is only one value and it seems that it is not refreshing if I sent more than one times.

For example like that:

<DataValue xmlns="http://opcfoundation.org/UA/2008/02/Types.xsd" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Value><Value><UInt16>98</UInt16></Value></Value><StatusCode><Code>0</Code></StatusCode><SourceTimestamp>0001-01-01T00:00:00</SourceTimestamp><SourcePicoseconds>0</SourcePicoseconds><ServerTimestamp>2021-08-04T11:54:12.906Z</ServerTimestamp><ServerPicoseconds>0</ServerPicoseconds></DataValue>
<DataValue xmlns="http://opcfoundation.org/UA/2008/02/Types.xsd" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Value><Value><UInt16>100</UInt16></Value></Value><StatusCode><Code>0</Code></StatusCode><SourceTimestamp>0001-01-01T00:00:00</SourceTimestamp><SourcePicoseconds>0</SourcePicoseconds><ServerTimestamp>2021-08-04T11:54:06.927Z</ServerTimestamp><ServerPicoseconds>0</ServerPicoseconds></DataValue>

I think the reason is, that my connection is refusing, now.

2021-08-04T11:58:08Z E! [telegraf] Error running agent: starting input inputs.mqtt_consumer: network Error : dial tcp 127.0.0.1:1883: connect: connection refused

Hi Franky again, thanks a lot. Now all it is running. :grinning:
I had some mistakes in my telegraf.conf.

Here is my version of the MQTT conumer with XML parser

My XML, which I have sent over a AWS MQTT Broker

<DataValue xmlns="http://opcfoundation.org/UA/2008/02/Types.xsd" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Value><Value><UInt16>98</UInt16></Value></Value><StatusCode><Code>0</Code></StatusCode><SourceTimestamp>0001-01-01T00:00:00</SourceTimestamp><SourcePicoseconds>0</SourcePicoseconds><ServerTimestamp>2021-08-04T11:54:12.906Z</ServerTimestamp><ServerPicoseconds>0</ServerPicoseconds></DataValue>

Part of the Telegraf.conf

[[inputs.mqtt_consumer]]
  ## Broker URLs for the MQTT server or cluster.  To connect to multiple
  ## clusters or standalone servers, use a seperate plugin instance.
  ##   example: servers = ["tcp://localhost:1883"]
  ##            servers = ["ssl://localhost:1883"]
  ##            servers = ["ws://localhost:1883"]
  servers = ["ssl://aaaaaaaaaaa-ats.iot.us-east-2.amazonaws.com:8883"]

  ## Topics that will be subscribed to.
  topics = [
    "/TestAlex1",
  ]

  ## The message topic will be stored in a tag specified by this value.  If set
  ## to the empty string no topic tag will be created.
  # topic_tag = "topic"

  ## QoS policy for messages
  ##   0 = at most once
  ##   1 = at least once
  ##   2 = exactly once
  ##
  ## When using a QoS of 1 or 2, you should enable persistent_session to allow
  ## resuming unacknowledged messages.
  # qos = 0

  ## Connection timeout for initial connection in seconds
  # connection_timeout = "30s"

  ## Maximum messages to read from the broker that have not been written by an
  ## output.  For best throughput set based on the number of metrics within
  ## each message and the size of the output's metric_batch_size.
  ##
  ## For example, if each message from the queue contains 10 metrics and the
  ## output metric_batch_size is 1000, setting this to 100 will ensure that a
  ## full batch is collected and the write is triggered immediately without
  ## waiting until the next flush_interval.
  # max_undelivered_messages = 1000

  ## Persistent session disables clearing of the client session on connection.
  ## In order for this option to work you must also set client_id to identify
  ## the client.  To receive messages that arrived while the client is offline,
  ## also set the qos option to 1 or 2 and don't forget to also set the QoS when
  ## publishing.
  # persistent_session = false

  ## If unset, a random client ID will be generated.
  # client_id = ""

  ## Username and password to connect MQTT server.
  # username = "telegraf"
  # password = "metricsmetricsmetricsmetrics"

  ## Optional TLS Config
  tls_ca = "/etc/telegraf/AmazonRootCA1.pem"
  tls_cert = "/etc/telegraf/aaaaaaaa.crt"
  tls_key = "/etc/telegraf/cccccccccccc-private.pem.key"
  ## Use TLS but skip chain & host verification
  insecure_skip_verify = true

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md

  data_format = "xml"
  [[inputs.mqtt_consumer.xml]]
    timestamp = "/DataValue/ServerTimestamp"
    timestamp_format = "2006-01-02T15:04:05.999Z"
    [inputs.mqtt_consumer.xml.fields_int]
      value = "/DataValue/Value/Value/UInt16"
      statuscode = "/DataValue/StatusCode/Code"
1 Like