Telegraf and MQTT Consumer, string contains XML format

PhoenixAlex · August 3, 2021, 3:08pm

Hi, there.
Is it possible to decode also string in XML format, which I receiving over MQTT consumer?

The string should have this content:
9900001-01-01T00:00:0002021-08-03T13:38:42.656Z0

Well, I need only this the value “99” from this 99

End of the day, I like to have the value in my influxDB.

Thanks a lot.

Franky1 · August 3, 2021, 3:11pm

If the MQTT payload is proper XML, yes, you have to configure the XML input format:

PhoenixAlex · August 3, 2021, 3:32pm

Hi Franky, I saw this, but I have not a file. It is part of MQTT consumer. Instead JSON I get this XML format.
Is is possible to combine this? Or do I need a own plugin?
Thanks.

Franky1 · August 3, 2021, 3:37pm

Yes, should be possible. XML input format can be configured for all plugins, the inputs.file example is perhaps a bit irritating here.

[[inputs.mqtt_consumer]]
  # do the whole mqtt_consumer config here

  data_format = "xml"
  
  [[inputs.mqtt_consumer.xml]]
    #metric_selection = "/Bus/child::Sensor"
    #metric_name = "string('example')"
    #timestamp = "/Gateway/Timestamp"
    #timestamp_format = "2006-01-02T15:04:05Z"

    ## Tag definitions using the given XPath queries.
    [inputs.mqtt_consumer.xml.tags]
      name   = "substring-after(Sensor/@name, ' ')"
      device = "string('the ultimate sensor')"

    ## Integer field definitions using XPath queries.
    [inputs.mqtt_consumer.xml.fields_int]
      consumers = "Variable/@consumers"

    ## Non-integer field definitions using XPath queries.
    [inputs.mqtt_consumer.xml.fields]

PhoenixAlex · August 4, 2021, 8:30am

Sorry, I´ am confused. What I need to do, if I need to catch this value?

16

1600001-01-01T00:00:0002021-08-03T13:38:42.656Z0

Thanks

PhoenixAlex · August 4, 2021, 8:31am

<DataValue xmlns="http://opcfoundation.org/UA/2008/02/Types.xsd" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Value><Value><UInt16>16</UInt16></Value>9999</Value><StatusCode><Code>0</Code></StatusCode><SourceTimestamp>0001-01-01T00:00:00</SourceTimestamp><SourcePicoseconds>0</SourcePicoseconds><ServerTimestamp>2021-08-03T13:38:42.656Z</ServerTimestamp><ServerPicoseconds>0</ServerPicoseconds></DataValue>

Pooh · August 4, 2021, 8:43am

The two consecutive tags look most odd for XML to me.

Antony.

PhoenixAlex · August 4, 2021, 9:08am

Yes, this will be send by a visualisation software (named VISU+ or Movicon). Normally this is used to send from one device to another device over MQTT Broker (it is xml silmilar). Well. it seems that it is not possible to send universal topics with this kind of software

<DataValue xmlns="http://opcfoundation.org/UA/2008/02/Types.xsd" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Value><Value><UInt16>16</UInt16></Value></Value><StatusCode><Code>0</Code></StatusCode><SourceTimestamp>0001-01-01T00:00:00</SourceTimestamp><SourcePicoseconds>0</SourcePicoseconds><ServerTimestamp>2021-08-03T13:38:42.656Z</ServerTimestamp><ServerPicoseconds>0</ServerPicoseconds></DataValue>type or paste code here

Franky1 · August 4, 2021, 10:53am

@PhoenixAlex

Please post the snippet of your Telegraf config for the mqtt plugin and xml formatting here.

Franky1 · August 4, 2021, 11:17am

I put this xml snippet in a file and ran this telegraf config, seems to work.
You just have to transfer this config to your mqtt input plugin.

[[inputs.file]]
  name_override = "xml"
  files = ["xmlparser.xml"]
  data_format = "xml"
  [[inputs.file.xml]]
    timestamp = "/DataValue/ServerTimestamp"
    timestamp_format = "2006-01-02T15:04:05.999Z"
    [inputs.file.xml.fields_int]
      value = "/DataValue/Value/Value/UInt16"
      statuscode = "/DataValue/StatusCode/Code"

[[outputs.file]]  # only for debugging
  files = ["xmlparser.out"]
  influx_sort_fields = true

Output:

xml statuscode=0i,value=16i 1627997922656000000

PhoenixAlex · August 4, 2021, 11:23am

Hi Franky, that sounds good. Thanks,
Now I need to transfer to MQTT plugin and XML parser, right?

Similar like that?


## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "xml"
  
  [[inputs.mqtt_consumer.xml]]
    #metric_selection = "/Bus/child::Sensor"
    #metric_name = "string('example')"
    timestamp = "/DataValue/ServerTimestamp"
    timestamp_format = "2006-01-02T15:04:05.999Z"

    ## Tag definitions using the given XPath queries.
    [inputs.mqtt_consumer.xml.tags]
      name   = "substring-after(Sensor/@name, ' ')"
      device = "string('the ultimate sensor')"

    ## Integer field definitions using XPath queries.
    [inputs.mqtt_consumer.xml.fields_int]
      value = "/DataValue/Value/Value/UInt16"
      statuscode = "/DataValue/StatusCode/Code"
	  
    ## Non-integer field definitions using XPath queries.
    [inputs.mqtt_consumer.xml.fields]

Franky1 · August 4, 2021, 11:29am

not quite, better this way:

[[inputs.mqtt_consumer]]
  # your mqtt plugin config here

  data_format = "xml"
  [[inputs.mqtt_consumer.xml]]
    timestamp = "/DataValue/ServerTimestamp"
    timestamp_format = "2006-01-02T15:04:05.999Z"
    [inputs.mqtt_consumer.xml.fields_int]
      value = "/DataValue/Value/Value/UInt16"
      statuscode = "/DataValue/StatusCode/Code"

PhoenixAlex · August 4, 2021, 11:56am

Hi Franky, you are the best. It seems that I can see a value, now
Thanks very much for that.

But I it is only one value and it seems that it is not refreshing if I sent more than one times.

For example like that:

<DataValue xmlns="http://opcfoundation.org/UA/2008/02/Types.xsd" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Value><Value><UInt16>98</UInt16></Value></Value><StatusCode><Code>0</Code></StatusCode><SourceTimestamp>0001-01-01T00:00:00</SourceTimestamp><SourcePicoseconds>0</SourcePicoseconds><ServerTimestamp>2021-08-04T11:54:12.906Z</ServerTimestamp><ServerPicoseconds>0</ServerPicoseconds></DataValue>

<DataValue xmlns="http://opcfoundation.org/UA/2008/02/Types.xsd" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Value><Value><UInt16>100</UInt16></Value></Value><StatusCode><Code>0</Code></StatusCode><SourceTimestamp>0001-01-01T00:00:00</SourceTimestamp><SourcePicoseconds>0</SourcePicoseconds><ServerTimestamp>2021-08-04T11:54:06.927Z</ServerTimestamp><ServerPicoseconds>0</ServerPicoseconds></DataValue>

PhoenixAlex · August 4, 2021, 11:59am

I think the reason is, that my connection is refusing, now.

2021-08-04T11:58:08Z E! [telegraf] Error running agent: starting input inputs.mqtt_consumer: network Error : dial tcp 127.0.0.1:1883: connect: connection refused

PhoenixAlex · August 4, 2021, 12:09pm

Hi Franky again, thanks a lot. Now all it is running.
I had some mistakes in my telegraf.conf.

Here is my version of the MQTT conumer with XML parser

My XML, which I have sent over a AWS MQTT Broker

<DataValue xmlns="http://opcfoundation.org/UA/2008/02/Types.xsd" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Value><Value><UInt16>98</UInt16></Value></Value><StatusCode><Code>0</Code></StatusCode><SourceTimestamp>0001-01-01T00:00:00</SourceTimestamp><SourcePicoseconds>0</SourcePicoseconds><ServerTimestamp>2021-08-04T11:54:12.906Z</ServerTimestamp><ServerPicoseconds>0</ServerPicoseconds></DataValue>

Part of the Telegraf.conf

[[inputs.mqtt_consumer]]
  ## Broker URLs for the MQTT server or cluster.  To connect to multiple
  ## clusters or standalone servers, use a seperate plugin instance.
  ##   example: servers = ["tcp://localhost:1883"]
  ##            servers = ["ssl://localhost:1883"]
  ##            servers = ["ws://localhost:1883"]
  servers = ["ssl://aaaaaaaaaaa-ats.iot.us-east-2.amazonaws.com:8883"]

  ## Topics that will be subscribed to.
  topics = [
    "/TestAlex1",
  ]

  ## The message topic will be stored in a tag specified by this value.  If set
  ## to the empty string no topic tag will be created.
  # topic_tag = "topic"

  ## QoS policy for messages
  ##   0 = at most once
  ##   1 = at least once
  ##   2 = exactly once
  ##
  ## When using a QoS of 1 or 2, you should enable persistent_session to allow
  ## resuming unacknowledged messages.
  # qos = 0

  ## Connection timeout for initial connection in seconds
  # connection_timeout = "30s"

  ## Maximum messages to read from the broker that have not been written by an
  ## output.  For best throughput set based on the number of metrics within
  ## each message and the size of the output's metric_batch_size.
  ##
  ## For example, if each message from the queue contains 10 metrics and the
  ## output metric_batch_size is 1000, setting this to 100 will ensure that a
  ## full batch is collected and the write is triggered immediately without
  ## waiting until the next flush_interval.
  # max_undelivered_messages = 1000

  ## Persistent session disables clearing of the client session on connection.
  ## In order for this option to work you must also set client_id to identify
  ## the client.  To receive messages that arrived while the client is offline,
  ## also set the qos option to 1 or 2 and don't forget to also set the QoS when
  ## publishing.
  # persistent_session = false

  ## If unset, a random client ID will be generated.
  # client_id = ""

  ## Username and password to connect MQTT server.
  # username = "telegraf"
  # password = "metricsmetricsmetricsmetrics"

  ## Optional TLS Config
  tls_ca = "/etc/telegraf/AmazonRootCA1.pem"
  tls_cert = "/etc/telegraf/aaaaaaaa.crt"
  tls_key = "/etc/telegraf/cccccccccccc-private.pem.key"
  ## Use TLS but skip chain & host verification
  insecure_skip_verify = true

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md

  data_format = "xml"
  [[inputs.mqtt_consumer.xml]]
    timestamp = "/DataValue/ServerTimestamp"
    timestamp_format = "2006-01-02T15:04:05.999Z"
    [inputs.mqtt_consumer.xml.fields_int]
      value = "/DataValue/Value/Value/UInt16"
      statuscode = "/DataValue/StatusCode/Code"

Topic		Replies	Views
Telegraf MQTT Consumer telegraf	4	4759	June 2, 2020
Telegraf as MQTT consumer - handling string and float values Telegraf telegraf , kapacitor , iot	10	15226	October 10, 2021
Convert MQTT messages with telegraf Telegraf telegraf	3	4385	May 30, 2018
MQTT input example needed Telegraf telegraf	5	7661	May 19, 2019
Reading all the fields from XPath in one go without defining each field Telegraf telegraf , xml	5	1168	March 15, 2022

Telegraf and MQTT Consumer, string contains XML format

Related Topics