Telegraf xml parser

Paroles_Paroles · January 24, 2022, 10:51pm

Hello, can someone help with telegaf plugin to parse xml data into influx, here is sample xml data which I’m getting from my device.

What I would need to collect is:
localDn = SubNetwork=Router1,ManagedElement=node-1
measType p=“1”>VS.EthernetKBytesInRate + value 0.00
measObjLdn=“PoolId=0, PoolMember=1, Machine=Router1-interface1”

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="MeasDataCollection.xsl"?>
<measCollecFile
	xmlns="http://www.3gpp.org/ftp/specs/archive/32_series/32.435#measCollec">
	<fileHeader fileFormatVersion="32.435 V15.0"
		vendorName="Cisco"
		dnPrefix="">
		<fileSender
			localDn="SubNetwork=Router1,ManagedElement=node-1"
			elementType="Router,Switch" />
		<measCollec beginTime="2022-01-20T12:00:00+02:00" />
	</fileHeader>
	<measData>
		<managedElement
			localDn="SubNetwork=Router1,ManagedElement=node-1"
			userLabel=""
			swVersion="Router20.5.0.15" />
		<measInfo>
			<job jobId="defaultJob" />
			<granPeriod duration="PT900S" endTime="2022-01-20T12:15:00+02:00" />
			<measType p="1">VS.EthernetKBytesInRate</measType>
			<measType p="2">VS.EthernetKBytesOutRate</measType>
			<measType p="3">VS.EthernetPacketsInRate</measType>
			<measType p="4">VS.EthernetPacketsOutRate</measType>
			<measValue measObjLdn="PoolId=0, PoolMember=0, Machine=Router1-interface0, UUID=f97cb975-8b09-30cf3d51b037">
				<r p="1">0.00</r>
				<r p="2">0.00</r>
				<r p="3">0.00</r>
				<r p="4">0.00</r>
			</measValue>
			<measValue measObjLdn="PoolId=0, PoolMember=1, Machine=Router1-interface1, UUID=10da4adc-a4bd-360cdb8342de">
				<r p="1">146.00</r>
				<r p="2">130.00</r>
				<r p="3">23.00</r>
				<r p="4">250.00</r>
			</measValue>
		</measInfo>
	</measData>
	<fileFooter>
		<measCollec endTime="2022-01-20T12:15:00+02:00" />
	</fileFooter>
</measCollecFile>

Thanks for help

Franky1 · January 25, 2022, 9:57am

How about you invest some time and try it yourself first? Read the documentation and configure the XML parser. If you get stuck, post your configuration here.

Paroles_Paroles · January 26, 2022, 12:20pm

Hello, thanks for response. Trying some basics

telegraf config under /etc/telegraf/telegraf.d/
[inputs.file]
files = ["/home/user/example.xml"]
data_format = “xml”

[inputs.file.xpath]
[inputs.file.xpath.tags]
instance = “//fileHeader/@localDn”

Any idea why i’m getting error?
telegraf --once --config /etc/telegraf/telegraf.d/xml.conf
2022-01-26T12:17:19Z I! Starting Telegraf 1.21.1
2022-01-26T12:17:19Z E! [telegraf] Error running agent: Error: no outputs found, did you provide a valid config file?

Franky1 · January 26, 2022, 12:51pm

You need at least one input and one output plugin configured in your telegraf config file.
For debugging purposes you can just add for example the file output plugin:

# file output only for debugging
[[outputs.file]]
  files = ["xml.out"]
  influx_sort_fields = true

Paroles_Paroles · January 31, 2022, 8:25pm

Thanks for hint, trying to start building config, stopped here

[[inputs.file]]
name_override = “test”
files = ["/home/user/example.xml"]
data_format = “xml”

[[inputs.file.xml]]
metric_selection = “/measCollecFile”
[inputs.file.xml.tags]
instance = “string(fileHeader/fileSender/@localDn)”

[[outputs.file]]
files = [“test.out”]
influx_sort_fields = true

but nothing is written into file.

Giovanni_Luisotto · February 1, 2022, 8:36am

You need at least a field in your data points.
Without a field, the data point does not exist (and so far you only mapped a tag)

I suggest to take a look at the InflxDB basic concepts here

Paroles_Paroles · February 1, 2022, 10:12am

Ok, got it

[[inputs.file]]
name_override = “test”
files = ["/home/user/example.xml"]
data_format = “xml”

[[inputs.file.xml]]
metric_selection = "/measCollecFile"
[inputs.file.xml.fields]
instance = "string(/measData/managedElement/@localDn)"
measurement = "string(/measData/measInfo/measType)"
value = "number(/measData/measInfo/measValue/r)"

output:
instance="SubNetwork=SubNetwork=Router1,ManagedElement=node-1",measurement="VS.EthernetKBytesInRate",value=0 1643709037000000000

Question:
how can I get all 4x “measType” objects (now I get only one) and map them with values and “measObjLdn”, it should look something like this:

instance="SubNetwork=Router1,ManagedElement=node-1",measurement="VS.EthernetKBytesInRate",object="PoolId=0, PoolMember=0, Machine=Router1-interface0, UUID=f97cb975-8b09-30cf3d51b037",value=0
instance="SubNetwork=Router1,ManagedElement=node-1",measurement="VS.EthernetKBytesOutRate",object="PoolId=0, PoolMember=0, Machine=Router1-interface0, UUID=f97cb975-8b09-30cf3d51b037",value=0
instance="SubNetwork=Router1,ManagedElement=node-1",measurement="VS.EthernetPacketsInRate",object="PoolId=0, PoolMember=0, Machine=Router1-interface0, UUID=f97cb975-8b09-30cf3d51b037",value=0
instance="SubNetwork=Router1,ManagedElement=node-1",measurement="VS.EthernetPacketsOutRate",object="PoolId=0, PoolMember=0, Machine=Router1-interface0, UUID=f97cb975-8b09-30cf3d51b037",value=0
#
instance="SubNetwork=Router1,ManagedElement=node-1",measurement="VS.EthernetKBytesInRate",object="PoolId=0, PoolMember=1, Machine=Router1-interface1, UUID=f97cb975-8b09-30cf3d51b037",value=146
instance="SubNetwork=Router1,ManagedElement=node-1",measurement="VS.EthernetKBytesOutRate",object="PoolId=0, PoolMember=1, Machine=Router1-interface1, UUID=f97cb975-8b09-30cf3d51b037",value=130
instance="SubNetwork=Router1,ManagedElement=node-1",measurement="VS.EthernetPacketsInRate",object="PoolId=0, PoolMember=1, Machine=Router1-interface1, UUID=f97cb975-8b09-30cf3d51b037",value=23
instance="SubNetwork=Router1,ManagedElement=node-1",measurement="VS.EthernetPacketsOutRate",object="PoolId=0, PoolMember=1, Machine=Router1-interface1, UUID=f97cb975-8b09-30cf3d51b037",value=250

Paroles_Paroles · February 4, 2022, 12:22pm

any suggestion how can I achieve such export ? Thanks

Franky1 · February 6, 2022, 4:30pm

This works partially, however the measType is not captured right.
Quite tricky to get the right index from the parent - i think we need an xpath expert
I also added the timestamp.

[[inputs.file.xml]]
  metric_selection = "/measCollecFile/measData/measInfo/measValue/r"
  timestamp = "/measCollecFile/fileFooter/measCollec/@endTime"
  timestamp_format = "2006-01-02T15:04:05-07:00"
  [inputs.file.xml.fields]
    localDn = "string(../../../managedElement/@localDn)"
    measType = "string(../../measType)"
    measObjLdn = "string(../@measObjLdn)"
    value = "number(.)"

Paroles_Paroles · February 7, 2022, 12:42pm

Thanks @Franky1 for help

If I test measType with http://xpather.com
then values is returned
/measCollecFile/measData/measInfo/measType

    VS.EthernetKBytesInRate
    VS.EthernetKBytesOutRate
    VS.EthernetPacketsInRate
    VS.EthernetPacketsOutRate

Franky1 · February 7, 2022, 1:01pm

My example configuration above does not work completely correctly.
I get VS.EthernetKBytesInRate for measType for all four values.

We want to read the <r> tag, but the measType is unfortunately one level higher.
I am stumped on how to write the query so to get the correct index of measType.

Paroles_Paroles · February 9, 2022, 2:04pm

Anyone else has some idea what is wrong with query and how get it work?

Franky1 · February 10, 2022, 2:43pm

I don’t have an exact solution to your question. I’m also not sure if it works at all with xpath. The problem is that the measType is in the parent object.

But I have an alternative solution that might be at least as good - provided that the xml payload remains as in the example.

Btw, I would limit the tags and fields to the necessary metrics. Also, I would avoid strings in fields if possible, as InfluxDB is not meant for “logging”.

[[inputs.file.xml]]
  metric_selection = "/measCollecFile/measData/measInfo/measValue"
  timestamp = "/measCollecFile/fileFooter/measCollec/@endTime"
  timestamp_format = "2006-01-02T15:04:05-07:00"
  [inputs.file.xml.tags]
    localDn = "string(../../managedElement/@localDn)"
    Machine = "string(substring-before(substring-after(@measObjLdn, 'Machine='), ', UUID='))"
    SubNetwork = "string(substring-before(substring-after(../../managedElement/@localDn, 'SubNetwork='), ',ManagedElement='))"
    ManagedElement = "string(substring-after(../../managedElement/@localDn, 'ManagedElement='))"
  [inputs.file.xml.fields]
    measObjLdn = "string(@measObjLdn)"
    PoolId = "number(substring-before(substring-after(@measObjLdn, 'PoolId='), ', PoolMember='))"
    PoolMember = "number(substring-before(substring-after(@measObjLdn, 'PoolMember='), ', Machine='))"
    EthernetKBytesInRate = "number(r[@p='1'])"
    EthernetKBytesOutRate = "number(r[@p='2'])"
    EthernetPacketsInRate = "number(r[@p='3'])"
    EthernetPacketsOutRate = "number(r[@p='4'])"

Paroles_Paroles · February 16, 2022, 9:22am

Thanks @Franky1 seems this will not work and also this is only small part of xml file. File is quit big with many values.
I’m thinking maybe to do some processing before, for example parse xml > json and after parse json with telegraf.

Franky1 · February 16, 2022, 12:18pm

If you show the whole file with real data, i might be able to give you some ideas.

I’m not sure the result will be that much easier. Then you have the same problems, only with the json parser. You go from the frying pan into the fire.

If the data is really that complex, you might be better off writing a custom inputs.execd plugin or processors.execd or processors.starlark plugin.

Topic		Replies	Views
Need help parsing XML to telegraf, happy to pay Telegraf telegraf , xml	3	569	August 20, 2021
XML Parser code not working Telegraf influxdb , telegraf , query	2	364	January 4, 2023
Parse XML data using telegraf Telegraf telegraf	6	642	July 7, 2021
How to use inputs.http plugin for the http APIs which return XML data Telegraf telegraf , xml	11	1755	October 20, 2021
XML parsing using telegraf Telegraf telegraf , xml	2	544	April 20, 2022

Telegraf xml parser

Related Topics