Xpath_json nest array

Telegraf
1

Hi. I’m really close to solving my issue and just need a little help getting over the last piece of data. I have a nested array “events” in this dataset:

{
    "count": 9,
    "limit": 200,
    "offset": 0,
    "items": [
        {
            "uid": "52e2e31a-19a9-4bf1-9259-cab75d722ea1",
            "status": "ACTIVE",
            "lastEventDate": "2025-07-10T03:26:25Z",
            "entityUid": "1e9b9771-0bcf-4e55-9e70-7fad82241b27",
            "entityName": "MGR",
            "events": []
        },
        {
            "uid": "a03431ca-893f-491c-924b-7053720b93a8",
            "status": "COMPLETED",
            "lastEventDate": "2025-07-10T02:56:27Z",
            "entityUid": "1e9b9771-0bcf-4e55-9e70-7fad82241b27",
            "entityName": "MGR",
            "events": [
                {
                    "description": "Read MGR device records successfully",
                    "username": "System User",
                    "date": "2025-07-10T02:56:27Z",
                    "action": "UPDATE"
                }
            ]
        }
    ]
}

I can get all of the data in the output, but the events array enumerates as a map[string]. I’m unsure how to unspool this inner array.

scc_changelog,host=cc05ab3969dd uid="a03431ca-893f-491c-924b-7053720b93a8",status="COMPLETED",entityName="MGR",entityUid="1e9b9771-0bcf-4e55-9e70-7fad82241b27",lastEventDate="2025-07-10T02:56:27Z",events="[map[action:UPDATE date:2025-07-10T02:56:27Z description:Read MGR device records successfully username:System User]]" 1752116187000000000

This is the part that I need help on:

events="[map[action:UPDATE date:2025-07-10T02:56:27Z description:Read MGR device records successfully username:System User]]"

I’m guessing it’s because I have the array as a field. I’m ok with just flattening the events array as a string minus the “map” syntax. But looking for someone with more mojo with this than I for advice.
Here is my config:

[[inputs.file]]
  name_override = "scc_changelog"
  files = ["/etc/telegraf/telegraf.d/test.json"]
  data_format = "xpath_json"
  xpath_allow_empty_selection = true
  xpath_native_types = true

  [[inputs.file.xpath]]
      metric_name = "'scc_changelog'"
      metric_selection = "//items/*"
      timestamp = "lastEventDate"
      timestamp_format = "2006-01-02T15:04:05.9999999Z"
      [inputs.file.xpath.fields]
        uid = "uid"
        status = "status"
        entityName = "entityName"
        entityUid = "entityUid"
        lastEventDate = "lastEventDate"
        events = "events"

[[outputs.file]]
  files = [ "stdout" ]

Thanks in advance!

2

Handling Nested Arrays in Telegraf XPath JSON Parser

When using Telegraf’s XPath JSON parser with nested arrays, you might encounter output like:

events="[map[action:UPDATE date:2025-07-10T02:56:27Z description:Read MGR device records successfully username:System User]]"

This occurs because Telegraf converts JSON arrays to Go’s internal map representation when xpath_native_types = true.

Solutions

Solution 1: Separate Metrics for Events (Recommended)

The cleanest approach is to create separate metrics for each event, following time-series best practices:

[[inputs.file]]
  name_override = "scc_changelog"
  files = ["/etc/telegraf/telegraf.d/test.json"]
  data_format = "xpath_json"
  xpath_allow_empty_selection = true
  xpath_native_types = true

  # Main changelog entries
  [[inputs.file.xpath]]
      metric_name = "'scc_changelog'"
      metric_selection = "//items/*"
      timestamp = "lastEventDate"
      timestamp_format = "2006-01-02T15:04:05Z"
      [inputs.file.xpath.fields]
        uid = "uid"
        status = "status"
        entityName = "entityName"
        entityUid = "entityUid"
        lastEventDate = "lastEventDate"
        events_count = "count(events/*)"

  # Events as separate metrics
  [[inputs.file.xpath]]
      metric_name = "'scc_changelog_events'"
      metric_selection = "//items/*/events/*"
      timestamp = "date"
      timestamp_format = "2006-01-02T15:04:05Z"
      [inputs.file.xpath.tags]
        parent_uid = "../../../uid"
        parent_status = "../../../status"
        parent_entityName = "../../../entityName"
      [inputs.file.xpath.fields]
        description = "description"
        username = "username"
        action = "action"

This creates two metric types:

  • scc_changelog: Main entries with event count
  • scc_changelog_events: Individual events with parent relationship

Solution 2: Force String Conversion

If you need events as a single field, disable native types:

[[inputs.file]]
  name_override = "scc_changelog"
  files = ["/etc/telegraf/telegraf.d/test.json"]
  data_format = "xpath_json"
  xpath_allow_empty_selection = true
  xpath_native_types = false  # Prevents Go map conversion

  [[inputs.file.xpath]]
      metric_name = "'scc_changelog'"
      metric_selection = "//items/*"
      timestamp = "lastEventDate"
      timestamp_format = "2006-01-02T15:04:05Z"
      [inputs.file.xpath.fields]
        uid = "uid"
        status = "status"
        entityName = "entityName"
        entityUid = "entityUid"
        lastEventDate = "lastEventDate"
        events = "string(events)"  # Force string conversion

Solution 3: Batch Field Processing

Use field selection to extract individual event properties:

[[inputs.file]]
  name_override = "scc_changelog"
  files = ["/etc/telegraf/telegraf.d/test.json"]
  data_format = "xpath_json"
  xpath_allow_empty_selection = true
  xpath_native_types = false

  [[inputs.file.xpath]]
      metric_name = "'scc_changelog'"
      metric_selection = "//items/*"
      timestamp = "lastEventDate"
      timestamp_format = "2006-01-02T15:04:05Z"
      
      # Process events as individual fields
      field_selection = "events/*"
      field_name = "concat('event_', position(), '_', name())"
      field_value = "string(.)"
      
      [inputs.file.xpath.tags]
        uid = "uid"
        status = "status"
        entityName = "entityName"
        entityUid = "entityUid"
      
      [inputs.file.xpath.fields]
        lastEventDate = "lastEventDate"

Key Points

  • Setting xpath_native_types = false prevents Go map serialization
  • The field_selection feature allows batch processing of array elements
  • Separate metrics provide better queryability and follow time-series conventions
  • Use XPath functions like string() to force type conversion when needed

Recommendation: Use Solution 1 for the best time-series data structure and queryability.

3

Wow thanks so much for the details and excellent explanation. I really, really appreciate the effort. Cheers!