Can't parse every field of my JSON file

Telegraf
, , , ,
1

Hello,

I’ve written a python script that requests a vrealizeops API and retrieves the response in JSON.
First of all, before going any further, because of the security measures in place in my company, the VM I’m currently working on doesn’t have the ports open to update telegraf from the repositories, the version I have doesn’t have the json_v2 parser…

So I’m recovering this input:

{
  "viewsData": [
    {
      "rows": [
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1693785600000,
            "3": 495440.0,
            "2": 28.15151596069336,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        },
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1694390400000,
            "3": 501264.0,
            "2": 27.820512771606445,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        },
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1694995200000,
            "3": 502272.0,
            "2": 26.972028732299805,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        },
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1695600000000,
            "3": 507136.0,
            "2": 26.82517433166504,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        },
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1696204800000,
            "3": 508384.0,
            "2": 26.11655044555664,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        },
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1696809600000,
            "3": 510128.0,
            "2": 25.93473243713379,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        },
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1697414400000,
            "3": 513584.0,
            "2": 25.680652618408203,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        },
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1698019200000,
            "3": 520144.0,
            "2": 25.177156448364258,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        },
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1698624000000,
            "3": 522368.0,
            "2": 24.221445083618164,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        },
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1699228800000,
            "3": 525240.0,
            "2": 23.897436141967773,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        },
        {
          "cells": {
            "objUUID": "2cf65c5b-58d3-4962-8bdf-9cca87897586",
            "grandTotal": false,
            "groupUUID": null,
            "summary": false,
            "1": 1699833600000,
            "3": 525304.0,
            "2": 23.479021072387695,
            "4": 686400.0,
            "objId": "hp-3par"
          }
        }
      ],
      "name": "hp3par",
      "startTime": 1693859599104,
      "endTime": 1699907599104,
      "type": "list-view",
      "columns": [
        {
          "unit": null,
          "key": "objId",
          "label": "Name"
        },
        {
          "unit": null,
          "key": "1",
          "label": "week"
        },
        {
          "unit": "%",
          "key": "2",
          "label": "Free scape (%)"
        },
        {
          "unit": null,
          "key": "3",
          "label": "used capacity(GiB)"
        },
        {
          "unit": null,
          "key": "4",
          "label": "Total capacity (GiB)"
        },
        {
          "unit": null,
          "key": "summary",
          "label": "Summary"
        },
        {
          "unit": null,
          "key": "grandTotal",
          "label": "Grand Total"
        },
        {
          "unit": null,
          "key": "groupUUID",
          "label": "Group UUID"
        },
        {
          "unit": null,
          "key": "objUUID",
          "label": "Object UUID"
        }
      ],
      "description": ""
    }
  ]
}

So far the only output I can retrieve at best is this:

viewsData_0_description="",
viewsData_0_endTime=1699907309072,
viewsData_0_name="hp3par",
viewsData_0_rows_0_cells_1=1693785600000,
viewsData_0_rows_0_cells_2=28.15151596069336,
viewsData_0_rows_0_cells_3=495440,
viewsData_0_rows_0_cells_4=686400,
viewsData_0_rows_10_cells_1=1699833600000,
viewsData_0_rows_10_cells_2=23.479021072387695,
viewsData_0_rows_10_cells_3=525304,
viewsData_0_rows_10_cells_4=686400,
viewsData_0_rows_1_cells_1=1694390400000,
viewsData_0_rows_1_cells_2=27.820512771606445,
viewsData_0_rows_1_cells_3=501264,
viewsData_0_rows_1_cells_4=686400,
viewsData_0_rows_2_cells_1=1694995200000,
viewsData_0_rows_2_cells_2=26.972028732299805,
viewsData_0_rows_2_cells_3=502272,
viewsData_0_rows_2_cells_4=686400,
viewsData_0_rows_3_cells_1=1695600000000,
viewsData_0_rows_3_cells_2=26.82517433166504,
viewsData_0_rows_3_cells_3=507136,
viewsData_0_rows_3_cells_4=686400,
viewsData_0_rows_4_cells_1=1696204800000,
viewsData_0_rows_4_cells_2=26.11655044555664,
viewsData_0_rows_4_cells_3=508384,
viewsData_0_rows_4_cells_4=686400,
viewsData_0_rows_5_cells_1=1696809600000,
viewsData_0_rows_5_cells_2=25.93473243713379,
viewsData_0_rows_5_cells_3=510128,
viewsData_0_rows_5_cells_4=686400,
viewsData_0_rows_6_cells_1=1697414400000,
viewsData_0_rows_6_cells_2=25.680652618408203,
viewsData_0_rows_6_cells_3=513584,
viewsData_0_rows_6_cells_4=686400,
viewsData_0_rows_7_cells_1=1698019200000,
viewsData_0_rows_7_cells_2=25.177156448364258,
viewsData_0_rows_7_cells_3=520144,
viewsData_0_rows_7_cells_4=686400,
viewsData_0_rows_8_cells_1=1698624000000,
viewsData_0_rows_8_cells_2=24.221445083618164,
viewsData_0_rows_8_cells_3=522368,
viewsData_0_rows_8_cells_4=686400,
viewsData_0_rows_9_cells_1=1699228800000,
viewsData_0_rows_9_cells_2=23.897436141967773,
viewsData_0_rows_9_cells_3=525240,
viewsData_0_rows_9_cells_4=686400,
viewsData_0_type="list-view" 1693859309000000000

With this config:

[[inputs.exec]]
        ## Commands array
        commands = ["/etc/telegraf/scripts/get_views_vrops.py"]
        ## Timeout for each command to complete.
        timeout = "120s"
        interval = "86400s"
        name_override = "Get_vROps_storage_views"
        data_format = "json"
        json_time_key = "viewsData_0_startTime"
        json_time_format = "unix_ms"
        json_name_key = "viewsData_0_startTime"
        json_string_fields = ["viewsData_0_name", "viewsData_0_description", "viewsData_0_type"]

My questions are:

1/ Why does it manage to retrieve the “rows” array but not the “columns” array?
2/ The “rows” and “columns” arrays are never of a fixed size, so how can I, for example, retrieve the “objId” fields found in all the “cells” of the “rows” array?

Thank you in advance for your answers !
Best regards

2

Doing a bit of research and I found the correct syntax in order to answer my questions.
If you want to retrieve a certain type of string field in a array of undetermined size, the key here is

‘*’ char

For exemple if I want to retrieve the field “key” of each occurence from the “columns” array, here is the correct syntax:

"viewsData_0_columns_*_key

This way you tell the parser, no matter how long the array is to check in every cells of it

3

Hello @Dmh991.2,
Thank you so much for including your solution here. Im sure it’ll help others thank you.