Telegraf Parsing

Telegraf
,
1

Hi All

I am really breaking my head over how to setup parsing in telegraf. I am probably missing something stupid, but I am new so trial and error has unfortunately not been my friend.

I am trying to retrieve information from our local electrical loadshedding (blackouts) schedules. My expectation is that each line relates to the appropriate day (probably as a tag) with fields representing the various stages of load shedding, ie, ranging from stages 1 - 8. A copy of the HTTP request is below:

{
    "events": [
        {
            "end": "2023-01-16T19:30:00+02:00",
            "note": "Stage 8 (TESTING: current)",
            "start": "2023-01-16T17:00:00+02:00"
        }
    ],
    "info": {
        "name": "TESTING Honeydew (11)",
        "region": "JHB City Power"
    },
    "schedule": {
        "days": [
            {
                "date": "2023-01-16",
                "name": "Monday",
                "stages": [
                    [
                        "18:00-20:30"
                    ],
                    [
                        "10:00-12:30",
                        "18:00-20:30"
                    ],
                    [
                        "02:00-04:30",
                        "10:00-12:30",
                        "18:00-20:30"
                    ],
                    [
                        "02:00-04:30",
                        "10:00-12:30",
                        "18:00-20:30"
                    ],
                    [
                        "02:00-04:30",
                        "10:00-12:30",
                        "18:00-22:30"
                    ],
                    [
                        "02:00-04:30",
                        "10:00-14:30",
                        "18:00-22:30"
                    ],
                    [
                        "02:00-06:30",
                        "10:00-14:30",
                        "18:00-22:30"
                    ],
                    [
                        "02:00-06:30",
                        "10:00-14:30",
                        "18:00-22:30"
                    ]
                ]
            },
            {
                "date": "2023-01-17",
                "name": "Tuesday",
                "stages": [
                    [],
                    [
                        "16:00-18:30"
                    ],
                    [
                        "08:00-10:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-10:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-10:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-10:30",
                        "16:00-20:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-12:30",
                        "16:00-20:30"
                    ],
                    [
                        "00:00-04:30",
                        "08:00-12:30",
                        "16:00-20:30"
                    ]
                ]
            },
            {
                "date": "2023-01-18",
                "name": "Wednesday",
                "stages": [
                    [
                        "00:00-02:30"
                    ],
                    [
                        "00:00-02:30"
                    ],
                    [
                        "00:00-02:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-10:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-04:30",
                        "08:00-10:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-04:30",
                        "08:00-10:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-04:30",
                        "08:00-10:30",
                        "16:00-20:30"
                    ],
                    [
                        "00:00-04:30",
                        "08:00-12:30",
                        "16:00-20:30"
                    ]
                ]
            },
            {
                "date": "2023-01-19",
                "name": "Thursday",
                "stages": [
                    [
                        "08:00-10:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-10:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-10:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-10:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-12:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-04:30",
                        "08:00-12:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-04:30",
                        "08:00-12:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-04:30",
                        "08:00-12:30",
                        "16:00-20:30"
                    ]
                ]
            },
            {
                "date": "2023-01-20",
                "name": "Friday",
                "stages": [
                    [
                        "16:00-18:30"
                    ],
                    [
                        "08:00-10:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-10:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-10:30",
                        "16:00-18:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-10:30",
                        "16:00-20:30"
                    ],
                    [
                        "00:00-02:30",
                        "08:00-12:30",
                        "16:00-20:30"
                    ],
                    [
                        "00:00-04:30",
                        "08:00-12:30",
                        "16:00-20:30"
                    ],
                    [
                        "00:00-04:30",
                        "08:00-12:30",
                        "16:00-20:30"
                    ]
                ]
            },
            {
                "date": "2023-01-21",
                "name": "Saturday",
                "stages": [
                    [
                        "22:00-00:30"
                    ],
                    [
                        "14:00-16:30",
                        "22:00-00:30"
                    ],
                    [
                        "06:00-08:30",
                        "14:00-16:30",
                        "22:00-00:30"
                    ],
                    [
                        "06:00-08:30",
                        "14:00-16:30",
                        "22:00-00:30"
                    ],
                    [
                        "06:00-08:30",
                        "14:00-16:30",
                        "22:00-00:30"
                    ],
                    [
                        "06:00-08:30",
                        "14:00-18:30",
                        "22:00-00:30"
                    ],
                    [
                        "06:00-10:30",
                        "14:00-18:30",
                        "22:00-00:30"
                    ],
                    [
                        "06:00-10:30",
                        "14:00-18:30",
                        "22:00-00:30"
                    ]
                ]
            },
            {
                "date": "2023-01-22",
                "name": "Sunday",
                "stages": [
                    [],
                    [
                        "22:00-00:30"
                    ],
                    [
                        "14:00-16:30",
                        "22:00-00:30"
                    ],
                    [
                        "06:00-08:30",
                        "14:00-16:30",
                        "22:00-00:30"
                    ],
                    [
                        "00:00-02:30",
                        "06:00-08:30",
                        "14:00-16:30",
                        "22:00-00:30"
                    ],
                    [
                        "00:00-02:30",
                        "06:00-08:30",
                        "14:00-16:30",
                        "22:00-00:30"
                    ],
                    [
                        "00:00-02:30",
                        "06:00-08:30",
                        "14:00-18:30",
                        "22:00-00:30"
                    ],
                    [
                        "00:00-02:30",
                        "06:00-10:30",
                        "14:00-18:30",
                        "22:00-00:30"
                    ]
                ]
            }
        ],
        "source": "https://example.com/test.schedule/current"
    }
}

I have tried with various fields and tags configuration but I am getting nowhere. Here is some of the variations I have tried. I have added line comments on some of them to see if I get the response I would like, but alas.

data_format = "json_v2"
  [[inputs.http.json_v2]]
    measurement_name = "loadshedding"
     [[inputs.http.json_v2.object]]
         path = "schedule.days.0.stages"
          # [[inputs.http.json_v2.tag]]
          #   path = "schedule.days.#.date"
          # [[inputs.http.json_v2.field]]
          #   path = "schedule.days.0.stages.0.0"
          #   rename = "stage 1"
          # [[inputs.http.json_v2.field]]
          #   path = "schedule.days.0.stages.1"
          #   rename = "stage 2"

If I had to take a wild guess at the LP I would like it would look something like this, all the way to stage 8:

Eskom,date=2023-01-16 stage_1="18:00-20:30" stage_2="10:00-12:30", "18:00-20:30" stage_3="02:00-04:30", "10:00-12:30","18:00-20:30"...

Apologies for the incorrect line protocol but its the best way I can explain it.

Thanks in advance.

2

Yeah as you said this is invalid. In order to best help you it would be good to make sure you understand how line protocol works so we both can translate the JSON to correct tags and fields. With JSON data that has arrays of data, it can be a little difficult.

In your suggestion it looks like you are trying to create one big string with all the stages laid out. Unfortunately, I’m not sure how helpful that will be as a metric as how would you generate any interesting aggregations over those strings?

3

Thanks so much for the reply.

My goal is to incorporate it in a graph (grafana) that would notify me of any upcoming outages and changes in schedules. I thought that one long string would be easier to query as it is associated with a particular date and stage. My difficulty is that some stages have multiple times in one day, and the only way I thought to solve it would be tagged by date. I just can’t get it to work.

If there is a better way to do this I am all ears. I just couldn’t think of any. As my background is far relating from IT, this is a weekend hobby for me. Will definitely have a look at LP, thank you.

Again I appreciate all the assistance.