We record time series with values and, when a parameter goes out of range, we record an additional logId field that we use to link to our logging database so we can pull up more details for cases that are overtime. i.e. we have series like:
11/08/18 17:30:01 apdexScore=1 time=200
11/08/18 17:40:07 apdexScore=1 time=250
11/08/18 17:45:01 apdexScore=0 time=1200 logid=inf_abdf32a1
11/08/18 17:49:43 apdexScore=1 time=210
I tried visualizing this in Grafana, and the answer appears to be that you have to create two different dashboard items, a table and a graph, that are disconnected. You can, alternatively, create “annotations queries” - that involves discovering to a whole different area of the UI, it requires making two distinct (yet very similar queries), and then it doesn’t display well - it draws lots of vertical lines all over the graph data.
I thought Chronograf might support this better. Sure enough, I can see the fields I want right there on the UI, which is nice. I can click on them - great. However, as soon as I click on the text field, I get an error because you cannot mean() a label - so I changed the mean() to a first() and now I get an error “The data returned from this query can’t be visualized with this graph type” with the suggestion that I Switch to a Table Graph. So close…
The killer feature, for me, would be a graph tool that natively supports visualizing data that combines mixed fields, including string fields, and not only numbers. After all, its one of the things InfluxDb supports out of the box as part of its line protocol. Its disappointing that the visualization tool just gives up and presents users with such a black and white “either/or” option - either a table or a graph.
Even if a field is textual, you still have a time value - so you have an X axis, and moreover this time value aligns with other fields that may be numeric. What I want, in a case where I select multiple fields, is for the Graph viewer to recognize textual fields, default to a selector such as “first non-null”, and create “marker rows” in the visualization for drawing markers when those fields have values (ideally with the ability to filter which values you care about).
I say this because we do something similar with our scheduling tool, which looks something like this:
If these marker rows appeared beneath any data that is graphed, but using the same timescale, then I could then mouse over the graph and, in the cases where there is a marker, the pop-up tooltip would show the numeric and text values together. Boom. Ideally if I clicked on a marker I would see the log data from ElasticSearch pulled up in a separate pane, but I’d settle for a way to be able to copy/paste the Id so I can search for it in Kibana.
I think along similar lines in terms of the Table view. I don’t want to be forced into choosing either a graph or a table. I want a linked view, where the top half shows my graph, and the bottom shows a datagrid, and clicking on places in the graph navigates to the appropriate rows in the data grid, and scrolling in the datagrid moves a cursor on the graph. I imagine a combination visualizer that has the graph+markers in the top area and a datagrid in the bottom with a splitter bar between them, and options to make one or the other full-screen. This to me is the difference between a data visualizer and a bunch of dashboard widgets.
I’m posting this here because it looks like there’s energy around improving Chronograf and making it more of a data visualizer. If there was a plugin api and an example to start from, I’d probably have a stab at something like this myself. Any comments about how feasible it is?