Can't do this with grafana, can I do this with telegraf? Hello @Anaisdg

Hello @Anaisdg
The problem: update an operations status display that is 10 years-old python using xmlrpc, and writes to html (haven’t analyzed the html part yet, might be pjamas, might be django). a dozen parameters to display at at 1 hz rate. The data right now is dumped after display, if it had a need for retention, it would be at most a 2 week data retention time.
The goal: go from what is now, to something newer with more features.
Plot data in addition to displaying current values.
I studied prometheus, telegraf, and grafana.
prometheus: too complex for my problem
telegraf: at the point I was ready to do something, I also reached the point in the tutorial that said: for more info, contact sales. Thanks but no thanks.
grafana: simple to use, tutorial was helpful

We had been looking at redis for a number of years and considered the redis interface to grafana, the redis time series is new, and has a retention time. At this point I started coding. When I was ready to move from data collection to display, I ran into problems with grafana.
The following is copied from an email I sent to my boss, I apologize in advance for the formatting issues in copy-and-paste to this forum:

I successfully scraped seven xmlrpc servers into various redis time series and redis streams, depending on the type of data:

floats and integers -> redis time series
strings and composites fields of numbers and strings -> python dictionary composite -> redis streams

The next step, replicating the operations page in grafana I ran into a number of problems, all have workarounds, but some of the workarounds may be complex, or unworkable, or undesireable. My thinking is as follows:

  1. grafana doesn’t recognize redis streams.
    (streams is composite data in a time series)
    the redis people haven’t yet created a streams-to-grafana interface and offered to let me to create the interface this if I wanted to.
    I replied I look into it when I was ready, but that might take a few weeks before I was ready.

  2. Even if grafana recognized streams, grafan does not recognize a time series of the string data type, i.e. “on”, “off”, a string mode description, etc. and this is MOST of the existing operations panel

    However, grafana can map a time-series of integers to a strings, then print the strings as a table. This is not the same format as the operations panel but replicates its function.

examples here: https://grafana.com/docs/features/panels/table_panel/
and here: https://grafana.com/grafana/plugins/yesoreyeram-boomtable-panel

  1. To do the above, I would, as part of data scraping, need to convert the strings to a time series of integer codes prior to redis, then convert the codes to string after importing the data into grafana per a grafana panel definition.

    The problem I see is, as the timing modes parameter seem to be an arbitrary string, I need a complete mapping, on the fly, inside grafana.
    However the grafana panel setup at this point, appears to be static, created by the designer prior to run-time use.
    This also means I’d have to look at the internals of all the servers to see all possible strings for all possible mappings, for example the wg switch error codes parameter. This could be done but does not simplify operations evolution over time, any
    change to status would require reworking the display code.
    I would hope displays would remain independent from data displayed but mapping strings to codes and back to strings does not follow this paradigm.

An alternative would be to use grafana only for time-series integer and float data, and use html (as is done now) for the string fields. For example, use time-series for the parameters of power, system temperature, azimuth, voltage, current, etc, and use html for everything that can’t be represented as a time-series.

My concern with a hybrid approach, is the result will be multiple web pages and the ability to view system state at-a-glance will be lost. Of course I could go directly from redis to html and cut out grafana entirely, but then I’d lose the time-series graphing.

Any help with solving this problem in Telegraf, in a single page, would be appreciated.

Please note that MIT Haystack Observatory is a non profit with limited funds towards software tools purchases.

thank you,
bob s.

Hello @robert_schaefer,

Welcome and thanks for your question :slight_smile: . No worries, funds or not we’re here to help. I’m a little confused though.
For the sake of clarification and simplicity, can you please provide me with some examples of points you’re trying to write and what you’d like to see?
Have you tried using chronograf?
Have you checked out the redis input plugin?
Maybe Giraffe, a React-based visualization library used to implement the InfluxDB 2.0 UI could help you?
Have you explored Flux at all?

Thanks,
Anais

I am not surprised that you are confused. Basically I was asked to contact you through this forum by joseph enno (who I assume is a sales representative).

I am not currently using redis-telegraf, I am using redis-grafana. I have not studied Flux, I have studied redis.

The data I’ve got is mostly key-value pairs, and only recently have we considered the time-series form.

That is, I will need to handle both kinds of data on the same web page.

On the grafana side, the problem is that grafana only supports time series, it doesn’t support non-time oriented data such as label+variable field.

When I researched Telegraf, the tutorial provided was, not to be insulting, light weight. When it got to the point where I’d like to create and run an example on my own, the tutorial said: for more information please contact a sales representative.

And Flux, I believe from following the link you provided, is also solely time series.

After thinking a bit, I am now investigating the Dash tool (based on Flask and Python) for label+field oriented data, and using Dash invoke grafana for time-oriented data.

Two stacks to code, but each stack can do what it does best.

Hello @robert_schaefer,
I think you have a few concepts confused:
-Telegraf and Grafana are unrelated. Telegraf is a plugin driven collection agent. Grafana is a visualization dashboard. Telegraf is part of Influx. Grafana isn’t.
-Flux is a functional scripting language that you can use to query Influx, write functions, do data analytics, etc…For example, you could use flux to assign codes to different logs if you wanted. Flux and Redis are unrelated.
-If you follow the link I shared with you for Telegraf (I specifically shared the redis input plugin on git), there isn’t any option to contact sales. Here are the links again: github and documentation