Hello @Anaisdg
The problem: update an operations status display that is 10 years-old python using xmlrpc, and writes to html (haven’t analyzed the html part yet, might be pjamas, might be django). a dozen parameters to display at at 1 hz rate. The data right now is dumped after display, if it had a need for retention, it would be at most a 2 week data retention time.
The goal: go from what is now, to something newer with more features.
Plot data in addition to displaying current values.
I studied prometheus, telegraf, and grafana.
prometheus: too complex for my problem
telegraf: at the point I was ready to do something, I also reached the point in the tutorial that said: for more info, contact sales. Thanks but no thanks.
grafana: simple to use, tutorial was helpful
We had been looking at redis for a number of years and considered the redis interface to grafana, the redis time series is new, and has a retention time. At this point I started coding. When I was ready to move from data collection to display, I ran into problems with grafana.
The following is copied from an email I sent to my boss, I apologize in advance for the formatting issues in copy-and-paste to this forum:
I successfully scraped seven xmlrpc servers into various redis time series and redis streams, depending on the type of data:
floats and integers → redis time series
strings and composites fields of numbers and strings → python dictionary composite → redis streams
The next step, replicating the operations page in grafana I ran into a number of problems, all have workarounds, but some of the workarounds may be complex, or unworkable, or undesireable. My thinking is as follows:
-
grafana doesn’t recognize redis streams.
(streams is composite data in a time series)
the redis people haven’t yet created a streams-to-grafana interface and offered to let me to create the interface this if I wanted to.
I replied I look into it when I was ready, but that might take a few weeks before I was ready. -
Even if grafana recognized streams, grafan does not recognize a time series of the string data type, i.e. “on”, “off”, a string mode description, etc. and this is MOST of the existing operations panel
However, grafana can map a time-series of integers to a strings, then print the strings as a table. This is not the same format as the operations panel but replicates its function.
examples here: Table | Grafana documentation
and here: Boom Table plugin for Grafana | Grafana Labs
-
To do the above, I would, as part of data scraping, need to convert the strings to a time series of integer codes prior to redis, then convert the codes to string after importing the data into grafana per a grafana panel definition.
The problem I see is, as the timing modes parameter seem to be an arbitrary string, I need a complete mapping, on the fly, inside grafana.
However the grafana panel setup at this point, appears to be static, created by the designer prior to run-time use.
This also means I’d have to look at the internals of all the servers to see all possible strings for all possible mappings, for example the wg switch error codes parameter. This could be done but does not simplify operations evolution over time, any
change to status would require reworking the display code.
I would hope displays would remain independent from data displayed but mapping strings to codes and back to strings does not follow this paradigm.
An alternative would be to use grafana only for time-series integer and float data, and use html (as is done now) for the string fields. For example, use time-series for the parameters of power, system temperature, azimuth, voltage, current, etc, and use html for everything that can’t be represented as a time-series.
My concern with a hybrid approach, is the result will be multiple web pages and the ability to view system state at-a-glance will be lost. Of course I could go directly from redis to html and cut out grafana entirely, but then I’d lose the time-series graphing.
Any help with solving this problem in Telegraf, in a single page, would be appreciated.
Please note that MIT Haystack Observatory is a non profit with limited funds towards software tools purchases.
thank you,
bob s.
. No worries, funds or not we’re here to help. I’m a little confused though.
