Setup question - how to combine offline and online data


#1

Hi,
I need some help with setting up my system correctly. The main issue I have right now is how I cope with online and offline data.
So:
I have process that is being monitored online. Everything I want is dumped in influxdb, visualisation is currently grafana.

The process sensors give continous output, however the process is run in batches.
At the start of the batch a couple of parameters are written down (unable to collect those with influx).

However I would need these for calculations during the process/batch.
What is the best approach for doing this?

Things I think about:

  • adding written down measurement as a measurement to the influx database, using some kind of online form (can this be done in chronograf/grafana?). And then use a combination of two Tickscripts: A stream for continous calculation, and a batch script to recalculate (if the written down values are only inserted during a running batch).
  • What can be done with the annotations? Can I extract data at the timepoint of the annotation?

#2

There are several methods to do that ,
You can for example execute a script launched by Telegraf , use python , and much more …

Once uploaded in Influxdb you can use 1 tick script

Can you explain a bit more about the annotations ?


#3

Hi,
One of the issues I have is that the written down data is often not complete at a certain time point and I probably will have to do multiple write and deletes. This might also influence the calculations. I would like to have continuous data processing, but I will have to use a combination of stream and batch scripts then: Stream to calculate continuously (albeit the written down variable might be wrong) and the batch to correct afterwards (if needed)… And this is already pretty complex…
I tend to use a second database for this (postgres or so) or would I be better off dumping everything in influx?

I think in grafana and chronograf you can set annotations on the grafs. This could be nice to indicate the batch timings (let’s say start and end) on the continuous data. However this does not go the database, I guess? And thus can not be used in TICKscript?

Can I call a postgres database in a TICKscript? I guess not?