Connect flux to other TSDB

Fluxlang
1

Hi guys,

I am currently testing flux to see how the language can help user to build Time series scripts.
I know that it is not one of your priority, but I am searching to an hint on how we can develop a few functions allowing us to load data from an other storage tool than Influx and then manipulate them using flux.

Best

2

Itā€™s still pretty early so I donā€™t have great things to point you to. Last month I prototyped adding support for using Prometheus as a back end for Flux. You can see that work here: https://github.com/influxdata/platform/compare/pd-from-prometheus?expand=1

However, weā€™ve since moved Flux to its own repo here: GitHub - influxdata/flux: Flux is a lightweight scripting language for querying databases (like InfluxDB) and working with data. It's part of InfluxDB 1.7 and 2.0, but can be run independently of those.

Basically, at a minimum youā€™ll need to implement a new source, reader, and the from function. Youā€™ll also want to update the pushdown logic for the range and filter functions so that at least those get translated to be pushed down whatever database youā€™re querying.

My guess is that weā€™ll be refactoring how the sources work so that its easier to add any kind of source. This initial version of the code was only concerned with connecting to InfluxDB. Also, we already know that weā€™re going to be rewriting the query planner and pushdown logic.

So Iā€™d say itā€™s early to jump in and add another source unless youā€™re willing to update it and possibly even help us refactor things so we can hook it up to any source (or provide feedback). Itā€™s a goal of Flux to be able to connect to any data source, but weā€™re probably at least a few months out from getting to that.

1 Like
3

First thanks a lot for your answer!
I will have a close look at your work for Prometheus and I will see afterwards.
As we are working with many Time series protocols, we have a good use case to play with flux and mixing data-sources, even if itā€™s quite early.
We will be happy to provide any feedback and/or contributions.

4

I manage to make it work to connect Flux to Prometheus and our LTS backend, re-using some of your code. It looks really promising. As soon as I have some time, I will propose a generic approach for some data sources.

The only drawback for the moment, is that I didnā€™t manage to make it work with the moved Flux repository.

5

Hi Aurrel,
I have some great news to share on this front. Last week we had an engineering all hands and a hack day. One of the projects that came out of that was a more standardized interface for adding new data sources. Thereā€™s still more work to be done and I donā€™t think itā€™s handling the pushdown logic for range and filter, but thereā€™s a good start. Check out this pull request here:

Any feedback you can give for your specific case or even adding a Prometheus connector using this new code would be very helpful. Thanks!

Paul

6

Hello Paul,
Sorry for the late answer, I was off last week. This looks like a great improvement on that sense. I will test to implement a new data-source this week. Thanks for the update.
Aurelien

7

Hi,
I had the time to look on the result of your hack day. Thanks a lot for the work done! It really provides a generic way to add custom TSDB as Flux data-sources. As a result, I wrote the following code to connect it to a Prometheus backend:

What would be the best way to catch an error in the Range function?

If you are interested, I can open a Pull Request. I would like to add also the following missing features:

  • Handle a metric name selector (as Matcher)
  • Handle filtering on metrics tags
  • Add a prometheus provider to manage a basic auth
  • Whitelist prometheus hosts on flux config
8

Hi updated my PR on Prometheus to match the last update and our need. If you wanna review it: feat(inputs): add fromProm methods by aurrelhebert Ā· Pull Request #202 Ā· influxdata/flux Ā· GitHub