Downsampling utility open source

#1

Hello Everybody,

I’ve created a small project that helps me transform InfluxDB data.
It’s open source and written in Rust.
It can create downsampled series from other series and can split a single measurement with many series into many single-series measurements.
Feel free to use for your needs.
PRs and code reviews are more than welcome!

Thanks,
Michael

#2

Finally something interesting here. How does it work? Maybe you can put something like an example on the github page?

Are you doing this ‘transparent’ in the same database? Or also with retention policies. I was thinking of using downsampling via the kapacitor.

#3

Hey @f1outsourcing!

I’m glad that you found the project interesting.

The basics to get up and running are:

  1. Install Rust (to build the project): https://rustup.rs/
  2. git clone git@github.com:michaelr524/downsampler.git
  3. cd downsampler
  4. cargo build --release
  5. Edit config.toml to your needs
  6. Run like this: downsample -s ‘2018-08-13 15:22:06’ -e ‘2018-08-17 19:03:17’

This command will downsample the given period and exit. I’m currently working on adding continuous integration. When it’s ready, I’ll also add some more documentation. Probably in the next few days.

It reads from existing series and creates or adds to new ones with a similar name. It’s configurable via config.toml.

It’s currently using this algorithm for downsampling: https://github_com/sveinn-steinarsson/flot-downsample

Since it’s written in Rust, it consumes a tiny amount of RAM and is supposed to be very fast. That said, there is a lot of space left for further optimization.

Thanks,
Michael

#4

Sorry, I wrote continous integration, but meant to write continous downsampling :slight_smile:

#5

It’s there. For anyone that wants to try it out.