Hi, we have a stream in Kafka with batched and compressed time series data. In an application we need to load as fast as possible ca. 5256000 datapoints in to main memory and process them there. We are evaluating writing the data in Kafka to influxdb and fetching the 5256000 datapoints from there. Is this a good fit for influxdb fetching that much data?
Thank you.
@Fredrik_Skeel_Lokke That sounds like a good potential usecase for Influx. I would think that Redis, Memcached or another caching layer might be a better fit however. Do you need on-disk persistence and long term storage for this data too?
Yes, having long term storage with really good compression would simplify the solution. That was what initially drew our attention toward influxdb. What I’m unsure of is it’s ability to move a lot of datapoints… I was guessing that it might be optimized for moving fewer points but being easy to integrate with, e.g. the usage of json and http…
@Fredrik_Skeel_Lokke It is very easy to integrate with. It is more designed to serve the results of aggregate queries for graphing which is the reason for the JSON/HTTP design. The query you are describing would return in ~1s. If you broke it up into multiple queries you could probably reduce that time. is that an acceptable amount of latency?
