Storing high-frequency (10kHz) data

Telegraf
1

Hello,

I am evaluating databases to collect, store and analyze high frequency sensor data. The requirement is to store different time series (around 500) each with a sampling rate of 10kHz. Can anyone tell if this is feasible with InfluxDB and what I have to consider? Can InfluxDb handle this amount of data (~500 billion rows / day) and what performance can I expect when I query some timeframe and aggregate the data?

Thanks! BR Hans

2

Hi Hans,

I think that, depending on your InfluxDB setup, you should be able to handle this load without a problem. What I mean by “depending on your setup” is the number of nodes in your cluster, amount of memory, etc.

Query performance is going to be an “it depends” answer because that will depend on a number of factors including the number of fields and tags in your query, the timeframe being queried, etc.

Hope this helps get you at least started, though if you’d like to post more details of the kind of data, number of measurements, tags, and fields, and the typical types of queries we can probably start getting a bit more precise in the answer.

Best regards,
dg

3

It seems to me that influxdb can currently only process individual timeseries using a single thread (correct?) per series. For high-frequency data I guess this can pose a problem, since you quickly get long series?

4

Hey Hans,

was there ever a conclusion? Did you end up using influxdb? If so, how did it work out?

Thanks for sharing your experiences
Philip

5

I am also curious about findings here. Any solution and metrics?

6

Let use mdsplus and MARTe2.
USING marte2 framework you can easly achive It on Linux. An High prio thread isolated on a CPU that each ms take 10 samples from the data source and store them on mdsplus pulse file USING a decoupled thread USING a circolare buffer.