I’m designing a system for quick and efficient exploration (visualisation) of large amounts of financial time series.
I’ll describe shortly my naive design below and would really like to hear comments on how it can be improved (performance, flexibility,…) from knowledgable Influx users and people who have done similar work before.
Data is downsampled. A separate measure for each time period:
raw - raw data points with nanosecond resolution
seconds - 1 point per second
minutes - 1 point per minute
hours - 1 point per hours
days - 1 point per day
weeks - 1 point per week
months - 1 point per month
years - 1 point per year
On a 600 points chart canvas display the points for the following period: 2018-01-16 12:12:12 - 2018-09-16 12:12:12
Naive algorithm pseudo code:
for each period in [years, months, weeks, .., raw] points_count = select count(1) from period if points_count >= 600 or period.is_raw() points = select fields from period where time between start and end aggregated = aggregate(points, 600) return aggregated
Is this a worthy solution?
Any ideas how to improve?