Influxdb 2.1 - using timeshift() function consumes all memory

InfluxDB 2
1

Hi everyone,

I recently switched from a raspberry pi 3 with influxdb 1.8 to a raspberry pi 4 (8GB) and moved to Influxdb 2.1 in a docker container. Influxdb is used by my home automation, openhab3 and grafana.

In Grafana I have a graph displaying 3 values.

  • Current Outside temperature
  • Max/Min value of the temperature prognosis in 3 days.

The max/min value is timeshifted by 3 days so that I get a graph where I can compare the current temperature value to the prognosis 3 days ago as well as look 3 days ahead.

The queries look as follows:

Current temperature

from(bucket: v.defaultBucket)
  |> range(start: -4d, stop:3d)
  |> filter(fn: (r) =>
    r._measurement == "OutsideSensor_ActualTemperature"
  )
  |> set(key: "_measurement", value: "")
  |> set(key: "_field", value: "Temperatur")

Max/Min value

from(bucket: v.defaultBucket)
  |> range(start: -10d, stop:3d)
  |> timeShift(duration:72h)
  |> filter(fn: (r) =>
    r._measurement == "WeatherService_ForecastDay3_MaxTemperature"
  )
  |> set(key: "_measurement", value: "")
  |> set(key: "_field", value: "Temperatur Max")

The queries and the graph worked very well on my raspberry pi3 with 1GB RAM for over 2 years.

In the new setup, when the query is issued, Influxdb2.1 consumes all the 8GB memory within seconds and a 100% CPU load. Grafana shows an http error 504 “Gateway Time-out” and Influxdb logs only the following warning:

lvl=warn msg="internal error not returned to client" log_id=0Y4i6gMG000 handler=error_logger error="context canceled"
From that point the memory is never freed again and the raspberry pi becomes almost irresponsive.

Any ideas ?

PS.: For the temperature values there are datapoints taken every 5 minutes or when changed. So there should not be to many information to shift.

2

Alright, I found the issue in plain sight.
Of course it is rather stupid to timeshift before filtering any value and therefore apply the timeshift to the whole bucket :wink:

So

from(bucket: v.defaultBucket)
  |> range(start: -10d, stop:3d)
  |> filter(fn: (r) =>
    r._measurement == "WeatherService_ForecastDay3_MaxTemperature"
  )
  |> set(key: "_measurement", value: "")
  |> set(key: "_field", value: "Temperatur Max")
  |> timeShift(duration:72h)

Works just fine.
However with Influxdb 1 also the first variant was possible without any crashes. Probably something which could be more robust in Influxdb 2