Influxd backup impact memory even after it is finished


We are running an Influxdb instance on a server along with kapacitor tasks and other specific python code.

We observe memory peak every day around the same time (between 12:00 and 12:30 UTC+8).

It might be that our configuration is too small but from my understanding we are under the minimal performance needs and our configuration meets these needs (see configuration below).

I would love your help to understand the following points that I can not explain:

  • If I stop influxdb instance but let running the rest of our stack, these memory peaks stop
  • If I stop nightly backup of our instance, these memory peaks stop (they are planned in a crontab at 2:30 so 10h before and finish correctly according to logs).

Here the CPU peaks shows when the backup is running

When the cron job is disabled, we don not observed the memory usage peaks

I imagine that all sharding process or other internal process are planned regularly by influxdb but I cannot understand how the nightly backup could influence this behaviour.

I can reproduce this problem on a second period and on another cloud instance form another provider.

Any help would be gladly appreciated :slight_smile:

Server configuration

  • Cloud provider: Aliyun
  • Ubuntu 16.04
  • 2vCPU, 8 GB memory
  • We are under minimal conditions here in terme of series, write per second and number of database