Memory consumption leading to queries timeout

InfluxDB 2
1

Hi guys,
I don’t know if what I want is currently possible so please if not just let me know why.

My situation is I have an InfluxDB container running on a very small machine (4Gb of RAM and 16Gb HD). The thing is as I have multiples containers on this machine I needed to limit the amount of RAM that each one of them is using and then my InfluxDB is running with a limit of 1.5Gb of memory.

When I do a fresh start everything runs fine, the problem start happening after some time of test. So after a 14 days test the influxDB starts to return timeout error for almost every query that I tried to run [eg. SELECT * FROM measurement LIMIT 1].

The first thing I checked was the cardinality using the show series cardinality and the result is that my cardinality is just 2.

After I checked the memory consumption and it is taking 1.4Gb from the 1.5Gb.

And to finish I also checked the size of my influxDB and using
du -sh /var/lib/influxdb/data/*
it seems that it is around 12Gb and using

select max(sum)/(1024 * 1024) from (select sum("diskBytes") as sum from (select diskBytes FROM "_internal".."tsm1_filestore" WHERE "database"='measurement' group by id, numFiles, "diskBytes" ORDER BY time DESC limit 1) group by time(1m) ORDER BY time DESC) ORDER BY time DESC

the result is 25Gb (here I must have make any mistake because 25Gb is bigger than my current disk :sweat_smile: ).

Right now I am trying to reduce my retention policy to 10days instead of the current 1 Month value to see if works. But this is a blind test as I am not sure if it makes sense to reduce that in order to have a working container again. If someone has any idea please let me know.

Thanks in advance

2

Hello @gabrielLira ,
Welcome! It’s important to remember, InfuxDB will use whatever memory is available to it in order to optimise reads and writes.

If you wish, you can try to create a cgroup to restrict it? Additionally what version of InfluxDB are you using? Your series cardinality is so small that using TSM shouldn’t be a problem, but you might want to make sure you have TSI. Time Series Index (TSI) details | InfluxDB OSS 1.8 Documentation

3

Hi @Anaisdg,
First, thanks for replying.
Currently I am using the version 1.7.3.
About the CGroup. I don`t know if my understanding about it is right but if it is the restriction of resources, I have already done this to restrict the influxDB to just consume 1.5Gb. So when I had this problem the others services kept running.

And about the TSI, thanks for the advice I will take a look on it for sure.

Ah and just to keep it up to date. It seems that having a RP of 10 days helped my influxDB container to keep running. So as you said the memory is completly used but at least until today it is responding to all of my queries. :slight_smile:

1 Like