[question] Influxdb RAM usage

Store
1

Currently we are using Influxdb on our testing server. We are pushing data from Telegraf( after every 10 sec), Netdata(10 seconds) and some data about vehicles from our django app( after every 1 min).

I have been using it for like 3-4 weeks now. But now, it is causing problems for us. The server has very low RAM( 1675 M ), and influxdb is taking more and more RAM day by day.

It usually takes between 500-800 mb. But today, influxdb crashed and did not start as it goes OUT OF MEMORY and cannot be allocated any memory. The RAM usage risen to 900mb and it was having trouble reading the next WAL files( I checked the log files ), and it could not start. We freed some RAM, and it started , and is taking around 1200 mb of RAM.

This question is important for architecture of our next server which we are going to use in production, so please direct me in the right way. We will be using influxdb for the same processes as defined above.

So, my question is that Does the RAM usage of influxdb will keep increasing if I keep pushing more and more data day by day ? It will increase for sure, but any idea of the maximum amount of RAM which can be occcupied by Influxdb if I have to push and store data for 2, 3 years ?

This is the /var/lib/influxdb directory’s size currently,

alphauser@AlphaServer:/var/lib/influxdb$ sudo du -hsc  *
329M	data
8.0K	meta
91M	wal
419M	total
2

@Luv Memory usage is primarily dependant on the number of series. So the more unique tags you push every day the more memory the server will use. For a small box like that I would suggest having a set number of series. That is a very low amount of RAM to be allocating for the database.

3

@jackzampolin

Thank You, I think we will not be adding more tags now. And Also we used this server just to test the influxdb database that whether it suits our need or not.

We will increase RAM definitely for our next server and I think the maximum influxdb can occupy is 2500mb.

During start, the memory usage is maximum. After that when it has been running for like 10 minutes, the memory usage is almost halved. ( 1200mb to 650mb)

4

@Luv Also make sure you are monitoring RSS as opposed to memory used/free. We use mmaping to cache reciently used data and allow the OS to take care of garbage collection. This leads to misleading memory readings.

1 Like
5

Oh I see !!

It almost takes up 2gb of the Virtual Memory.

1 Like