I am using InfluxDB 2.0.4 and Telegraf 1.21.
Everything was working fine suddenly InfluxDB started consuming too much memory and this is affecting the overall system performance.
I checked in process Influxd is consuming too much memory.
As you can see Influxd is taking too much memory.
I checked in docker stats as well InfluxDB is consuming too much memory.
After all this, I am not sure where to look why suddenly InfluxDB behaving like this.
is there any specific process or files which I should look for which is putting too much load on the system?
Thanks
Hi @Ravikant_Gautam,
Could you send the InfluxDB logs using the following command:
sudo journalctl -u influxdb.service
I would also advice updating to the latest version of InfluxDB OSS 2.2.
ts=2022-04-19T08:25:10.735948Z lvl=info msg="Beginning compaction" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=8
ts=2022-04-19T08:25:10.735952Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000025-000000001.tsm
ts=2022-04-19T08:25:10.735958Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000026-000000001.tsm
ts=2022-04-19T08:25:10.735963Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000027-000000001.tsm
ts=2022-04-19T08:25:10.735967Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000028-000000001.tsm
ts=2022-04-19T08:25:10.735971Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000029-000000001.tsm
ts=2022-04-19T08:25:10.735975Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000030-000000001.tsm
ts=2022-04-19T08:25:10.735980Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000031-000000001.tsm
ts=2022-04-19T08:25:10.735984Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000032-000000001.tsm
ts=2022-04-19T08:25:10.834673Z lvl=info msg="Compacted file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000032-000000002.tsm.tmp
ts=2022-04-19T08:25:10.834707Z lvl=info msg="Finished compacting files" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-04-19T08:25:10.834713Z lvl=info msg="TSM compaction (end)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=end op_elapsed=98.809ms
ts=2022-04-19T08:25:11.735813Z lvl=info msg="TSM compaction (start)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=2 tsm1_strategy=level op_name=tsm1_compact_group op_event=start
ts=2022-04-19T08:25:11.735848Z lvl=info msg="Beginning compaction" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=2 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=4
ts=2022-04-19T08:25:11.735853Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=2 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000008-000000002.tsm
ts=2022-04-19T08:25:11.735857Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=2 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000016-000000002.tsm
ts=2022-04-19T08:25:11.735861Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=2 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000024-000000002.tsm
ts=2022-04-19T08:25:11.735865Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=2 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000032-000000002.tsm
ts=2022-04-19T08:25:11.894161Z lvl=info msg="Compacted file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=2 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/root/.influxdbv2/engine/data/d1b0ebee955026ac/autogen/3221/000000032-000000003.tsm.tmp
ts=2022-04-19T08:25:11.894196Z lvl=info msg="Finished compacting files" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=2 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-04-19T08:25:11.894202Z lvl=info msg="TSM compaction (end)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=2 tsm1_strategy=level op_name=tsm1_compact_group op_event=end op_elapsed=158.395ms
ts=2022-04-19T08:27:49.055269Z lvl=info msg="Cache snapshot (start)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=start
ts=2022-04-19T08:27:49.220937Z lvl=info msg="Snapshot for path written" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172 duration=165.685ms
ts=2022-04-19T08:27:49.220990Z lvl=info msg="Cache snapshot (end)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=end op_elapsed=165.737ms
ts=2022-04-19T08:27:50.055760Z lvl=info msg="TSM compaction (start)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=start
ts=2022-04-19T08:27:50.055795Z lvl=info msg="Beginning compaction" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=8
ts=2022-04-19T08:27:50.055800Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172/000000361-000000001.tsm
ts=2022-04-19T08:27:50.055804Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172/000000362-000000001.tsm
ts=2022-04-19T08:27:50.055808Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172/000000363-000000001.tsm
ts=2022-04-19T08:27:50.055812Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172/000000364-000000001.tsm
ts=2022-04-19T08:27:50.055818Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172/000000365-000000001.tsm
ts=2022-04-19T08:27:50.055823Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172/000000366-000000001.tsm
ts=2022-04-19T08:27:50.055829Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172/000000367-000000001.tsm
ts=2022-04-19T08:27:50.055835Z lvl=info msg="Compacting file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172/000000368-000000001.tsm
ts=2022-04-19T08:27:52.187630Z lvl=info msg="Compacted file" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172/000000368-000000002.tsm.tmp
ts=2022-04-19T08:27:52.187677Z lvl=info msg="Finished compacting files" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group tsm1_files_n=1
ts=2022-04-19T08:27:52.187683Z lvl=info msg="TSM compaction (end)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 tsm1_level=1 tsm1_strategy=level op_name=tsm1_compact_group op_event=end op_elapsed=2131.929ms
ts=2022-04-19T08:36:22.316438Z lvl=info msg=Unauthorized log_id=0_o61YpG000 error="session not found"
ts=2022-04-19T08:37:21.055270Z lvl=info msg="Cache snapshot (start)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=start
ts=2022-04-19T08:37:21.214805Z lvl=info msg="Snapshot for path written" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172 duration=159.552ms
ts=2022-04-19T08:37:21.214857Z lvl=info msg="Cache snapshot (end)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=end op_elapsed=159.602ms
ts=2022-04-19T08:42:37.055249Z lvl=info msg="Cache snapshot (start)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=start
ts=2022-04-19T08:42:37.209719Z lvl=info msg="Snapshot for path written" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172 duration=154.482ms
ts=2022-04-19T08:42:37.209767Z lvl=info msg="Cache snapshot (end)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=end op_elapsed=154.529ms
ts=2022-04-19T08:42:46.998095Z lvl=info msg="Retention policy deletion check (start)" log_id=0_o61YpG000 service=retention op_name=retention_delete_check op_event=start
ts=2022-04-19T08:42:46.998836Z lvl=info msg="Retention policy deletion check (end)" log_id=0_o61YpG000 service=retention op_name=retention_delete_check op_event=end op_elapsed=0.757ms
ts=2022-04-19T08:55:18.055270Z lvl=info msg="Cache snapshot (start)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=start
ts=2022-04-19T08:55:18.229236Z lvl=info msg="Snapshot for path written" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172 duration=173.984ms
ts=2022-04-19T08:55:18.229282Z lvl=info msg="Cache snapshot (end)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=end op_elapsed=174.030ms
ts=2022-04-19T09:02:02.055266Z lvl=info msg="Cache snapshot (start)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=start
ts=2022-04-19T09:02:02.243658Z lvl=info msg="Snapshot for path written" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/root/.influxdbv2/engine/data/3660481e9ba72191/autogen/3172 duration=188.404ms
ts=2022-04-19T09:02:02.243784Z lvl=info msg="Cache snapshot (end)" log_id=0_o61YpG000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=end op_elapsed=188.527ms
Hmm no tell-tell signs within the logs. Just seems to be compacting series. So my advice would be to update, see if the problem persists if so have you:
- Added a new datastream which could be writing large amounts of data in realtime
- Running a series of complex queries
- Any scrapers which are failing to pull data.
