Tracking down usage growth on influxDB 2.x

InfluxDB 2
1

Hi All, I am trying to track down what might be going on with a sudden acceleration of disk usage for InfluxDB v2.7.6. Normally things hovered around 60% usage and then all of a sudden started growing at a faster pace 2 days ago. We didnt add new clients that I know of and I dont see any particular one adding a lot more looking at traffic graphs to influxDB. It seems to coincide with an update on May 9th after a kernel update (ubuntu 20 LTS). There is the odd

May 11 00:27:22 influxdb. influxd-systemd-start.sh[872]: ts=2024-05-11T00:27:22.117835Z lvl=error msg="Expired phantom shard detected during retention check, removing from metadata" log_id=0p3lSOcW000 service=retention op_name=retention_delete_check db_instance=7a038e6d102418c6 db_shard_id=1791 db_rp=autogen
May 11 00:27:22 influxdb. influxd-systemd-start.sh[872]: ts=2024-05-11T00:27:22.119124Z lvl=error msg="Expired phantom shard detected during retention check, removing from metadata" log_id=0p3lSOcW000 service=retention op_name=retention_delete_check db_instance=7a038e6d102418c6 db_shard_id=1073 db_rp=autogen
May 11 00:27:22 influxdb. influxd-systemd-start.sh[872]: ts=2024-05-11T00:27:22.120244Z lvl=error msg="Expired phantom shard detected during retention check, removing from metadata" log_id=0p3lSOcW000 service=retention op_name=retention_delete_check db_instance=7a038e6d102418c6 db_shard_id=1516 db_rp=autogen
May 11 00:27:22 influxdb. influxd-systemd-start.sh[872]: ts=2024-05-11T00:27:22.121350Z lvl=error msg="Expired phantom shard detected during retention check, removing from metadata" log_id=0p3lSOcW000 service=retention op_name=retention_delete_check db_instance=7a038e6d102418c6 db_shard_id=1541 db_rp=autogen
May 11 00:27:22 influxdb. influxd-systemd-start.sh[872]: ts=2024-05-11T00:27:22.122487Z lvl=error msg="Expired phantom shard detected during retention check, removing from metadata" log_id=0p3lSOcW000 service=retention op_name=retention_delete_check db_instance=7a038e6d102418c6 db_shard_id=1343 db_rp=autogen

message in syslog but that seems to be after the growth started.
Any idea how to track this down ? I added some more disk space today so thats the sudden drop this AM, but the growth seems to be accelerated still.

disk-usage
disk-usage3371×686 92.4 KB

2

OK more strangeness. It seems the /var/lib dir is NOT growing, yet overall disk space available is still shrinking. Its like some file being discarded is not being properly released or closed ? This feels like a bug of some sort maybe ?
Linux influxdb.sentex.ca 5.4.0-181-generic #201-Ubuntu SMP Thu Mar 28 15:39:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@influxdb:/var# cat /etc/issue
Ubuntu 20.04.6 LTS \n \l

root@influxdb:/var#
New kernel was installed on Thursday as well for security patches

3

OK looks like it wasnt influxDB in the end. I let disk space grow and did a restart of it. Still a lot of disk spaced being used and not reclaimed. I restarted prometheus and low an behold my disk space dropped. Some of the compaction was failing and it just kept growing. Sorry for the noise

4

Hello @mdtancsa Thanks so much for sharing your solution with the community I’m sure itll help lots of other people encountering the same issues.