I am using an InfluxDB OSS installation (version 2.3.0), installed via the package manager.
My setup has 2 organizations, where one has 11 buckets (9 + 2 system buckets) and the other has 4 buckets (2 + 2 system buckets). The first organization is the most used one, where a few of the buckets are currently storing some metrics and logs still with a large retention.
My issue is that recently I noticed that the disk utilization is starting to increase at a higher rate than I expected. When I look at the disk utilization of the directories under /var/lib/influxdb/engine/data I see that there’s a single bucket that is occupying most of the space (151GB, when 2nd most used is 2.0GB). The problem is that I cannot find out what this bucked ID is associated with:
It does not correspond to any bucket listed in the UI (Load Data → Buckets)
It does not appear in the list of buckets given by the CLI (for both orgs)
I currently am out of ideas over what might be causing this issue, or how to debug it.
Is it possible for additional system buckets to exist, which are not listed in the UI?
Is it possible that the data from the other buckets is being saved under this directory?
Does anyone have any idea how I can find out what is being saved under this directory?
The best way to try to track this down would be to investigate the influxd.bolt file.
You can do this using boltdbweb. With influxdb stopped (or you could take a backup and then point at that boltdb file), if you run boltdbweb -d <path-to-your-bolt-file> you will get a web page that you can use to inspect the file. Under bucketsv1 you can see JSON-like objects for each bucket. That should show you the name that corresponds to the id of the bucket that is growing out of hand.
I have successfully used the boltdweb tool to assess the influxdb.bolt file, but the ID is not part of the list of IDs accessible in the bucketsv1 page. And yet the directory keeps growing (from 150GB to 174GB since last time).
Update: I did find a reference to the bucket ID in the page authorizationsv1. I have now deleted the token that was referencing it (even though in the InfluxDB UI it was listed as a token with general access to ALL Buckets).
Update x2: I found a second reference to the ID in the page scraperv2. In the InfluxDB UI there was indeed a scraper pointing to the /metrics page without any bucked associated with it. I deleted the scraper, hopefully the directory stops increasing now.
I found a second reference to the ID in the page scraperv2 . In the InfluxDB UI there was indeed a scraper pointing to the /metrics page without any bucked associated with it. I deleted the scraper, hopefully the directory stops increasing now.
That is very interesting, please let us know if deleting that scraper resolves the issue as that is clearly a bug we should address.
As a final update, I can confirm that after deleting the scraper the directory stopped increasing in size and remains still with 174GB.
I will have to confirm if deleting the directory will not affect any of the other buckets.
According to du -hs /var/lib/influxdb2/engine/data/ a directory was eating up my entire disk space.
I couldn’t find a ralative bucket (wether in GUI nor with CLI), but I found a Scraper which was named “new scraper” and had no bucket specified.
