Decided to do a little preventative maintenance on a large infludb instance and ended up creating a big issue.
The database was about 600gb, and we had about 50g to spare. So I decided to truncate one of the large series by deleting some old data. I used a DELETE FROM measurement WHERE time < some_date. To my surprise the data footprint started growing… a lot… to the point where it ate up the rest of the disk space. We then expanded the partition, and it ate the rest of that as well! All write clients have been turned off and the wal cleared.
Now I’m stuck, the database will start up but cannot write any additional data as the disk is full, no logs either.
I guess first question is whether someone can explain why the data swells during a delete operation? Second is there a crafty way to delete some data even though the disk is full?