Decided to do a little preventative maintenance on a large infludb instance and ended up creating a big issue.
The database was about 600gb, and we had about 50g to spare. So I decided to truncate one of the large series by deleting some old data. I used a DELETE FROM measurement WHERE time < some_date. To my surprise the data footprint started growing… a lot… to the point where it ate up the rest of the disk space. We then expanded the partition, and it ate the rest of that as well! All write clients have been turned off and the wal cleared.
Now I’m stuck, the database will start up but cannot write any additional data as the disk is full, no logs either.
I guess first question is whether someone can explain why the data swells during a delete operation? Second is there a crafty way to delete some data even though the disk is full?
Known issue, and we submit this a few weeks back seeing the same problems. In our case, we had gobs of space so that was fine but ultimately we OOM’d the process instead.
According to the developers, this is slated for 1.3.0 relase which may be a bit of time out. You can always build a custom image where you cherry-pick that commit.
Do you know if the tombstones will ever be compacted/cleaned up in 1.2? I still see loads of them hanging around, even after things have been running stable for a couple hours.
I’d have to defer to Jack or Jason on this one, but AFAIK you should be able to remove them manually if they don’t get cleaned up on their own.
Typically these are compacted at app startup (as well as normal execution of the app), but if you’re able to restart the binary it should take care of any legacy ones.
