# Unable to delete data from a large series

**URL:** <https://community.influxdata.com/t/unable-to-delete-data-from-a-large-series/506>\
**Category:** Store\
**Created:** [April 6, 2017, 1:37pm UTC](https://community.influxdata.com/t/unable-to-delete-data-from-a-large-series/506 "2017-04-06T13:37:01Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul\_Wolfe](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@Paul\_Wolfe](https://community.influxdata.com/u/Paul_Wolfe)\
**Post date:** [April 6, 2017, 1:37pm UTC](https://community.influxdata.com/t/unable-to-delete-data-from-a-large-series/506/1 "2017-04-06T13:37:02Z")

</div>

Hey all,

Decided to do a little preventative maintenance on a large infludb instance and ended up creating a big issue.

The database was about 600gb, and we had about 50g to spare. So I decided to truncate one of the large series by deleting some old data. I used a DELETE FROM measurement WHERE time \< some\_date. To my surprise the data footprint started growing… a lot… to the point where it ate up the rest of the disk space. We then expanded the partition, and it ate the rest of that as well! All write clients have been turned off and the wal cleared.

Now I’m stuck, the database will start up but cannot write any additional data as the disk is full, no logs either.

I guess first question is whether someone can explain why the data swells during a delete operation? Second is there a crafty way to delete some data even though the disk is full?

Cheers,  
Paul

---

<div class="post-metadata">

**Author:** ![Paul\_Wolfe](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@Paul\_Wolfe](https://community.influxdata.com/u/Paul_Wolfe)\
**Post date:** [April 6, 2017, 2:02pm UTC](https://community.influxdata.com/t/unable-to-delete-data-from-a-large-series/506/2 "2017-04-06T14:02:42Z")

</div>

Figured something out… in the data shards I see “tombstone” files everywhere that basically double the size of data I was trying to delete… I guess.

Anyone can confirm my assumptions and know when these files should be deleted?

---

<div class="post-metadata">

**Author:** ![sebito91](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/sebito91/32/42_2.png) [@sebito91](https://community.influxdata.com/u/sebito91)\
**Post date:** [April 6, 2017, 3:03pm UTC](https://community.influxdata.com/t/unable-to-delete-data-from-a-large-series/506/3 "2017-04-06T15:03:37Z")

</div>

Known issue, and we submit this a few weeks back seeing the same problems. In our case, we had gobs of space so that was fine but ultimately we OOM’d the process instead.

> <https://github.com/influxdata/influxdb/pull/8204>
>
> \###### Required for all non-trivial PRs
> \- \[\] Rebased/mergable
> \- \[\] Tests pas…s
> \- \[\] CHANGELOG.md updated
> 
> This PR speeds up all drop and deletes statements as well as startup times. The drops and delete statements were slow due to cleaning up the in-memory index as well as some inefficient code paths in the TSM tombstoning path.
> 
> \* Deleting series keys using \`FileStore.Walk\` which iterated over every key in every TSM file sequentially. It spent a lot of time and memory deduplicating keys and converting them to the underlying series keys. This was switch to leverage the already sorted TSM file indexes to merge and iterate over them in parallel.  
> \* A side effect of optimizing \`FileStore.Walk\` is that it is also used at startup to reload the index. This now reduces the number of keys that are parsed across all files. I noticed ~50% speedup with my dataset, but did not experiment with other datasets. It's possible the \_optimize\_ compactions may be able to be removed in the future if this change keeps start times consistent.
> \* Writing tombstone files did not buffer writes. When deleting many series keys, a lot of IO time due to small writes would slow the writes down. This now uses a buffered writer.
> \* Deleting a retention policy or \`delete series\` used a slower code path that needed to remove series ids from the in-memory measurement index. There is a sorted slice of series IDs that was constantly maintained and cause memory to be continuously re-copied. Keeping this sorted during large bulk deletes was too slow. Instead, we create the sorted slice on-demand the first time it's requested by a query. Deletes now just delete from a map which is much faster.
> \* Deleting keys in various parts of the TSM engine used some intermediate maps. This created garbage and high memory usage. These were switched to use approaches that leveraged the already sorted series keys to avoid the intermediate allocations.
> 
> Some benchmarks from testing w/ a dataset of 5M series in some unoptimized TSM files (e.g the keys were repeated across many big TSM files). I test the same dataset w/ 1.2.2 and this branch:
> 
> \### Drop retention policy
> Before \*\*10m35s\*\*, after \*\*7s\*\*
> 
> \### Drop measurement
> Before \*\*25m6s\*\*, after \*\*29s\*\*
> 
> Delete series and drop database (also optimized further in #8179) had similar speedups.

According to the developers, this is slated for 1.3.0 relase which may be a bit of time out. You can always build a custom image where you cherry-pick that commit.

---

<div class="post-metadata">

**Author:** ![Paul\_Wolfe](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@Paul\_Wolfe](https://community.influxdata.com/u/Paul_Wolfe)\
**Post date:** [April 6, 2017, 3:30pm UTC](https://community.influxdata.com/t/unable-to-delete-data-from-a-large-series/506/4 "2017-04-06T15:30:13Z")

</div>

Thanks for the reply.

Do you know if the tombstones will ever be compacted/cleaned up in 1.2? I still see loads of them hanging around, even after things have been running stable for a couple hours.

---

<div class="post-metadata">

**Author:** ![sebito91](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/sebito91/32/42_2.png) [@sebito91](https://community.influxdata.com/u/sebito91)\
**Post date:** [April 20, 2017, 11:58pm UTC](https://community.influxdata.com/t/unable-to-delete-data-from-a-large-series/506/5 "2017-04-20T23:58:41Z")

</div>

I’d have to defer to Jack or Jason on this one, but AFAIK you should be able to remove them manually if they don’t get cleaned up on their own.

Typically these are compacted at app startup (as well as normal execution of the app), but if you’re able to restart the binary it should take care of any legacy ones.

---

<div class="post-metadata">

**Author:** ![Paul\_Wolfe](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@Paul\_Wolfe](https://community.influxdata.com/u/Paul_Wolfe)\
**Post date:** [May 3, 2017, 11:31am UTC](https://community.influxdata.com/t/unable-to-delete-data-from-a-large-series/506/6 "2017-05-03T11:31:52Z")

</div>

Update , they do eventually get cleaned up.