# TSM Compaction loop **URL:** \ **Category:** Systems\ **Created:** [May 3, 2021, 2:06pm UTC](https://community.influxdata.com/t/tsm-compaction-loop/19709 "2021-05-03T14:06:33Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Frans\_Coetzee](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/frans_coetzee/32/4978_2.png) [@Frans\_Coetzee](https://community.influxdata.com/u/Frans_Coetzee)\ **Post date:** [May 3, 2021, 2:06pm UTC](https://community.influxdata.com/t/tsm-compaction-loop/19709/1 "2021-05-03T14:06:33Z")
I have had a problem where every few weeks Influx gets stuck in a compaction loop. This eats the CPU and disk. This is InfluxDB 2.0 (happened with multiple versions). - What is the fix? Is it possible to delete some files while keeping data? So far only solution was to completely reload the database from original source - when I deleted some of these possibly corrupted files I neded up with missing data) s=2021-05-03T13:57:34.631567Z lvl=info msg=“TSM compaction (start)” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group op\_event=start ts=2021-05-03T13:57:34.631691Z lvl=info msg=“Beginning compaction” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_files\_n=4 ts=2021-05-03T13:57:34.631695Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=0 tsm1\_file=/root/.influxdbv2/engine/data/000000000002680-000000003.tsm ts=2021-05-03T13:57:34.631705Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=1 tsm1\_file=/root/.influxdbv2/engine/data/000000000002712-000000003.tsm ts=2021-05-03T13:57:34.631708Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=2 tsm1\_file=/root/.influxdbv2/engine/data/000000000002744-000000003.tsm ts=2021-05-03T13:57:34.631711Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=3 tsm1\_file=/root/.influxdbv2/engine/data/000000000002776-000000003.tsm ts=2021-05-03T13:57:41.934475Z lvl=info msg=“Error compacting TSM files” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group error=EOF ts=2021-05-03T13:57:42.934667Z lvl=info msg=“TSM compaction (end)” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group op\_event=end op\_elapsed=8303.103ms ts=2021-05-03T13:57:43.631706Z lvl=info msg=“TSM compaction (start)” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group op\_event=start ts=2021-05-03T13:57:43.631742Z lvl=info msg=“Beginning compaction” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_files\_n=4 ts=2021-05-03T13:57:43.631745Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=0 tsm1\_file=/root/.influxdbv2/engine/data/000000000002680-000000003.tsm ts=2021-05-03T13:57:43.631750Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=1 tsm1\_file=/root/.influxdbv2/engine/data/000000000002712-000000003.tsm ts=2021-05-03T13:57:43.631754Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=2 tsm1\_file=/root/.influxdbv2/engine/data/000000000002744-000000003.tsm ts=2021-05-03T13:57:43.631758Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=3 tsm1\_file=/root/.influxdbv2/engine/data/000000000002776-000000003.tsm ts=2021-05-03T13:57:50.970879Z lvl=info msg=“Error compacting TSM files” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group error=EOF ts=2021-05-03T13:57:51.971082Z lvl=info msg=“TSM compaction (end)” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group op\_event=end op\_elapsed=8339.375ms ts=2021-05-03T13:57:52.631539Z lvl=info msg=“TSM compaction (start)” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group op\_event=start ts=2021-05-03T13:57:52.631569Z lvl=info msg=“Beginning compaction” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_files\_n=4 ts=2021-05-03T13:57:52.631573Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=0 tsm1\_file=/root/.influxdbv2/engine/data/000000000002680-000000003.tsm ts=2021-05-03T13:57:52.631576Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=1 tsm1\_file=/root/.influxdbv2/engine/data/000000000002712-000000003.tsm ts=2021-05-03T13:57:52.631579Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=2 tsm1\_file=/root/.influxdbv2/engine/data/000000000002744-000000003.tsm ts=2021-05-03T13:57:52.631594Z lvl=info msg=“Compacting file” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group tsm1\_index=3 tsm1\_file=/root/.influxdbv2/engine/data/000000000002776-000000003.tsm ts=2021-05-03T13:57:59.975213Z lvl=info msg=“Error compacting TSM files” log\_id=0TbhIEYl000 service=storage-engine engine=tsm1 tsm1\_level=3 tsm1\_strategy=level op\_name=tsm1\_compact\_group error=EOF ts=2021-05-03T1 ---
**Author:** ![Frans\_Coetzee](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/frans_coetzee/32/4978_2.png) [@Frans\_Coetzee](https://community.influxdata.com/u/Frans_Coetzee)\ **Post date:** [May 5, 2021, 8:39pm UTC](https://community.influxdata.com/t/tsm-compaction-loop/19709/2 "2021-05-05T20:39:16Z")
Any chance of an update here – pointer to documentation on tsm files etc? ---
**Author:** ![dgnorton](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/dgnorton/32/40_2.png) [@dgnorton](https://community.influxdata.com/u/dgnorton)\ **Post date:** [May 6, 2021, 2:48pm UTC](https://community.influxdata.com/t/tsm-compaction-loop/19709/3 "2021-05-06T14:48:40Z")
@Frans_Coetzee there’s a tool called `influx_inspect` in the 1.8.x packages that hasn’t been fully migrated into 2.0.x yet. If you pull that tool out of the latest 1.8.x, you can run: ``` influx_inspect verify -dir path/to/data ``` where `path/to/data` is the path to your root data dir. This is an off-line tool that will scan the `data` dir recursively for `.tsm` files and check to ensure each is valid. Salvaging data from a corrupt `.tsm` file is challenging and not always practical. You can move them aside to get the database into a healthy state again. Documentation on the `.tsm` file format [can be found here](https://github.com/influxdata/influxdb/blob/v2.0.6/tsdb/engine/tsm1/DESIGN.md). ---
**Author:** ![Frans\_Coetzee](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/frans_coetzee/32/4978_2.png) [@Frans\_Coetzee](https://community.influxdata.com/u/Frans_Coetzee)\ **Post date:** [May 6, 2021, 7:13pm UTC](https://community.influxdata.com/t/tsm-compaction-loop/19709/4 "2021-05-06T19:13:23Z")
Thanks. Just to be clear, this corruption has happened twice now; it seems to happen when server was rebooted. From documentation it is not clear to me what the downside would be are when you remove a corrupted tsm file. The docs seem to say the WAL files (or other source files) are only removed once a new tsm file has been successfully written and checksummed. Hence, removing a corrupted file should be safe, compaction will just restart from the (still present) source files. Here eg I found the corrupted file, and moved it, so should be ok. But last time, I also ended removing a corrupted file and then found test queries failed and definitely data was lost. In the end I had to completely rebuild the whole database from raw csv files, a massive compute load. Can you clarify for me that (a) I can be sure data is not missing or (b) tell me how to figure out what was missing. Seems pretty bad if the DB is not ACID ---
**Author:** ![Frans\_Coetzee](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/frans_coetzee/32/4978_2.png) [@Frans\_Coetzee](https://community.influxdata.com/u/Frans_Coetzee)\ **Post date:** [May 6, 2021, 7:14pm UTC](https://community.influxdata.com/t/tsm-compaction-loop/19709/5 "2021-05-06T19:14:58Z")
![image](https://us1.discourse-cdn.com/flex023/uploads/influxdata/original/2X/6/6f4daf8f615774af8a347437032f2fd488153d16.png)