Broken blocks and Error compacting TSM files

1

Hi!

I’m new to the community. I have searched for smilar topics but without success.

I noticed my syslog was getting too big so I had to investigate.

okt 04 10:10:06 myhabpi influxd[20000]: ts=2020-10-04T08:10:06.604106Z lvl=info msg="TSM compaction (start)" log_id=0PddTX2W000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0PdfQDZ0000 op_name=tsm1_compact_group op_event=start
okt 04 10:10:06 myhabpi influxd[20000]: ts=2020-10-04T08:10:06.604117Z lvl=info msg="Compacting file" log_id=0PddTX2W000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0PdfQDYl000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/media/usbstick/influxdb/data/openhab_db/autogen/128/000000011-000000001.tsm
okt 04 10:10:06 myhabpi influxd[20000]: ts=2020-10-04T08:10:06.604144Z lvl=info msg="Beginning compaction" log_id=0PddTX2W000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0PdfQDZ0000 op_name=tsm1_compact_group tsm1_files_n=2
okt 04 10:10:06 myhabpi influxd[20000]: ts=2020-10-04T08:10:06.604172Z lvl=info msg="Compacting file" log_id=0PddTX2W000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0PdfQDZ0000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/media/usbstick/influxdb/data/openhab_db/autogen/124/000000008-000000002.tsm
okt 04 10:10:06 myhabpi influxd[20000]: ts=2020-10-04T08:10:06.604206Z lvl=info msg="Compacting file" log_id=0PddTX2W000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0PdfQDZ0000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/media/usbstick/influxdb/data/openhab_db/autogen/124/000000010-000000001.tsm
okt 04 10:10:06 myhabpi influxd[20000]: ts=2020-10-04T08:10:06.631485Z lvl=warn msg="Error compacting TSM files" log_id=0PddTX2W000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0PdfQDZ0000 op_name=tsm1_compact_group error="block read error on /media/usbstick/influxdb/data/openhab_db/autogen/124/000000010-000000001.tsm: decode error: unable to decompress block type float for key 'AqaraTemp1_TEMPERATURE#!~#value': EOF, block read error on /media/usbstick/influxdb/data/openhab_db/autogen/124/000000010-000000001.tsm: decode error: unable to decompress block type float for key 'AqaraTemp1_TEMPERATURE#!~#value': EOF"
okt 04 10:10:06 myhabpi influxd[20000]: ts=2020-10-04T08:10:06.716383Z lvl=warn msg="Error compacting TSM files" log_id=0PddTX2W000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0PdfQDYl000 op_name=tsm1_compact_group error="block read error on /media/usbstick/influxdb/data/openhab_db/autogen/128/000000011-000000001.tsm: decode error: unable to decompress block type float for key 'SM_Grid#!~#value': EOF"
okt 04 10:10:07 myhabpi influxd[20000]: ts=2020-10-04T08:10:07.631945Z lvl=info msg="TSM compaction (end)" log_id=0PddTX2W000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0PdfQDZ0000 op_name=tsm1_compact_group op_event=end op_elapsed=1027.769ms
okt 04 10:10:07 myhabpi influxd[20000]: ts=2020-10-04T08:10:07.718168Z lvl=info msg="TSM compaction (end)" log_id=0PddTX2W000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0PdfQDYl000 op_name=tsm1_compact_group op_event=end op_

Made somehow the assumption that there might be a problem with my USB-stick which is where the database is residing. I bought a new one but the problem persists.
Used backup/restore of the data with the “-portable” option, as explained in the tutorial. Perhaps the bad blocks are also present in the backup?

Tried to troubleshoot the situation using:

sudo influx_inspect verify -dir /media/usbstick/influxdb/

Got the following response (mostly healthy blocks but also some broken ones, included output is only a snapshot):

/media/usbstick/influxdb/data/openhab_db/autogen/1/000000003-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/10/000000004-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/100/000000003-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/103/000000001-000000001.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/104/000000003-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/105/000000001-000000001.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/106/000000002-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/107/000000004-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/109/000000001-000000001.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/11/000000003-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/110/000000004-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/112/000000003-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/113/000000003-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/114/000000001-000000001.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/115/000000001-000000001.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/116/000000005-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/117/000000012-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 2533118966 but expected 3096900255 for key [65 113 97 114 97 84 101 109 112 49 95 80 82 69 83 83 85 82 69 35 33 126 35 118 97 108 117 101], block 4
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 595096414 but expected 199764357 for key [83 77 95 71 114 105 100 35 33 126 35 118 97 108 117 101], block 34
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 1058826734 but expected 3803992953 for key [83 77 95 71 114 105 100 35 33 126 35 118 97 108 117 101], block 35
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 4156880968 but expected 707273624 for key [83 77 95 76 111 97 100 35 33 126 35 118 97 108 117 101], block 52
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 4038349877 but expected 4222930412 for key [83 77 95 76 111 97 100 73 110 118 35 33 126 35 118 97 108 117 101], block 78
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 62012049 but expected 1196079705 for key [83 77 95 76 111 97 100 73 110 118 35 33 126 35 118 97 108 117 101], block 82
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 2741383842 but expected 97729334 for key [83 77 95 76 111 97 100 73 110 118 35 33 126 35 118 97 108 117 101], block 88
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 1420472025 but expected 3024409889 for key [83 77 95 80 86 35 33 126 35 118 97 108 117 101], block 98
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 1696416413 but expected 1007499038 for key [83 77 95 80 86 35 33 126 35 118 97 108 117 101], block 100
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 2902752269 but expected 3187907171 for key [83 77 95 80 86 35 33 126 35 118 97 108 117 101], block 101
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 2638082796 but expected 3221999693 for key [83 77 95 80 95 69 120 112 111 114 116 35 33 126 35 118 97 108 117 101], block 105
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 3159458829 but expected 1568346434 for key [83 77 95 80 95 69 120 112 111 114 116 35 33 126 35 118 97 108 117 101], block 106
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 187954222 but expected 1686036644 for key [83 77 95 80 95 69 120 112 111 114 116 35 33 126 35 118 97 108 117 101], block 110
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 2388781431 but expected 1711961005 for key [83 77 95 80 95 73 109 112 111 114 116 35 33 126 35 118 97 108 117 101], block 116
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 731605022 but expected 3149697961 for key [83 77 95 80 95 73 109 112 111 114 116 35 33 126 35 118 97 108 117 101], block 117
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 88211475 but expected 382976569 for key [83 77 95 80 95 73 109 112 111 114 116 35 33 126 35 118 97 108 117 101], block 121
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 3056881563 but expected 1220658157 for key [83 77 95 83 101 108 102 67 35 33 126 35 118 97 108 117 101], block 127
/media/usbstick/influxdb/data/openhab_db/autogen/118/000000009-000000002.tsm: got 2541177025 but expected 2320559569 for key [83 77 95 83 101 108 102 67 35 33 126 35 118 97 108 117 101], block 128
/media/usbstick/influxdb/data/openhab_db/autogen/119/000000011-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/12/000000001-000000001.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/120/000000012-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/93/000000003-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/94/000000003-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/96/000000003-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/97/000000002-000000002.tsm: healthy
/media/usbstick/influxdb/data/openhab_db/autogen/98/000000003-000000002.tsm: healthy
Broken Blocks: 66 / 7734, in 0.449958134s

Where should I go from here? Can I somehow dump the bad content and keep the healthy data?
Some assistance would be much appreciated!

Some system information. If I need to specify something more please say so:

  • Raspberry Pi 3b

  • PRETTY_NAME=“Raspbian GNU/Linux 10 (buster)”
    VERSION_ID=“10”

  • influx -version: InfluxDB shell version: 1.8.3

  • data recorded/gathered by an Openhab service, a home automation software
    openHAB 2.5.9 Release Build

Regards, Niklas.

2

Is my data salvageable or do I have to dump the incomplete or corrupted tsm-files.
And is it safe to just delete them or will I mess things up in the database?

Doesn’t anyone have any experience from bad blocks?

3

This is not the busiest of forums it seems.
Finally got fed up with this issue as it clogs the system logs.

Tried to do the following using the Influx documentation as a guide:

[19:56:47] pi@myhabpi:/media/usbstick/influxdb/data$ sudo influx_inspect deletetsm  -measurement AqaraTemp1_TEMPERATURE /media/usbstick/influxdb/data/openhab_db/autogen/*/*.tsm
2020/10/22 19:57:16 processing: /media/usbstick/influxdb/data/openhab_db/autogen/*/*.tsm
deletetsm: open openhab_db/autogen/*/*.tsm: no such file or directory

Apparently there seems to be some kind of permission issue, my guess anyway, as the file was not found. I checked and the file is clearly there.

Instead started to manually delete the TSM-files:

[20:09:22] pi@myhabpi:/media/usbstick/influxdb/data$ sudo rm openhab_db/autogen/121/000000011-000000002.tsm
[20:09:26] pi@myhabpi:/media/usbstick/influxdb/data$ sudo rm openhab_db/autogen/124/000000008-000000002.tsm
[20:09:36] pi@myhabpi:/media/usbstick/influxdb/data$ sudo rm openhab_db/autogen/128/000000009-000000002.tsm
[20:09:44] pi@myhabpi:/media/usbstick/influxdb/data$ sudo rm openhab_db/autogen/131/000000009-000000002.tsm
[20:09:55] pi@myhabpi:/media/usbstick/influxdb/data$ sudo rm openhab_db/autogen/25/000000001-000000001.tsm

I then ran the verify command again and the broken blocks are no longer present. Success!

Will come back with more information if I encounter any negative effects. :pray:

1 Like
4

Hello @NickBuilder,
I apologize! Sometimes questions get buried or lost. Please feel free to tag me if this ever happens again. I’m also forwarding this question to the storage team so they can give you additional help.