I have a databse which has stopped collecting data i would apper to be in a compaction loop
can anyone help me resolve this?
Feb 14 11:47:21 raspberrypi influxd[447]: [httpd] 192.168.0.133 - - [14/Feb/2019:11:47:21 +0000] “POST /write?db=sensors&precision=s HTTP/1.1” 204 0 “-” “YoctoHub-Ethernet” 49efdd04-304e-11e9-8535-b827eb45f62d 10915
Feb 14 11:47:22 raspberrypi influxd[447]: ts=2019-02-14T11:47:22.621052Z lvl=info msg=“TSM compaction (start)” log_id=0Dbu0at0000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0DbyAFVG000 op_name=tsm1_compact_group op_event=
Feb 14 11:47:22 raspberrypi influxd[447]: ts=2019-02-14T11:47:22.621150Z lvl=info msg=“Beginning compaction” log_id=0Dbu0at0000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0DbyAFVG000 op_name=tsm1_compact_group tsm1_files_
Feb 14 11:47:22 raspberrypi influxd[447]: ts=2019-02-14T11:47:22.621186Z lvl=info msg=“Compacting file” log_id=0Dbu0at0000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0DbyAFVG000 op_name=tsm1_compact_group tsm1_index=0 tsm
Feb 14 11:47:22 raspberrypi influxd[447]: ts=2019-02-14T11:47:22.621221Z lvl=info msg=“Compacting file” log_id=0Dbu0at0000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0DbyAFVG000 op_name=tsm1_compact_group tsm1_index=1 tsm
Feb 14 11:47:22 raspberrypi influxd[447]: ts=2019-02-14T11:47:22.621252Z lvl=info msg=“Compacting file” log_id=0Dbu0at0000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0DbyAFVG000 op_name=tsm1_compact_group tsm1_index=2 tsm
Feb 14 11:47:22 raspberrypi influxd[447]: ts=2019-02-14T11:47:22.621284Z lvl=info msg=“Compacting file” log_id=0Dbu0at0000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0DbyAFVG000 op_name=tsm1_compact_group tsm1_index=3 tsm
Feb 14 11:47:22 raspberrypi influxd[447]: ts=2019-02-14T11:47:22.647142Z lvl=info msg=“Error compacting TSM files” log_id=0Dbu0at0000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0DbyAFVG000 op_name=tsm1_compact_group error
Feb 14 11:47:23 raspberrypi influxd[447]: ts=2019-02-14T11:47:23.648920Z lvl=info msg=“TSM compaction (end)” log_id=0Dbu0at0000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0DbyAFVG000 op_name=tsm1_compact_group op_event=en
lines 20276-20339/20339 (END)
We are hitting a similar issue with 2.7.5 and CPU spikes close to 100% and recovers in about 3 days. Any insights into this?
Hello @Raji_Seeker,
You can use the influxd inspect
tool to check the health of your TSM files.
Or check your configuration for compaction settings such as compact-full-write-cold-duration, max-concurrent-compactions, and others.
Check out:
Or look at compactions-failed metrics
However I will mention that were at v3 now so there isn’t a lot of support for v1 and TSM consider upgrading to 3.
Thank you
We did hit the ‘expired phantom shards’ issue due to retention service being unresponsive. So the plan is to move to the next stable version 2.7.11 (since 2.7.12 is yet to be marked stable) and then to 3.x. Do you see any value in moving directly to 3.x? I am guessing it will be seamless to the users of our application.
