Writes to InfluxDB timing out during level compaction

We are using InfluxDB 1.7.2.

We are observing that the writes to the InfluxDB timing out when a level compaction is happening.

We have a timeout of 5sec at client side. Normally the writes to InfluxDB finishes with in few milliseconds. However, when the level compaction is going on the response times increases beyond 5 secs and results in a client side timeout.

The log during the level compaction.

el=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group op_event=start
ts=2020-10-18T08:01:14.835429Z lvl=info msg=“Beginning compaction” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_files_n=8
ts=2020-10-18T08:01:14.835435Z lvl=info msg=“Compacting file” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/opt/influx/data/downsampled/one_year/7835/000000514-000000001.tsm
ts=2020-10-18T08:01:14.835440Z lvl=info msg=“Compacting file” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/opt/influx/data/downsampled/one_year/7835/000000515-000000001.tsm
ts=2020-10-18T08:01:14.835446Z lvl=info msg=“Compacting file” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/opt/influx/data/downsampled/one_year/7835/000000516-000000001.tsm
ts=2020-10-18T08:01:14.835451Z lvl=info msg=“Compacting file” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/opt/influx/data/downsampled/one_year/7835/000000517-000000001.tsm
ts=2020-10-18T08:01:14.835456Z lvl=info msg=“Compacting file” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/opt/influx/data/downsampled/one_year/7835/000000518-000000001.tsm
ts=2020-10-18T08:01:14.835461Z lvl=info msg=“Compacting file” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/opt/influx/data/downsampled/one_year/7835/000000519-000000001.tsm
ts=2020-10-18T08:01:14.835466Z lvl=info msg=“Compacting file” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/opt/influx/data/downsampled/one_year/7835/000000520-000000001.tsm
ts=2020-10-18T08:01:14.835482Z lvl=info msg=“Compacting file” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/opt/influx/data/downsampled/one_year/7835/000000521-000000001.tsm
ts=2020-10-18T08:01:16.122029Z lvl=info msg=“Compacted file” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/opt/influx/data/downsampled/one_year/7835/000000521-000000002.tsm.tmp
ts=2020-10-18T08:01:16.122065Z lvl=info msg=“Finished compacting files” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2020-10-18T08:01:16.122073Z lvl=info msg=“TSM compaction (end)” log_id=0PjiSVJ0000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0PvgTtKl000 op_name=tsm1_compact_group op_event=end op_elapsed=1286.695ms

The response time of writes/read requests to InfluxDB.

image

How can I optimize the level compaction behavior so that the writes during this period will not timeout?

Regards,
Robert