InfluxDB 2 v2.7.0 Freezes Time After Time

Azimuth · October 12, 2023, 7:30am

A very big hug to the community!

I have some problems with InfluxDB freezing because of an “internal error”. I suppose the reason could be something related to reindexing. Currently, I can’t catch the problem.

So, I’m interested in either somebody who has resolved such a problem or has some ideas on how to overcome an issue. Thanks!

Some logs:

[user@monitoring ~]$ journalctl  --since "2023-10-11 02:07" --until "2023-10-11 02:10" | grep -i 'influxd-systemd-start'
Oct 11 02:08:06 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:06.254103Z lvl=info msg="index opened with 8 partitions" log_id=0knAdzwG000 service=storage-engine index=tsi
Oct 11 02:08:06 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:06.254484Z lvl=info msg="loading changes (start)" log_id=0knAdzwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
Oct 11 02:08:06 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:06.254530Z lvl=info msg="loading changes (end)" log_id=0knAdzwG000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.055ms
Oct 11 02:08:06 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:06.255025Z lvl=info msg="Reindexing TSM data" log_id=0knAdzwG000 service=storage-engine engine=tsm1 db_shard_id=809
Oct 11 02:08:06 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:06.255042Z lvl=info msg="Reindexing WAL data" log_id=0knAdzwG000 service=storage-engine engine=tsm1 db_shard_id=809
Oct 11 02:08:06 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:06.271975Z lvl=info msg="saving field index changes (start)" log_id=0knAdzwG000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=start
Oct 11 02:08:06 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:06.277287Z lvl=info msg="saving field index changes (end)" log_id=0knAdzwG000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=end op_elapsed=5.323ms
Oct 11 02:08:06 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:06.643584Z lvl=warn msg="internal error not returned to client" log_id=0knAdzwG000 handler=error_logger error="context canceled"
Oct 11 02:08:06 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:06.644776Z lvl=warn msg="internal error not returned to client" log_id=0knAdzwG000 handler=error_logger error="context canceled"
Oct 11 02:08:11 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:11.634037Z lvl=warn msg="internal error not returned to client" log_id=0knAdzwG000 handler=error_logger error="context canceled"
Oct 11 02:08:11 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:11.642932Z lvl=warn msg="internal error not returned to client" log_id=0knAdzwG000 handler=error_logger error="context canceled"
Oct 11 02:08:26 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:26.646839Z lvl=warn msg="internal error not returned to client" log_id=0knAdzwG000 handler=error_logger error="context canceled"
Oct 11 02:08:26 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:26.650471Z lvl=warn msg="internal error not returned to client" log_id=0knAdzwG000 handler=error_logger error="context canceled"
Oct 11 02:08:56 monitoring influxd-systemd-start.sh[3174967]: ts=2023-10-11T02:08:56.655019Z lvl=warn msg="internal error not returned to client" log_id=0knAdzwG000 handler=error_logger error="context canceled"

Sometimes I have:

Oct 12 07:21:02 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:21:02.395281Z lvl=info msg="Cache snapshot (start)" log_id=0komBSfl000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=start
Oct 12 07:21:02 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:21:02.672531Z lvl=info msg="Snapshot for path written" log_id=0komBSfl000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot path=/opt/influxdb/engine/data/f34d8a2245cccd9f/autogen/812 duration=277.273ms
Oct 12 07:21:02 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:21:02.672577Z lvl=info msg="Cache snapshot (end)" log_id=0komBSfl000 service=storage-engine engine=tsm1 op_name=tsm1_cache_snapshot op_event=end op_elapsed=277.325ms
Oct 12 07:22:32 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:22:32.198702Z lvl=info msg="index opened with 8 partitions" log_id=0komBSfl000 service=storage-engine index=tsi
Oct 12 07:22:32 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:22:32.199096Z lvl=info msg="loading changes (start)" log_id=0komBSfl000 service=storage-engine engine=tsm1 op_name="field indices" op_event=start
Oct 12 07:22:32 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:22:32.199180Z lvl=info msg="loading changes (end)" log_id=0komBSfl000 service=storage-engine engine=tsm1 op_name="field indices" op_event=end op_elapsed=0.101ms
Oct 12 07:22:32 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:22:32.199927Z lvl=info msg="Reindexing TSM data" log_id=0komBSfl000 service=storage-engine engine=tsm1 db_shard_id=811
Oct 12 07:22:32 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:22:32.199945Z lvl=info msg="Reindexing WAL data" log_id=0komBSfl000 service=storage-engine engine=tsm1 db_shard_id=811
Oct 12 07:22:32 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:22:32.213748Z lvl=info msg="saving field index changes (start)" log_id=0komBSfl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=start
Oct 12 07:22:32 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:22:32.221870Z lvl=info msg="saving field index changes (end)" log_id=0komBSfl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=end op_elapsed=8.132ms
Oct 12 07:22:32 monitoring influxd-systemd-start.sh[3231510]: ts=2023-10-12T07:22:32.221942Z lvl=info msg="Write failed" log_id=0komBSfl000 service=storage-engine service=write shard=811 error="engine: context canceled"

Configs:

bolt-path = "/var/lib/influxdb/influxd.bolt"
engine-path = "/opt/influxdb/engine"
query-concurrency = 2048
query-initial-memory-bytes = 0
query-max-memory-bytes = 0
query-memory-bytes = 0
query-queue-size = 2048
storage-compact-throughput-burst = 134217728
storage-series-id-set-cache-size = 1600

Anaisdg · October 13, 2023, 5:54pm

Hello @Azimuth,
Hmm I’m not sure I haven’t encountered this issue a lot.
I’m seeing other users encounter it though. Have you investigated any of the solutions mentioned here:

github.com/influxdata/influxdb

influxd "context canceled" error, queries never complete

opened 01:56AM - 13 Nov 22 UTC

closed 03:40PM - 14 May 23 UTC

max0x7ba

I have a Python script which makes 66 read queries into a local InfluxDB databas…e using 32 clients. It measures response time of each query and the total time. With version 2.0.4 the total time for these queries to complete was 50 seconds. These read queries only differ in measurement filter: ``` from(bucket:"A") |> range(start: 1670617080, stop: 1664578800) |> filter(fn: (r) => r._measurement == "M" and (r._field == "F1" or r._field == "F2" or r._field == "F3" or r._field == "F4" or r._field == "F5" or r._field == "F6")) |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value") |> keep(columns: ["_time","F1","F2","F3","F4","F5","F6"]) |> map(fn: (r) => ({r with _time: uint(v: r._time)})) ``` and they return tables with 1 `uint` and 6 `float` columns and up to ~100,000 rows. I upgraded from version 2.0.4 to 2.5.1. Now only 34 out of 66 queries complete while the rest get stuck forever. When that happens `influxd` writes the following error message (~20 times a minute) into the system log: ``` msg="internal error not returned to client" log_id=0e7eiZSl000 handler=error_logger error="context canceled" ``` I downgraded through earlier versions until I found one that completes all the queries - version 2.0.7. With version 2.0.8 half of the queries never complete. With version 2.0.4 the total time for these queries was 50 seconds. With version 2.0.5 the total query time doubled. __Expected behavior:__ 66 read queries complete. __Actual behavior:__ 34 out of 66 read queries complete while the rest get stuck forever. __Environment info:__ * System info: `Linux 5.15.0-52-lowlatency x86_64`, Ubuntu 20.04 LTS. * InfluxDB version: `InfluxDB v2.5.1 (git: 5b6fdbf05d) build_date: 2022-11-02T18:06:28Z` * Other relevant environment details: `influxdb-client 1.34.0`. __Config:__ ``` query-concurrency = 32 query-queue-size = 32 ``` __Logs:__ ``` Nov 13 01:50:56 influxd-systemd-start.sh[3874]: ts=2022-11-13T01:50:56.555889Z lvl=warn msg="internal error not returned to client" log_id=0e7eiZSl000 handler=error_logger error="context canceled" Nov 13 01:51:00 influxd-systemd-start.sh[3874]: ts=2022-11-13T01:51:00.051461Z lvl=warn msg="internal error not returned to client" log_id=0e7eiZSl000 handler=error_logger error="context canceled" Nov 13 01:51:02 influxd-systemd-start.sh[3874]: ts=2022-11-13T01:51:02.330052Z lvl=warn msg="internal error not returned to client" log_id=0e7eiZSl000 handler=error_logger error="context canceled" Nov 13 01:51:03 influxd-systemd-start.sh[3874]: ts=2022-11-13T01:51:03.608479Z lvl=warn msg="internal error not returned to client" log_id=0e7eiZSl000 handler=error_logger error="context canceled" ``` __Performance:__ [iostat.txt](https://github.com/influxdata/influxdb/files/9996356/iostat.txt) [profiles.tar.gz](https://github.com/influxdata/influxdb/files/9996357/profiles.tar.gz)

Is it when you’re running queries? What queries are you running?
Are you using Grafana?

Topic		Replies	Views
Influxdb freeze after a few hours running influxdb	2	1001	December 12, 2018
Influx crashes/stop showing data and we can't query InfluxDB 2 windows	5	777	August 2, 2022
Influxdb 2 container problems InfluxDB 2 influxdb , backup , performance , raspberry	2	308	January 4, 2024
Query engine stuck InfluxDB 2	8	204	October 31, 2024
How do you handle time sync issues? InfluxDB 2 influxdb , time-series , query , flux , timestamp	3	23	October 14, 2024

InfluxDB 2 v2.7.0 Freezes Time After Time

Related topics