I am using a v2.7.12 InfluxDB server and experiencing a deadlock very similar to this bug, which was supposedly fixed in v2.7.12: Possible Deadlock in TagValueIterator() and AddSeriesList() · Issue #26164 · influxdata/influxdb · GitHub
I have a v2.7.12 InfluxDB server which freezes and becomes non-responsive at random times with zero explanation as to why. I will try to do my best to explain the issue and what steps I’ve taken to debug. Please let me know if anymore information is needed or would be helpful, I am happy to provide.
The issues began after months of this server running. Data was being written to the server for a few months, but it wasn’t until about 6 months in we started noticing issues. The server is running on win10 and using NSSM as a wrapper for the windows service. There are about 200 series. We just recently deleted the entire database to start fresh and the problem is still happening. We also have countless other installs setup identical (except for what data is written) to this one and none have experienced this issue in the past 3-4 years. Also, there are no performance issues on this machine, InfluxDB has plenty of RAM and CPU, even when deadlocking/after deadlocking.
The issues begin with C# API errors saying “A task was cancelled”. A bit after that we will begin receiving HTTP connection errors. At this point “influxd.exe” is still running, but not responding to anything, including API calls or accessing the UI via :8086. This seems to occur randomly, different times of the day, and sometimes it runs for 2-3 weeks, other times only a few hours or days.
My first debugging step was to turn on Influx’s debug logging. I let that run until the server stopped responding and the logs stopped on a “Bucket Find” call. The Bucket Find call seemed like it was repeating 10-15 times and then the logs would just stop until we restarted the .exe. This was consistent on just about every crash that I debugged. Here is an example:
ts=2025-09-08T18:28:42.677880Z lvl=debug msg="bucket find" log_id=0yreEBzW000 store=new took=0.000ms
ts=2025-09-08T18:28:42.677880Z lvl=debug msg="bucket find" log_id=0yreEBzW000 store=new took=0.000ms
ts=2025-09-08T18:28:42.677880Z lvl=debug msg="bucket find" log_id=0yreEBzW000 store=new took=0.000ms
I then started to look at the event viewer to ensure the computer wasn’t updating, restarting, or limiting Influx’s ability to execute. Nothing was found.
After that I began to inspect the data files using the commands in influxd.exe. Nothing out of the ordinary was found there, and keep in mind this is a freshly made database its happening on. There was new data being written however.
I then started to look at the data drive to ensure there are no issues there. All of the checks I have run have resulted in no issues being found. We even took influx and everything else off of the drive, reformatted, and re-installed. still occurring.
Lastly I have taken a .dmp of influxd while the “deadlock” is occurring. I am unable to compile influxd as of right now so I do not have the correct symbols however I will post what I did find in hopes someone here understands it. Here is the stack:
STACK_TEXT:
00000000`0733f298 00007ffc`aa47da3e : 00000000`00482140 00000000`06ff06a0 00000000`00000102 00000000`00480601 : ntdll!NtWaitForSingleObject+0x14
00000000`0733f2a0 00000000`004821c9 : 00000000`06ff06a0 00000000`0028b000 00000000`00000000 00000000`00000134 : KERNELBASE!WaitForSingleObjectEx+0x8e
00000000`0733f340 00000000`0048054d : 00000000`06fee360 00007ffc`aa494d2c 00000000`0733f728 00000000`00000778 : influxd+0x821c9
00000000`0733f4b0 00000000`00480601 : 00000000`0733f4f8 00000000`00000000 00000000`0733f520 00000000`0733f530 : influxd+0x8054d
00000000`0733f4e0 00000000`0043c91b : 00000000`00482140 00000000`06ff06a0 00000000`00000000 00000000`0733f548 : influxd+0x80601
00000000`0733f508 00000000`0043ca56 : 00000000`00000002 00000000`0733f5b0 00000000`0043bb45 00000000`02907c68 : influxd+0x3c91b
00000000`0733f540 00000000`0043bb45 : 00000000`02907c68 00000000`00000134 00000000`ffffffff 00000000`00000000 : influxd+0x3ca56
00000000`0733f558 00000000`00410e65 : 00000000`04b80370 00000000`00432df3 00000000`00000004 00000000`00000005 : influxd+0x3bb45
00000000`0733f5c0 00000000`0043446e : 00000000`00000003 00000000`00000000 000000c0`00068480 00000000`00000002 : influxd+0x10e65
00000000`0733f628 00000000`0043436b : 00000000`00019391 00000000`00000001 000000c0`00068480 00152940`00000000 : influxd+0x3446e
00000000`0733f670 00000000`00433fff : 00000000`06ff0600 00000000`0733f710 00000000`00410f96 00000000`06ff0600 : influxd+0x3436b
00000000`0733f6d8 00000000`00410f96 : 00000000`06ff0600 00000000`0733f710 00000000`06fee360 00000000`0703d108 : influxd+0x33fff
00000000`0733f6f0 00000000`0043d677 : 00000000`02907e08 00000000`00000778 00000000`0733f770 00000000`00000001 : influxd+0x10f96
00000000`0733f720 00000000`0044fd55 : 00000000`0733fca0 00000000`0733fcb8 00000000`00424f96 000000c0`04c0c800 : influxd+0x3d677
00000000`0733fc80 00000000`00424f96 : 000000c0`04c0c800 00000000`00000060 00000006`0042a96a 00000000`00000000 : influxd+0x4fd55
00000000`0733fc98 00000000`0042ab45 : 000000c0`04c0c800 00000000`0733fd00 00000000`0042331d 000000c0`0007b250 : influxd+0x24f96
00000000`0733fcc8 00000000`0042331d : 000000c0`0007b250 000000c0`0007b250 00000000`00000000 000000c0`02b7f340 : influxd+0x2ab45
00000000`0733fce0 00000000`00421eaf : 000000c0`0007b250 00000000`00010000 00000000`0733fd78 00000000`0044a7ab : influxd+0x2331d
00000000`0733fd10 00000000`00421d7b : 000000c0`02b7f340 00000000`00010000 000000c0`04b5a7c8 00000000`0047e9a9 : influxd+0x21eaf
00000000`0733fd70 00000000`0047e9a9 : 00000000`0733fda8 00000000`00485885 00000000`06fee360 00000000`00000000 : influxd+0x21d7b
00000000`0733fd90 00000000`00421af3 : 000000c0`04b5a810 00000000`00000008 00000000`00010000 00000000`00000020 : influxd+0x7e9a9
000000c0`04b5a7d8 00000000`00412534 : 000000c0`02b7f340 00000000`00472845 000000c0`02b7f340 000000c0`04b5a8f0 : influxd+0x21af3
000000c0`04b5a838 00000000`0047223e : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`4edddba8 : influxd+0x12534
000000c0`04b5a860 00000000`00412625 : 00000000`00000068 00000000`045d21e0 00000000`52610101 000000c0`04b5a958 : influxd+0x7223e
000000c0`04b5a900 00000000`01143268 : 00000000`00000030 00000000`00000000 00000000`00000000 00000000`01145220 : influxd+0x12625
000000c0`04b5a928 00000000`011a598e : 00000000`00000000 00000000`00000000 00000000`00000008 00000000`3ffffed8 : influxd+0xd43268
000000c0`04b5a968 00000000`011a403b : 000000c0`04b5a9f8 00000000`52610130 00000000`00000059 00000000`3ffffed0 : influxd+0xda598e
000000c0`04b5a9c0 00000000`011a463b : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0xda403b
000000c0`04b5aa68 00000000`0119693e : 00000000`00479989 000000c0`0335c060 000000c0`01fa4000 000000c0`04b5abe0 : influxd+0xda463b
000000c0`04b5abd8 00000000`0118bba8 : 000000c0`04b5ac80 000000c0`0335c060 000000c0`04b5ac40 00000000`00412625 : influxd+0xd9693e
000000c0`04b5ac08 00000000`011aa8d3 : 000000c0`01c2e6d8 000000c0`0335c060 00000000`00000027 00000000`00000030 : influxd+0xd8bba8
000000c0`04b5ac90 00000000`0119081e : 00000000`00000000 000000c0`0335c060 00000000`00000027 00000000`00000030 : influxd+0xdaa8d3
000000c0`04b5acc8 00000000`0112d013 : 000000c0`01a1a0f0 000000c0`0335c060 00000000`00000027 00000000`00000030 : influxd+0xd9081e
000000c0`04b5ad68 00000000`011306d6 : 000000c0`0335c060 00000000`00000027 00000000`00000030 000000c0`04982620 : influxd+0xd2d013
000000c0`04b5ae00 00000000`0112ffea : 000000c0`0335c060 00000000`00000027 00000000`00000030 000000c0`049881e8 : influxd+0xd306d6
000000c0`04b5aed0 00000000`0112f7d2 : 000000c0`0335c060 00000000`00000027 00000000`00000030 000000c0`04982620 : influxd+0xd2ffea
000000c0`04b5b000 00000000`0112e4b6 : 000000c0`0335c060 00000000`00000027 00000000`00000030 000000c0`04982620 : influxd+0xd2f7d2
000000c0`04b5b090 00000000`0112e32a : 000000c0`0335c060 00000000`00000027 00000000`00000030 000000c0`04982620 : influxd+0xd2e4b6
000000c0`04b5b130 00000000`011378d9 : 000000c0`0335c060 00000000`00000027 00000000`00000030 000000c0`04982620 : influxd+0xd2e32a
000000c0`04b5b1d0 00000000`0113769d : 000000c0`049e4000 000000c0`0335c060 000000c0`04990168 00000000`00000000 : influxd+0xd378d9
000000c0`04b5b2d8 00000000`027dbb5e : 000000c0`049e4000 00000000`0703fa60 000000c0`04987f80 00000000`00000016 : influxd+0xd3769d
000000c0`04b5b328 00000000`02760390 : 000000c0`0461e140 00000000`00000000 00000000`00000001 000000c0`04b5b518 : influxd+0x23dbb5e
000000c0`04b5b3d8 00000000`02784c8f : 000000c0`049e6000 00000000`00000000 00000000`00000000 00000000`00a70a00 : influxd+0x2360390
000000c0`04b5b528 00000000`027848a5 : 000000c0`0460e300 000000c0`04530c40 00000000`04f55b18 000000c0`049e6000 : influxd+0x2384c8f
000000c0`04b5bab8 00000000`023bf77a : 000000c0`0460e300 000000c0`04530c40 00000000`04f42738 000000c0`04531f80 : influxd+0x23848a5
000000c0`04b5bb40 00000000`023c13a6 : 000000c0`04113b80 00000000`04f42738 000000c0`04552fc0 00000000`04f2ca50 : influxd+0x1fbf77a
000000c0`04b5bb90 00000000`023bf343 : 000000c0`04113b80 00000000`04f42738 000000c0`04552fc0 c143e7f7`8bd9971f : influxd+0x1fc13a6
000000c0`04b5bd40 00000000`023c91e8 : 000000c0`04113b80 00000000`04f42738 000000c0`04552fc0 000000c0`04b5bfb0 : influxd+0x1fbf343
000000c0`04b5bde8 00000000`012f706d : 00000000`04f42770 00000000`04f42738 000000c0`04552fc0 00000000`0703cca0 : influxd+0x1fc91e8
000000c0`04b5be10 00000000`012f6ce8 : 00000000`04f44378 000000c0`04113b80 00000000`00000000 00000000`00480981 : influxd+0xef706d
000000c0`04b5bfc0 00000000`00480981 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0xef6ce8
000000c0`04b5bfe0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : influxd+0x80981
Some more:
0 Id: 41c8.1160 Suspend: 0 Teb: 00000000`0028b000 Unfrozen
# Child-SP RetAddr Call Site
00 00000000`0733f298 00007ffc`aa47da3e ntdll!NtWaitForSingleObject+0x14
01 00000000`0733f2a0 00000000`004821c9 KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`0733f340 00000000`0048054d influxd+0x821c9
03 00000000`0733f4b0 00000000`00480601 influxd+0x8054d
04 00000000`0733f4e0 00000000`0043c91b influxd+0x80601
05 00000000`0733f508 00000000`0043ca56 influxd+0x3c91b
06 00000000`0733f540 00000000`0043bb45 influxd+0x3ca56
07 00000000`0733f558 00000000`00410e65 influxd+0x3bb45
08 00000000`0733f5c0 00000000`0043446e influxd+0x10e65
09 00000000`0733f628 00000000`0043436b influxd+0x3446e
0a 00000000`0733f670 00000000`00433fff influxd+0x3436b
0b 00000000`0733f6d8 00000000`00410f96 influxd+0x33fff
0c 00000000`0733f6f0 00000000`0043d677 influxd+0x10f96
0d 00000000`0733f720 00000000`0044fd55 influxd+0x3d677
0e 00000000`0733fc80 00000000`00424f96 influxd+0x4fd55
0f 00000000`0733fc98 00000000`0042ab45 influxd+0x24f96
10 00000000`0733fcc8 00000000`0042331d influxd+0x2ab45
11 00000000`0733fce0 00000000`00421eaf influxd+0x2331d
12 00000000`0733fd10 00000000`00421d7b influxd+0x21eaf
13 00000000`0733fd70 00000000`0047e9a9 influxd+0x21d7b
14 00000000`0733fd90 00000000`00421af3 influxd+0x7e9a9
15 000000c0`04b5a7d8 00000000`00412534 influxd+0x21af3
16 000000c0`04b5a838 00000000`0047223e influxd+0x12534
17 000000c0`04b5a860 00000000`00412625 influxd+0x7223e
18 000000c0`04b5a900 00000000`01143268 influxd+0x12625
19 000000c0`04b5a928 00000000`011a598e influxd+0xd43268
1a 000000c0`04b5a968 00000000`011a403b influxd+0xda598e
1b 000000c0`04b5a9c0 00000000`011a463b influxd+0xda403b
1c 000000c0`04b5aa68 00000000`0119693e influxd+0xda463b
1d 000000c0`04b5abd8 00000000`0118bba8 influxd+0xd9693e
1e 000000c0`04b5ac08 00000000`011aa8d3 influxd+0xd8bba8
1f 000000c0`04b5ac90 00000000`0119081e influxd+0xdaa8d3
20 000000c0`04b5acc8 00000000`0112d013 influxd+0xd9081e
21 000000c0`04b5ad68 00000000`011306d6 influxd+0xd2d013
22 000000c0`04b5ae00 00000000`0112ffea influxd+0xd306d6
23 000000c0`04b5aed0 00000000`0112f7d2 influxd+0xd2ffea
24 000000c0`04b5b000 00000000`0112e4b6 influxd+0xd2f7d2
25 000000c0`04b5b090 00000000`0112e32a influxd+0xd2e4b6
26 000000c0`04b5b130 00000000`011378d9 influxd+0xd2e32a
27 000000c0`04b5b1d0 00000000`0113769d influxd+0xd378d9
28 000000c0`04b5b2d8 00000000`027dbb5e influxd+0xd3769d
29 000000c0`04b5b328 00000000`02760390 influxd+0x23dbb5e
2a 000000c0`04b5b3d8 00000000`02784c8f influxd+0x2360390
2b 000000c0`04b5b528 00000000`027848a5 influxd+0x2384c8f
2c 000000c0`04b5bab8 00000000`023bf77a influxd+0x23848a5
2d 000000c0`04b5bb40 00000000`023c13a6 influxd+0x1fbf77a
2e 000000c0`04b5bb90 00000000`023bf343 influxd+0x1fc13a6
2f 000000c0`04b5bd40 00000000`023c91e8 influxd+0x1fbf343
30 000000c0`04b5bde8 00000000`012f706d influxd+0x1fc91e8
31 000000c0`04b5be10 00000000`012f6ce8 influxd+0xef706d
32 000000c0`04b5bfc0 00000000`00480981 influxd+0xef6ce8
33 000000c0`04b5bfe0 00000000`00000000 influxd+0x80981
1 Id: 41c8.b98 Suspend: 0 Teb: 00000000`00293000 Unfrozen
# Child-SP RetAddr Call Site
00 00000000`4d18faa8 00007ffc`aa47da3e ntdll!NtWaitForSingleObject+0x14
01 00000000`4d18fab0 00000000`004821c9 KERNELBASE!WaitForSingleObjectEx+0x8e
02 00000000`4d18fb50 00000000`0048054d influxd+0x821c9
03 00000000`4d18fcc0 00000000`00480601 influxd+0x8054d
04 00000000`4d18fcf0 00000000`0043c91b influxd+0x80601
05 00000000`4d18fd10 00000000`0043ca56 influxd+0x3c91b
06 00000000`4d18fd48 00000000`0044f97a influxd+0x3ca56
07 00000000`4d18fd60 00000000`0044f4d7 influxd+0x4f97a
08 00000000`4d18fdc0 00000000`0044611d influxd+0x4f4d7
09 00000000`4d18fe40 00000000`0044606a influxd+0x4611d
0a 00000000`4d18fe68 00000000`0047e8a5 influxd+0x4606a
0b 00000000`4d18fe90 00000000`02907c11 influxd+0x7e8a5
0c 00000000`4d18fe98 00000000`00000000 influxd!preUpdateHookTrampoline+0xd4571
I am not quite sure where to go from here as I feel like I have exhausted most of my debugging avenues. If anyone has any ideas or helpful hints it would be greatly appreciated!
