A lot of IO occurs for ifluxd process ,even application stop

InfluxDB 2
1

[root@prod1 06]# ps -ef|grep influxdb|grep data|grep -v grep
root 2889694 1 71 12:00 ? 05:11:48 /data/ces/evo/influxdb/influxd
[root@prod1 06]# pidstat -d 2 10 -p 2889694
Linux 4.19.90-25.29.v2101.ky10.x86_64 (prod1) 03/23/2024 x86_64 (16 CPU)

07:14:57 PM UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
07:14:59 PM 0 2889694 340794.00 1406.00 0.00 87 influxd
07:15:01 PM 0 2889694 329780.00 2412.00 0.00 50 influxd
07:15:03 PM 0 2889694 388044.00 456.00 0.00 133 influxd
07:15:05 PM 0 2889694 351654.00 2596.00 0.00 163 influxd
07:15:07 PM 0 2889694 349186.00 2296.00 0.00 117 influxd
07:15:09 PM 0 2889694 342432.00 2278.00 0.00 97 influxd
^C
Average: 0 2889694 350315.00 1907.33 0.00 108 influxd
[root@prod1 06]#

2

kB_rd/s column show 340794 KB per second

3

300MB I/O PER SECOND! even application had stoped!
We want to know what caused it and how to solve it. Thank you

4

At the very least I think you need to tell us something about the system
you’re asking about.

  • what sort of hardware is this? CPU cores / RAM would be a good start.

  • it looks like you’re running Linux - which distribution & version?

  • which version of InfluxDB are you running?

  • how much data do you have in InfluxDB?

  • what rate is new data added to InfluxDB?

  • what retention policy are you using?

Try to give us enough information that we would be able to reproduce the same
setup for ourselves in order to understand what’s happening.

Antony.

5

1、what sort of hardware is this? CPU cores / RAM would be a good start.

hardware: X86 server
CPU cores / RAM: 16Core / 64GB RAM

2、it looks like you’re running Linux - which distribution & version?

	# uname -a
	Linux XXX 4.19.90-25.29.x86_64 #1 SMP Thu Jun 29 12:23:04 CST 2023 x86_64 x86_64 x86_64 GNU/Linux

3、which version of InfluxDB are you running?

 InfluxDB v2.7.1 (git: 407fa622e9) build_date: 2023-04-28T13:24:27Z

4、how much data do you have in InfluxDB?

 130GByte data total

5、what rate is new data added to InfluxDB?

500MBytes per minute

6、what retention policy are you using?

 retention policy: 90 days

7、influxdb configuration file,influxdb.conf

	query-concurrency: 30
	storage-cache-max-memory-size: 10737418240
	storage-cache-snapshot-memory-size: 52214400
	storage-cache-snapshot-write-cold-duration: 30m0s
	storage-compact-full-write-cold-duration: 4h0m0s
	storage-max-concurrent-compactions: 1
	storage-compact-throughput-burst: 335544320
	storage-max-index-log-file-size: 8388608
	storage-series-id-set-cache-size: 100