Infuxdb 2.0 beta14: high cpu usage

InfluxDB 2
1

I have 2 instance of influxd, one is 2.0 alpha20, other one is 2.0 beta14.

I notice that influxd with version 2.0 beta14 occurs a very high CPU usage: about nearly 60% system usage, 10% user usage. 2.0 alpha20 does NOT have this issue.

top - 12:33:51 up 22 days, 19:17,  1 user,  load average: 6.95, 10.28, 16.19
Tasks: 369 total,   2 running, 367 sleeping,   0 stopped,   0 zombie
%Cpu0  : 13.3 us, 68.1 sy,  0.0 ni, 17.7 id,  0.0 wa,  0.0 hi,  0.9 si,  0.0 st
%Cpu1  : 21.7 us, 66.1 sy,  0.0 ni, 12.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  : 17.4 us, 62.6 sy,  0.0 ni, 20.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  : 13.9 us, 71.3 sy,  0.0 ni, 14.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  : 18.4 us, 71.9 sy,  0.0 ni,  9.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  : 15.4 us, 71.8 sy,  0.0 ni, 12.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  : 15.7 us, 65.2 sy,  0.0 ni, 19.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  : 20.0 us, 60.0 sy,  0.0 ni, 19.1 id,  0.9 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu8  : 12.9 us, 68.1 sy,  0.0 ni, 19.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu9  : 12.2 us, 65.2 sy,  0.0 ni, 22.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu10 : 12.9 us, 73.3 sy,  0.0 ni, 13.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu11 : 13.8 us, 62.9 sy,  0.0 ni, 23.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu12 : 38.3 us, 50.4 sy,  0.0 ni, 11.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu13 : 10.3 us, 60.3 sy,  0.0 ni, 29.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu14 : 18.8 us, 70.1 sy,  0.0 ni, 11.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu15 : 14.0 us, 69.3 sy,  0.0 ni, 16.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu16 : 13.0 us, 64.3 sy,  0.0 ni, 22.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu17 : 14.9 us, 65.8 sy,  0.0 ni, 19.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu18 : 17.1 us, 70.9 sy,  0.0 ni, 12.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu19 : 11.1 us, 70.1 sy,  0.0 ni, 18.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu20 : 17.5 us, 71.9 sy,  0.0 ni, 10.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu21 : 11.1 us, 68.4 sy,  0.0 ni, 20.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu22 : 13.0 us, 75.7 sy,  0.0 ni, 11.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu23 : 18.3 us, 70.4 sy,  0.0 ni, 11.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu24 : 12.8 us, 65.0 sy,  0.0 ni, 22.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu25 : 12.9 us, 68.1 sy,  0.0 ni, 19.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu26 : 18.1 us, 71.6 sy,  0.0 ni, 10.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu27 : 12.1 us, 70.7 sy,  0.0 ni, 17.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu28 : 12.0 us, 53.8 sy,  0.0 ni, 34.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu29 : 12.8 us, 61.5 sy,  0.0 ni, 25.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu30 : 14.7 us, 73.3 sy,  0.0 ni, 12.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu31 : 19.7 us, 71.8 sy,  0.0 ni,  8.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 13164649+total, 50271200 free,  2852248 used, 78523048 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 12699140+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
15191 influxdb  20   0  0.200t 0.021t 0.020t S  3200 17.5  50822:53 /usr/bin/influxd run --reporting-disabled --http-bind-address=:8080 --bolt-path=/data1/influxdb/bolt --engine-path=/data1/influxdb/engine --tracing-type=jaeger
21167 root      20   0  139892   5148   3940 S   0.9  0.0   0:00.02 sshd: root@pts/0
21242 root      20   0  144588   2292   1408 R   0.9  0.0   0:00.06 top
    1 root      20   0  208848  23556   2088 S   0.0  0.0   5:29.17 /usr/lib/systemd/systemd --system --deserialize 20
    2 root      20   0       0      0      0 S   0.0  0.0   0:13.70 [kthreadd]
    3 root      20   0       0      0      0 S   0.0  0.0   0:10.19 [ksoftirqd/0]
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 [kworker/0:0H]
    8 root      rt   0       0      0      0 S   0.0  0.0   0:05.45 [migration/0]

Then I run the profile for influxd, it seems libflux take most of cpu time.
influxdb-profile.tar.gz (91.5 KB)

2

  1. Does the query return faster, same timeframe, or slower compared with the alpha 20?

  2. Can you share the query youre running?

  3. If you can, can you share the data / describe the make up of the data

3

Thanks @FixTestRepeat.

Does the query return faster, same timeframe, or slower compared with the alpha 20?

Slower;
alapha20: most of request of /query requests are in 25-125ms buckets
beta14: most of request of /query requests are in 125ms-625ms buckets

Can you share the query youre running?
If you can, can you share the data / describe the make up of the data

Actually I don’t know how to find out which flux query causing this issue. There are too many measurements and flux query.

4

Definitely sounds like some regression issue , but may have to use process of elimination to figure out which one. How many different queries do you have running, and on what frequency?
Maybe worth trying turning on debug logging to gather more info too

5

We find that beta8 works well.