100% CPU and IO usage after starting influxdb

birbird · May 8, 2019, 5:47am

Hi

My problem is just start influxdb, all the resource will be occupied by influxdb, even ssh will be hard to respond. The high resource usage continues util I stop the influxdb service.

What I see is when starting influxdb, it reads a lot of files, then IO and CPU is fully used, and 80% memory is used.

Some logs are

May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.007046Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6187 duration=461.953ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.023842Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6191/_00001.wal size=10511980
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.036073Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6230 duration=80.818ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.040560Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6231/_00001.wal size=1984614
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.049466Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6247 duration=131.808ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.053762Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6192/_00001.wal size=4488153
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.118253Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6229 duration=163.096ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.124082Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6232/_00001.wal size=6272877
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.135969Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6231 duration=99.855ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.137619Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6193/_00001.wal size=4094322
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.273753Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6192 duration=224.258ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.275168Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6233/_00001.wal size=2647979
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.310064Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6191/_00002.wal size=107599
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.314540Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6193 duration=178.521ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.315665Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6194/_00001.wal size=1415328
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.321216Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6191 duration=314.126ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.321924Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6234/_00001.wal size=1188597
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.322386Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6232 duration=204.077ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.323325Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6195/_00001.wal size=3405310
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.365812Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6234 duration=44.569ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.366670Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6235/_00001.wal size=2319214
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.374033Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6194 duration=59.451ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.374883Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6196/_00001.wal size=5545096
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.396389Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6233 duration=122.583ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.397227Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6197/_00001.wal size=5698647
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.441782Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6195 duration=119.368ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.441863Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6235 duration=76.003ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.442686Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6198/_00001.wal size=2668446
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.443738Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6236/_00001.wal size=3314645
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.852819Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6236 duration=410.984ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.855105Z lvl=info msg="Opened file" log_id=0FHTlzaG000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/market_data_test/autogen/6237/000000001-000000001.tsm id=0 duration=0.504ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.857602Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6198 duration=415.707ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.859636Z lvl=info msg="Opened file" log_id=0FHTlzaG000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/market_data_test/autogen/6238/000000001-000000001.tsm id=0 duration=0.414ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.861868Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6237 duration=8.955ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.862899Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6239/_00001.wal size=3780422
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.890042Z lvl=info msg="Opened shard" log_id=0FHTlzaG000 service=store trace_id=0FHTlz~0000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/market_data_test/autogen/6238 duration=32.376ms
May 08 13:16:32 iZj6c7cyemktk3s0msphv8Z influxd[4926]: ts=2019-05-08T05:16:32.891176Z lvl=info msg="Reading file" log_id=0FHTlzaG000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/market_data_test/autogen/6199/_00001.wal size=4554514

And I can see some error

May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: fatal error: runtime: out of memory
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime stack:
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.throw(0x120f230, 0x16)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/panic.go:608 +0x72
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.sysMap(0xc1ac000000, 0x4000000, 0x1d45398)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/mem_linux.go:156 +0xc7
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.(*mheap).sysAlloc(0x1d2b6a0, 0x4000000, 0x58008cb1ef, 0x9aa9bf)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/malloc.go:619 +0x1c7
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.(*mheap).grow(0x1d2b6a0, 0x18, 0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/mheap.go:920 +0x42
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.(*mheap).allocSpanLocked(0x1d2b6a0, 0x18, 0x1d453a8, 0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/mheap.go:848 +0x337
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.(*mheap).alloc_m(0x1d2b6a0, 0x18, 0x100, 0x14654e8)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/mheap.go:692 +0x119
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.(*mheap).alloc.func1()
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/mheap.go:759 +0x4c
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.(*mheap).alloc(0x1d2b6a0, 0x18, 0x7eebc8010100, 0x7)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/mheap.go:758 +0x8a
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.largeAlloc(0x30000, 0x7eff62a80001, 0x459c9a)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/malloc.go:1019 +0x97
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.mallocgc.func1()
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/malloc.go:914 +0x46
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.systemstack(0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/asm_amd64.s:351 +0x66
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.mstart()
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/proc.go:1229
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: goroutine 377 [running]:
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.systemstack_switch()
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc08ede0a90 sp=0xc08ede0a88 pc=0x45b8e0
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.mallocgc(0x30000, 0x10bc5c0, 0xc08ede0b01, 0x2)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/malloc.go:913 +0x896 fp=0xc08ede0b30 sp=0xc08ede0a90 pc=0x40def6
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.growslice(0x10bc5c0, 0xc1a915c000, 0x24a3, 0x2600, 0x2710, 0x1000000000000f0, 0x0, 0x30)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/slice.go:204 +0x145 fp=0xc08ede0b98 sp=0xc08ede0b30 pc=0x444b25
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*entry).add(0xc1a0dc8080, 0xc1abe28800, 0x26d, 0x26d, 0xc1a0b44ff8, 0x30)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:96 +0x253 fp=0xc08ede0c10 sp=0xc08ede0b98 pc=0xe99943
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*partition).write(0xc1a0b40540, 0xc1aa9c9a70, 0x30, 0x30, 0xc1abe28800, 0x26d, 0x26d, 0x0, 0x0, 0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/ring.go:243 +0x2c8 fp=0xc08ede0c60 sp=0xc08ede0c10 pc=0xf01808
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*ring).write(0xc1a0b403a0, 0xc1aa9c9a70, 0x30, 0x30, 0xc1abe28800, 0x26d, 0x26d, 0xc0ee20a000, 0x0, 0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/ring.go:100 +0xa0 fp=0xc08ede0cc0 sp=0xc08ede0c60 pc=0xf00a70
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Cache).WriteMulti(0xc158311340, 0xc1a4e63200, 0x0, 0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:343 +0x2cb fp=0xc08ede0e60 sp=0xc08ede0cc0 pc=0xe9ac9b
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load.func1(0xc08ede1288, 0xc1a0b402a0, 0xc08ede1278, 0xc158311340, 0x0, 0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:747 +0x64c fp=0xc08ede1238 sp=0xc08ede0e60 pc=0xf1464c
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*CacheLoader).Load(0xc1a0b402a0, 0xc158311340, 0x1, 0xc1a0b402a0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:758 +0xa6 fp=0xc08ede12a8 sp=0xc08ede1238 pc=0xe9c516
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).reloadCache(0xc162091b00, 0x0, 0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2252 +0x186 fp=0xc08ede13f0 sp=0xc08ede12a8 pc=0xed0596
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).Open(0xc162091b00, 0xc000160200, 0xc1949ee9c0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:745 +0x2a0 fp=0xc08ede1478 sp=0xc08ede13f0 pc=0xec6060
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb.(*Shard).Open.func1(0xc048076fc0, 0x0, 0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/shard.go:340 +0x36d fp=0xc08ede16a8 sp=0xc08ede1478 pc=0x9c6ecd
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb.(*Shard).Open(0xc048076fc0, 0xc000160240, 0x34)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/shard.go:351 +0x2f fp=0xc08ede1700 sp=0xc08ede16a8 pc=0x9aa9bf
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb.(*Store).loadShards.func1(0xc00008a5a0, 0xc00010d600, 0xc00031acc0, 0xc00008a600, 0x11edba0, 0xc001554990, 0xc000532190, 0xc000388617, 0x10, 0xc0022ca088, ...)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:404 +0x517 fp=0xc08ede1f78 sp=0xc08ede1700 pc=0x9c8187
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: runtime.goexit()
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc08ede1f80 sp=0xc08ede1f78 pc=0x45d9c1
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: created by github.com/influxdata/influxdb/tsdb.(*Store).loadShards
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:362 +0x19d6
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: goroutine 1 [chan receive]:
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb.(*Store).loadShards(0xc00010d600, 0x0, 0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:421 +0x644
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/tsdb.(*Store).Open(0xc00010d600, 0x0, 0x0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/tsdb/store.go:221 +0x25e
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open(0xc00037c7e0, 0xc0000ba300, 0xc00037c7e0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:435 +0x857
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: github.com/influxdata/influxdb/cmd/influxd/run.(*Command).Run(0xc0001bad10, 0xc00003a0a0, 0x2, 0x2, 0x0, 0xc00003a0a0)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:149 +0xe2d
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: main.(*Main).Run(0xc000531f58, 0xc00003a0a0, 0x2, 0x2, 0xc000531f68, 0xf2ff56)
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:81 +0x14f
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: main.main()
May 08 13:16:36 iZj6c7cyemktk3s0msphv8Z influxd[4926]: /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:45 +0x12f

The host is a virtual machine with 4 CPU cores and 8GB memory

The influxdb version is 1.7.2

The data size is 80G

And I closed the http, can exclude the impact outside.

What’s the problem and how to fix it? Any help will be highly appreciated!

MarcV · May 8, 2019, 5:56am

Hi @birbird,

You are using inmem indexes,
Please have a look here for example to convert to tsi indexes…

https://community.influxdata.com/t/migrate-index-version-from-inmem-to-tsi1-missing-data/9429

birbird · May 8, 2019, 6:21am

Hi @MarcV
Thanks a lot.

I changed the index version to
index-version = “tsi1”

and restart the db, the problem still there

MarcV · May 8, 2019, 6:46am

Hi birbird ,

have you converted the indexes as well with influx_inspect buildtsi ?
( must be done with the user that runs influxd )

birbird · May 8, 2019, 6:49am

Yes, I have done that by

sudo -u influxdb bash -c 'influx_inspect buildtsi -datadir data -waldir wal'

MarcV · May 8, 2019, 6:57am

Hi ,

can you do the same but with the complete path to data and wal ?

probably something like this :

sudo -u influxdb bash -c 'influx_inspect buildtsi -datadir /var/lib/influxdb/data -waldir /var/lib/influxdb/wal'

can you check in the logfiles if you still find “op_name=tsdb_open index_version=inmem”

birbird · May 8, 2019, 7:44am

I ran buildtsi several times, the result are all “tsi1 index already exists, skipping”

sudo -u influxdb bash -c 'influx_inspect buildtsi -datadir /data/influxdb/data -waldir /data/influxdb/wal'

2019-05-08T07:33:27.292985Z     info    Rebuilding shard        {"log_id": "0FHaceUW000", "db_instance": "market_data_test", "db_rp": "autogen", "db_shard_id": 6386}
2019-05-08T07:33:27.293176Z     info    Checking index path     {"log_id": "0FHaceUW000", "db_instance": "market_data_test", "db_rp": "autogen", "db_shard_id": 6386, "path": "/data/influxdb/data/market_data_test/autogen/6386/index"}
2019-05-08T07:33:27.293191Z     info    tsi1 index already exists, skipping     {"log_id": "0FHaceUW000", "db_instance": "market_data_test", "db_rp": "autogen", "db_shard_id": 6386, "path": "/data/influxdb/data/market_data_test/autogen/6386/index"}
2019-05-08T07:33:27.293073Z     info    tsi1 index already exists, skipping     {"log_id": "0FHaceUW000", "db_instance": "market_data_test", "db_rp": "autogen", "db_shard_id": 6388, "path": "/data/influxdb/data/market_data_test/autogen/6388/index"}
2019-05-08T07:33:27.293136Z     info    tsi1 index already exists, skipping     {"log_id": "0FHaceUW000", "db_instance": "market_data_test", "db_rp": "autogen", "db_shard_id": 6390, "path": "/data/influxdb/data/market_data_test/autogen/6390/index"}

I read the logs of a db start, it generate thousands of lines in 30 seconds.
They are all “index_version=tsi1” now, no inmem.
What I see is, the start process is reading all the db file one by one, each need dozens or hundrends of ms. It’s no surprise the server freeze under the load.

But why influx need to read those files during start?

MarcV · May 8, 2019, 7:57am

Do you still have the same problem or is it better witht the tsi indexes ?

birbird · May 8, 2019, 8:22am

Yes, the problem is still there.
Whenever start the influxdb, the whole server freeze, the only way is restarting the server

MarcV · May 8, 2019, 8:23am

do you still see this out of memory in the logs ?

birbird · May 8, 2019, 8:26am

no out of memory now

MarcV · May 8, 2019, 8:35am

The server freezes completely?
If it is no memory problem anymore , it can be a cpu usage problem ,
you could try to start influxdb after setting GOMAXPROCS

here are also some recommendations for the hardware …

Topic		Replies	Views
Influxdb CPU usage - is this normal? influxdb	1	2659	May 26, 2017
InfluxDB write IO - sky rocketing on fast hardware influxdb	0	410	September 14, 2020
Debugging High CPU Usage when IDLE influxdb	2	1700	March 22, 2023
Disk I/O, CPU and Swap Store	4	2353	April 5, 2019
High Swap usage on influxdb InfluxDB 1 performance	3	247	March 26, 2024

100% CPU and IO usage after starting influxdb

Related topics