Hi,
I’m facing massive memory issues when querying data of my fairly small database. In my case the whole database size is less than 10% of the RAM available for InfluxDB, so InfluxDB should be able to cache EVERYTHING easily. But InfluxDB eats all my memory until the system stalls. If I set upper limits for the influxDB memory usage, an OOM error occurs, and influxDB terminates. I’m using InfluxDB version 1.7.9 .
How can I resolve this ?
Here are the numbers:
-
DB - Node has 4 CPU’s and 16 GB RAM
-
Database size < 1.5 GB:
du -sh *
1.3G data
4.0K meta
17M wal
- Cardinality < 100.000
show series cardinality
cardinality estimation
68002
- No write load at all (static data that was imported just once):
select time,last(queryReq) as queryReq,last(queryRespBytes) as queryRespBytes,last(req) as req,last(writeReq) as writeReq from httpd where time > now() - 10m group by time(1m);
name: httpd
time queryReq queryRespBytes req writeReq
2020-02-04T11:56:00Z 66 893287 66 0
2020-02-04T11:57:00Z 67 909189 67 0
2020-02-04T11:58:00Z 68 925091 68 0
2020-02-04T11:59:00Z 69 940993 69 0
2020-02-04T12:00:00Z 70 956895 70 0
2020-02-04T12:01:00Z 71 972797 71 0
2020-02-04T12:02:00Z 73 989075 73 0
2020-02-04T12:03:00Z 76 1005153 76 0
2020-02-04T12:04:00Z 78 1021148 78 0
2020-02-04T12:05:00Z 81 1038006 81 0
2020-02-04T12:06:00Z 83 1054846 83 0
And this I see in the syslog when setting the memory limit for the influxdb service like this: LimitAS=14500000000
2020-02-04T13:09:27.504490+01:00 i037algeat02 influxd[13324]: runtime: out of memory: cannot allocate 8192-byte block (12078055424 in use)
2020-02-04T13:09:27.505249+01:00 i037algeat02 influxd[13324]: fatal error: out of memory
2020-02-04T13:09:27.505478+01:00 i037algeat02 influxd[13324]: runtime: out of memory: cannot allocate 8192-byte block (12078055424 in use)
2020-02-04T13:09:27.505663+01:00 i037algeat02 influxd[13324]: fatal error: out of memory
2020-02-04T13:09:27.505855+01:00 i037algeat02 influxd[13324]: runtime: out of memory: cannot allocate 8192-byte block (12078055424 in use)
2020-02-04T13:09:27.506032+01:00 i037algeat02 influxd[13324]: fatal error: out of memory
2020-02-04T13:09:27.513353+01:00 i037algeat02 influxd[13324]: goroutine 63389 [running]:
2020-02-04T13:09:27.515729+01:00 i037algeat02 influxd[13324]: runtime.throw(0x1529579, 0xd)
2020-02-04T13:09:27.516017+01:00 i037algeat02 influxd[13324]: #011/usr/local/go/src/runtime/panic.go:617 +0x72 fp=0xc0e902a668 sp=0xc0e902a638 pc=0x42f482
2020-02-04T13:09:27.516201+01:00 i037algeat02 influxd[13324]: runtime.(*mcache).refill(0x7fd97904c6d0, 0x3c)
2020-02-04T13:09:27.516376+01:00 i037algeat02 influxd[13324]: #011/usr/local/go/src/runtime/mcache.go:137 +0xf3 fp=0xc0e902a688 sp=0xc0e902a668 pc=0x418b63
2020-02-04T13:09:27.517221+01:00 i037algeat02 influxd[13324]: runtime.(*mcache).nextFree(0x7fd97904c6d0, 0x3c, 0x8b, 0xc0e902a778, 0x40d922)
2020-02-04T13:09:27.517436+01:00 i037algeat02 influxd[13324]: #011/usr/local/go/src/runtime/malloc.go:786 +0x88 fp=0xc0e902a6c0 sp=0xc0e902a688 pc=0x40d298
2020-02-04T13:09:27.517610+01:00 i037algeat02 influxd[13324]: runtime.mallocgc(0x380, 0x1464c20, 0x1, 0x0)
2020-02-04T13:09:27.517782+01:00 i037algeat02 influxd[13324]: #011/usr/local/go/src/runtime/malloc.go:939 +0x76e fp=0xc0e902a760 sp=0xc0e902a6c0 pc=0x40dbae
2020-02-04T13:09:27.517968+01:00 i037algeat02 influxd[13324]: runtime.makeslice(0x1464c20, 0x21, 0x21, 0x21)
2020-02-04T13:09:27.518140+01:00 i037algeat02 influxd[13324]: #011/usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc0e902a790 sp=0xc0e902a760 pc=0x44572c
2020-02-04T13:09:27.518387+01:00 i037algeat02 influxd[13324]: github.com/influxdata/influxdb/tsdb/engine/tsm1.DecodeStringBlock(0x7fd937dcfef9, 0xf5, 0xce5721, 0xc2cff95330, 0x422825, 0xc2cf799af8, 0x7
fd8f1ce231b, 0x80, 0x35f)
Here are the corresponding lines in _internal.runtime:
time Alloc Frees HeapAlloc HeapIdle HeapInUse HeapObjects HeapReleased HeapSys Lookups Mallocs NumGC NumGoroutine PauseTotalNs Sys TotalAlloc hostname
2020-02-04T12:09:00Z 4426920008 11055755674 4426920008 6843310080 4765507584 33466245 4654440448 11608817664 0 11089221919 1118 26 162796136 12289245072 1308994101512 i037algeat02
2020-02-04T12:09:10Z 5664947552 11056372840 5664947552 5718212608 5890572288 49459594 4654440448 11608784896 0 11105832434 1118 28 162796136 12289245072 1310232129056 i037algeat02
2020-02-04T12:09:20Z 5916271312 11125393141 5916271312 5619187712 5989203968 37505715 4611866624 11608391680 0 11162898856 1122 49 163078714 12289507216 1318029134712 i037algeat02
thanks
Reinhard
