So we process lots of data in our project. Our data normally have around 1000 columns and more than 500K rows. Just quick numbers from 5-10 Gb of raw compressed files we generate around 100gb of influx data. Now our issue is when the we restart influxdb on Windows using CMD it tries to load TSM files into the RAM until we run out of ram “32 GB” then the fail. It could be idxl or tsm files that are filling the RAM but I am not sure. Can someone explain why influxdb trying to load so much data into the ram?
Startup logs that fails
ts=2024-02-09T11:23:44.000733Z lvl=info msg="Opened file" log_id=0nFP_psG000 service=storage-engine engine=tsm1 service=filestore path=.influxdbv2\\engine\\data\\dedff9e6f31668e5\\autogen\\100\\000026234-000000001.tsm id=13 duration=1703.891ms
ts=2024-02-09T11:23:44.001538Z lvl=info msg="Opened file" log_id=0nFP_psG000 service=storage-engine engine=tsm1 service=filestore path=.influxdbv2\\engine\\data\\dedff9e6f31668e5\\autogen\\100\\000026123-000000004.tsm id=2 duration=2041.443ms
ts=2024-02-09T11:23:44.001538Z lvl=info msg="Opened file" log_id=0nFP_psG000 service=storage-engine engine=tsm1 service=filestore path=.influxdbv2\\engine\\data\\dedff9e6f31668e5\\autogen\\100\\000026232-000000001.tsm id=11 duration=1705.209ms
ts=2024-02-09T11:23:46.249358Z lvl=info msg="Opened file" log_id=0nFP_psG000 service=storage-engine engine=tsm1 service=filestore path=.influxdbv2\\engine\\data\\dedff9e6f31668e5\\autogen\\100\\000023713-000000001.tsm id=0 duration=4290.368ms
ts=2024-02-09T11:23:46.382695Z lvl=info msg="Opened file" log_id=0nFP_psG000 service=storage-engine engine=tsm1 service=filestore path=.influxdbv2\\engine\\data\\dedff9e6f31668e5\\autogen\\84\\000004952-000000003.tsm id=0 duration=3181.329ms
ts=2024-02-09T11:23:46.411752Z lvl=info msg="Opened shard" log_id=0nFP_psG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=.influxdbv2\\engine\\data\\dedff9e6f31668e5\\autogen\\84 duration=6624.436ms
ts=2024-02-09T11:23:52.899079Z lvl=info msg="Opened file" log_id=0nFP_psG000 service=storage-engine engine=tsm1 service=filestore path=.influxdbv2\\engine\\data\\dedff9e6f31668e5\\autogen\\100\\000025995-000000005.tsm id=1 duration=10939.098ms
ts=2024-02-09T11:23:52.900119Z lvl=info msg="Opened shard" log_id=0nFP_psG000 service=storage-engine service=store op_name=tsdb_open index_version=tsi1 path=.influxdbv2\\engine\\data\\dedff9e6f31668e5\\autogen\\100 duration=13112.519ms
The error logs
runtime: VirtualAlloc of 8192 bytes failed with errno=1455
fatal error: out of memory
runtime: VirtualAlloc of 8192 bytes failed with errno=1455
fatal error: out of memory
runtime: VirtualAlloc of 655360 bytes failed with errno=1455
fatal error: out of memory
fatal error: runtime: cannot allocate memory
runtime stack:
runtime.throw({0x406e904?, 0xca4d83b000?})
/go/src/runtime/panic.go:1047 +0x65 fp=0x3304fd08 sp=0x3304fcd8 pc=0x43b905
runtime.sysUsedOS(0xca4d83a000, 0x2000)
/go/src/runtime/mem_windows.go:83 +0x1c5 fp=0x3304fd68 sp=0x3304fd08 pc=0x419e05
runtime.sysUsed(0x5eb79a0?, 0xfd37b778?, 0x24f0a00?)
/go/src/runtime/mem.go:77 +0x25 fp=0x3304fd88 sp=0x3304fd68 pc=0x419905
runtime.(*mheap).allocSpan(0x5eb79a0, 0x1, 0x0, 0x82?)
/go/src/runtime/mheap.go:1340 +0x455 fp=0x3304fe20 sp=0x3304fd88 pc=0x429d35
runtime.(*mheap).alloc.func1()
/go/src/runtime/mheap.go:961 +0x65 fp=0x3304fe68 sp=0x3304fe20 pc=0x429505
runtime.systemstack()
/go/src/runtime/asm_amd64.s:496 +0x48 fp=0x3304fe70 sp=0x3304fe68 pc=0x46d708
The config File
assets-path: ""
bolt-path: .\.influxdbv2\influxd.bolt
e2e-testing: false
engine-path: .\.influxdbv2\engine
feature-flags: {}
flux-log-enabled: false
hardening-enabled: false
http-bind-address: :8086
http-idle-timeout: 3m0s
http-read-header-timeout: 10s
http-read-timeout: 0s
http-write-timeout: 0s
influxql-max-select-buckets: 0
influxql-max-select-point: 0
influxql-max-select-series: 0
instance-id: ""
key-name: ""
log-level: info
metrics-disabled: false
nats-max-payload-bytes: 0
nats-port: 0
no-tasks: false
pprof-disabled: false
query-concurrency: 1024
query-initial-memory-bytes: 0
query-max-memory-bytes: 0
query-memory-bytes: 0
query-queue-size: 1024
reporting-disabled: false
secret-store: bolt
session-length: 60
session-renew-disabled: false
sqlite-path: ""
storage-cache-max-memory-size: 30041824
storage-cache-snapshot-memory-size: 24400
storage-cache-snapshot-write-cold-duration: 10m0s
storage-compact-full-write-cold-duration: 4h0m0s
storage-compact-throughput-burst: 31648
storage-max-concurrent-compactions: 0
storage-max-index-log-file-size: 48576
storage-no-validate-field-size: false
storage-retention-check-interval: 30m0s
storage-series-file-max-concurrent-snapshot-compactions: 0
storage-series-id-set-cache-size: 0
storage-shard-precreator-advance-period: 30m0s
storage-shard-precreator-check-interval: 10m0s
storage-tsm-use-madv-willneed: false
storage-validate-keys: false
storage-wal-fsync-delay: 0s
storage-wal-max-concurrent-writes: 0
storage-wal-max-write-delay: 10m0s
storage-write-timeout: 10s
store: disk
testing-always-allow-setup: false
tls-cert: ""
tls-key: ""
tls-min-version: "1.2"
tls-strict-ciphers: false
tracing-type: ""
ui-disabled: false
vault-addr: ""
vault-cacert: ""
vault-capath: ""
vault-client-cert: ""
vault-client-key: ""
vault-client-timeout: 0s
vault-max-retries: 0
vault-skip-verify: false
vault-tls-server-name: ""
vault-token: ""
index-version: "tsi1"
