Influxd fatal error: runtime: out of memory

Welcome & Getting Started
1

I am using influxdb 1.7.4 version and facing “out of memory” issue while restarting influxd service. Also while running influx command to connect database it is giving “Failed to connect to http://localhost:8086”

Request help on fixing these issues

List of issues:
Issue1.
influxd[7097]: fatal error: runtime: out of memory
influxd[7097]: runtime stack:
influxd[7097]: runtime.throw(0x123d974, 0x16)
influxd[7097]: #011/usr/local/go/src/runtime/panic.go:608 +0x72
influxd[7097]: runtime.sysMap(0xc49c000000, 0x4000000, 0x1da9518)

2

Hello @ashish.singh1,
Welcome! Can you please upgrade to 1.7.10 or 2.0.0-beta.6?
Have you successfully run Influx before?
How did you install it and what are you running it on?
Finally, have you changed your influxdb.conf?
Thank you

3

Also @ashish.singh1 do you know if you are using TSI1? If you aren’t, then InfluxDB defaults to inmem which uses significantly more memory

Time Series Index (TSI) details | InfluxDB OSS 1.7 Documentation

1 Like
4

Influx was running fine before.
It was already installed on ubuntu server.
I haven’t changed influxdb.conf file. Existing configuration details:

[meta]

Where the metadata/raft database is stored

dir = “/var/lib/influxdb/meta”

[data]

The directory where the TSM storage engine stores TSM files.

dir = “/var/lib/influxdb/data”

The directory where the TSM storage engine stores WAL files.

wal-dir = “/var/lib/influxdb/wal”

Values in the range of 0-100ms are recommended for non-SSD disks.

wal-fsync-delay = “10ms”

cardinality datasets.

index-version = “tsi1”

Values without a size suffix are in bytes.

cache-max-memory-size = “256m”

Values without a size suffix are in bytes.

cache-snapshot-memory-size = “10m”

a new TSM file if the shard hasn’t received writes or deletes

cache-snapshot-write-cold-duration = “3m”

write or delete

compact-full-write-cold-duration = “1h”

Values without a size suffix are in bytes.

max-index-log-file-size = “512k”

5

I am using TSI1 config in influxdb.conf

6

How much memory do you have on this server and how big is your /var/lib/influxdb directory?

7

@Esity
Server configuration details:
Memory: 18 GB
CPU: 2 processor Intel
Ubuntu 16.04
Influxdb directory size is 16GB

8

I really can’t help with this one as it is something on your ubuntu server side. When you run free -m, what does it say? I am guessing you are running too many things on that server and you don’t have enough ram

1 Like
9

@Esity Only influxdb service is consuming CPU and RAM. If I stop this service RAM and CPU has enough capacity. See below output of Top and free command

var/lib/influxdb# free -g
total used free shared buff/cache available
Mem: 17 17 0 0 0 0

/var/lib/influxdb# ps -ef | grep influx
influxdb 14036 1 99 15:48 ? 00:00:26 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
root 14047 10765 0 15:49 pts/0 00:00:00 grep --color=auto influx

Tasks: 176 total, 3 running, 173 sleeping, 0 stopped, 0 zombie
%Cpu(s): 69.8 us, 12.2 sy, 0.0 ni, 0.2 id, 17.2 wa, 0.0 hi, 0.7 si, 0.0 st
KiB Mem : 18497060 total, 612128 free, 13889176 used, 3995756 buff/cache
KiB Swap: 999420 total, 754848 free, 244572 used. 4287780 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14024 influxdb 20 0 26.331g 0.016t 3.427g R 162.7 93.5 0:51.29 influxd
5937 root 20 0 60220 6152 6020 S 0.7 0.0 18:46.26 systemd-journal
7 root 20 0 0 0 0 R 0.3 0.0 166:22.59 rcu_sched

10

@ashish.singh1 How big is the director that your influxdb data lives? You can run this sudo du -sh /var/lib/influxdb/data/* to determine it assume it is in the default path

11

I had the same issue with this. Turns out that the telegraf database had grown to 2.7g. Compare to data being used at 400,000+ data points at 56M. Deleted entire telegraf folder at /var/lib/influxdb/data/telegraf and instance booted up fine.

Tip: Found that telegraf.conf was recording metrics every 1s with no retention policy. see ‘/etc/telegraf/telegraf.conf’.