InfluxDB won't start - Failed to connect to streaming server

Hi all,

I have a problem with starting InfluxDB. I’m getting following error in log:
lvl=error msg=“Failed to connect to streaming server” log_id=0SG22jwl000 error=“read tcp 127.0.0.1:35466->127.0.0.1:39655: i/o timeout”
Feb 11 09:09:35 699ledaidb01-st influxd[2591]: Error: read tcp 127.0.0.1:35466->127.0.0.1:39655: i/o timeout
Feb 11 09:09:35 699ledaidb01-st influxd[2591]: See ‘influxd -h’ for help

OS: CentOS 8
InfluxDB version: Influx CLI 2.0.2 (git: 84496e507a) build_date: 2020-11-19T03:59:35Z

Until today it worked normally. Problems started first with error “too many files opened” and after that it’s not possible to restart InfluxDB service.

Can you please help me?

Thank you and best regards,
Domagoj

Hi,

It means you’ve hit the limit for maximum possible open files. You can increase this.

This article shows you how.

you can increase it by issuing a command, but this is temporary. For a permanent increase you will need to edit a few files.

The instructions i have saved for my Ubuntu server

Setting 

ulimit -n65535

This will only address the problem temporarily. To do this permanently we need to edit  some files. 


Files to edit: 

/etc/sysctl.conf
/etc/security/limits.conf
/etc/pam.d/common-session

Back all files up - Run this command for each file, creating a backup of each one in your home directory. 

Sudo cp <path/to/file/> ~/filename 


Edit sysctl:

Sudo nano /etc/sysctl.conf

Add the following line (in lowercase)

fs.file-max = 65535


Save the file and refresh - 

sudo sysctl -p 

Edit limits.conf

sudo nano /etc/security/limits.conf

Add the following

* soft     nproc          65535    
* hard     nproc          65535   
* soft     nofile         65535   
* hard     nofile         65535
root soft     nproc          65535    
root hard     nproc          65535   
root soft     nofile         65535   
root hard     nofile         65535

Edit /etc/pam.d/common-session

sudo nano /etc/pam.d/common-session

Add the following line

session required pam_limits.so

Once finished editing the required files logout of the session and log back in. Run the following command

ulimit -n

Those instructions have worked in the past for me, but i would recommend reading that article first. Just to be safe :slight_smile:

Hi Phil,

thank you for your answer! In the meantime, I’ve solved the problem. And it’s true that the problem was with number of opened files, although that’s not so obvious from the error message in log.

I’ve increased maximum number of opened files but that was not all that needed to be configured. What helped at the end was adding following line to the influx service definition file:
LimitNOFILE=65536

Thank you and best regards,
Domagoj

Awesome, glad its sorted. I only recognised the symptoms because I’ve had the exact same issue with Kapacitor in the past. I don’t think i added the LimitNOFILE part to the service definition. Slept since then