Too many file open when starting up influxdb in VM

InfluxDB 2
1

Hi everyone,
I am new to influxdb. I have trouble to run influxdb 2.1 on VM due to failed to open shard with no configuration file. I believe that means using default value.

Error message as following :
2022-03-26T06:09:08.261094Z error Failed to open shard {“log_id”: “0_TDJ5aG000”, “service”: “storage-engine”, “service”: “store”, “op_name”: “tsdb_open”, “db_shard_id”: 98, “error”: “[shard 98] open /root/.influxdbv2/engine/data/1ad36dd15b2a584d/autogen/98/index/1: too many open files”}
2022-03-26T06:09:08.266145Z error Failed to open shard {“log_id”: “0_TDJ5aG000”, “service”: “storage-engine”, “service”: “store”, “op_name”: “tsdb_open”, “db_shard_id”: 93, “error”: “[shard 93] open /root/.influxdbv2/engine/data/1ad36dd15b2a584d/autogen/93/index/0/MANIFEST: too many open files”}
2022-03-26T06:09:08.290014Z error Failed to open shard {“log_id”: “0_TDJ5aG000”, “service”: “storage-engine”, “service”: “store”, “op_name”: “tsdb_open”, “db_shard_id”: 94, “error”: “[shard 94] open /root/.influxdbv2/engine/data/1ad36dd15b2a584d/autogen/94/index/0/MANIFEST: too many open files”}
2022-03-26T06:09:08.290199Z error Failed to open shard {“log_id”: “0_TDJ5aG000”, “service”: “storage-engine”, “service”: “store”, “op_name”: “tsdb_open”, “db_shard_id”: 95, “error”: “[shard 95] open /root/.influxdbv2/engine/data/1ad36dd15b2a584d/autogen/95/index/0/MANIFEST: too many open files”}
2022-03-26T06:09:08.290328Z error Failed to open shard {“log_id”: “0_TDJ5aG000”, “service”: “storage-engine”, “service”: “store”, “op_name”: “tsdb_open”, “db_shard_id”: 96, “error”: “[shard 96] open /root/.influxdbv2/engine/data/1ad36dd15b2a584d/autogen/96/index/0/MANIFEST: too many open files”}
2022-03-26T06:09:08.341834Z error Failed to open shard {“log_id”: “0_TDJ5aG000”, “service”: “storage-engine”, “service”: “store”, “op_name”: “tsdb_open”, “db_shard_id”: 97, “error”: “[shard 97] open /root/.influxdbv2/engine/data/1ad36dd15b2a584d/autogen/97/index/0/MANIFEST: too many open files”}
2022-03-26T06:09:08.362184Z info index opened with 8 partitions {“log_id”: “0_TDJ5aG000”, “service”: “storage-engine”, “index”: “tsi”}
2022-03-26T06:09:08.372405Z error Failed to open shard {“log_id”: “0_TDJ5aG000”, “service”: “storage-engine”, “service”: “store”, “op_name”: “tsdb_open”, “db_shard_id”: 91, “error”: “[shard 91] open /root/.influxdbv2/engine/data/1ad36dd15b2a584d/autogen/91: too many open files”}
…
2022-03-26T06:09:09.319227Z fatal Error listening on port: localhost:0, “listen tcp: lookup localhost on [::1]:53: dial udp [::1]:53: socket: too many open files” {“log_id”: “0_TDJ5aG000”, “service”: “nats”, “nats_level”: “fatal”}

There is about 100 TSM files opened. Is there any way I can close some and data will be still there? How should I fix this other than increasing ulimit?

Some previous post suggested to increase the ulimit from 1024 to more. Since data is going to be stored as long as possible(right now set to infinite), will increasing ulimit delay the issue of too many file descriptor? Why are the files opened when influxdb starts up?

Also when I inspect one of the tsm files, it shows Time Range: 1970-01-01T00:00:01.643926868Z - 1970-01-01T00:00:01.643960397Z. However, when I check my data, I can find correct timestamp. What may cause this weird timestamp?

Thanks,

2

Hello @cynthiaho,

The default linux file handle limits are not good for database usages, most databases tell you to increase the file handle limits.
There was an issue reported for InfluxDB consuming all file descriptors that the eng closed recently for inactivity.

I don’t know about the timestamps.

3

Hi @Anaisdg ,

What is the min requirement for file handle?