My influxdb is not able to start anymore :-/

InfluxDB 1
1

Hey guys,

I have the problem that my influxdb service is not able to come up anymore. I really don’t know what to do. I had some permission issues in the last months when I updated my raspberrypi 4 by ‘apt update && apt upgrade’ as root, so I had to fixed this and everything worked fine again. But now it’s not a permission error I think, because I checked all relevant directories and all are fine for the user influxdb.

When I start the service manually the following comes up:

pi@raspberrypi:~ $ sudo systemctl start influxdb
Job for influxdb.service failed because the control process exited with error code.
See "systemctl status influxdb.service" and "journalctl -xe" for details.

So the next relevant output is this:

pi@raspberrypi:~ $ sudo systemctl start influxdb
Job for influxdb.service failed because the control process exited with error code.
See "systemctl status influxdb.service" and "journalctl -xe" for details.
pi@raspberrypi:~ $ sudo systemctl status influxdb
● influxdb.service - InfluxDB is an open-source, distributed, time series database
   Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
   Active: activating (start) since Thu 2021-09-16 12:43:06 CEST; 7s ago
     Docs: https://docs.influxdata.com/influxdb/
Cntrl PID: 22148 (influxd-systemd)
    Tasks: 15 (limit: 4915)
   CGroup: /system.slice/influxdb.service
           ├─22148 /bin/bash -e /usr/lib/influxdb/scripts/influxd-systemd-start.sh
           ├─22149 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
           └─22209 sleep 1

Sep 16 12:43:07 raspberrypi influxd-systemd-start.sh[22148]: ts=2021-09-16T10:43:07.425012Z lvl=info msg="Reading file" log_id=0WcZ5L_G000 engine=tsm1 service
Sep 16 12:43:07 raspberrypi influxd-systemd-start.sh[22148]: ts=2021-09-16T10:43:07.441425Z lvl=info msg="Opened shard" log_id=0WcZ5L_G000 service=store trace
Sep 16 12:43:07 raspberrypi influxd-systemd-start.sh[22148]: ts=2021-09-16T10:43:07.442808Z lvl=info msg="Opened file" log_id=0WcZ5L_G000 engine=tsm1 service=
Sep 16 12:43:07 raspberrypi influxd-systemd-start.sh[22148]: ts=2021-09-16T10:43:07.445466Z lvl=info msg="Opened file" log_id=0WcZ5L_G000 engine=tsm1 service=
Sep 16 12:43:07 raspberrypi influxd-systemd-start.sh[22148]: ts=2021-09-16T10:43:07.512055Z lvl=info msg="Opened shard" log_id=0WcZ5L_G000 service=store trace
Sep 16 12:43:07 raspberrypi influxd-systemd-start.sh[22148]: ts=2021-09-16T10:43:07.514635Z lvl=info msg="Opened shard" log_id=0WcZ5L_G000 service=store trace
Sep 16 12:43:07 raspberrypi influxd-systemd-start.sh[22148]: ts=2021-09-16T10:43:07.514938Z lvl=info msg="Opened file" log_id=0WcZ5L_G000 engine=tsm1 service=
Sep 16 12:43:07 raspberrypi influxd-systemd-start.sh[22148]: ts=2021-09-16T10:43:07.564893Z lvl=info msg="Opened shard" log_id=0WcZ5L_G000 service=store trace
Sep 16 12:43:08 raspberrypi influxd-systemd-start.sh[22148]: ts=2021-09-16T10:43:08.409828Z lvl=info msg="Opened shard" log_id=0WcZ5L_G000 service=store trace
Sep 16 12:43:11 raspberrypi influxd-systemd-start.sh[22148]: ts=2021-09-16T10:43:11.600492Z lvl=info msg="Reading file" log_id=0WcZ5L_G000 engine=tsm1 service
lines 1-21/21 (END)

The status will remain in the “activating” status until you kill the process manually.
I really don’t know what’s going on, because the influxdb worked fine for a long time and I didn’t change any relevant system settings on the raspberry.

Maybe this output could also be helpful:

Sep 16 07:40:44 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:44.761768Z lvl=info msg="Opened shard" log_id=0WcHm~S0000 service=store trace
Sep 16 07:40:44 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:44.766095Z lvl=info msg="Opened file" log_id=0WcHm~S0000 engine=tsm1 service=
Sep 16 07:40:44 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:44.773748Z lvl=info msg="Opened shard" log_id=0WcHm~S0000 service=store trace
Sep 16 07:40:44 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:44.776042Z lvl=info msg="Opened file" log_id=0WcHm~S0000 engine=tsm1 service=
Sep 16 07:40:44 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:44.789121Z lvl=info msg="Opened shard" log_id=0WcHm~S0000 service=store trace
Sep 16 07:40:44 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:44.791702Z lvl=info msg="Opened file" log_id=0WcHm~S0000 engine=tsm1 service=
Sep 16 07:40:44 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:44.839768Z lvl=info msg="Opened shard" log_id=0WcHm~S0000 service=store trace
Sep 16 07:40:44 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:44.842041Z lvl=info msg="Opened file" log_id=0WcHm~S0000 engine=tsm1 service=
Sep 16 07:40:44 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:44.862885Z lvl=info msg="Opened shard" log_id=0WcHm~S0000 service=store trace
Sep 16 07:40:44 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:44.871454Z lvl=info msg="Opened shard" log_id=0WcHm~S0000 service=store trace
Sep 16 07:40:45 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:45.721399Z lvl=info msg="Opened shard" log_id=0WcHm~S0000 service=store trace
Sep 16 07:40:48 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:48.547361Z lvl=info msg="Reading file" log_id=0WcHm~S0000 engine=tsm1 service
Sep 16 07:40:52 raspberrypi influxd-systemd-start.sh[20170]: ts=2021-09-16T05:40:52.578192Z lvl=info msg="Reading file" log_id=0WcHm~S0000 engine=tsm1 service
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20170]: Failed to reach influxdb http endpoint at http://localhost:8086/health
Sep 16 07:40:54 raspberrypi systemd[1]: influxdb.service: Control process exited, code=exited, status=1/FAILURE
Sep 16 07:40:54 raspberrypi systemd[1]: influxdb.service: Failed with result 'exit-code'.
Sep 16 07:40:54 raspberrypi systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.
Sep 16 07:40:54 raspberrypi systemd[1]: influxdb.service: Service RestartSec=100ms expired, scheduling restart.
Sep 16 07:40:54 raspberrypi systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 7696.
Sep 16 07:40:54 raspberrypi systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Sep 16 07:40:54 raspberrypi systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: Merging with configuration at: /etc/influxdb/influxdb.conf
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.766391Z lvl=info msg="InfluxDB starting" log_id=0WcHneRW000 version=1.8.9 
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.766447Z lvl=info msg="Go runtime" log_id=0WcHneRW000 version=go1.13.8 maxp
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: Merging with configuration at: /etc/influxdb/influxdb.conf
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.868397Z lvl=info msg="Using data dir" log_id=0WcHneRW000 service=store pat
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.868478Z lvl=info msg="Compaction settings" log_id=0WcHneRW000 service=stor
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.868529Z lvl=info msg="Open store (start)" log_id=0WcHneRW000 service=store
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.891369Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.892102Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.894391Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.896562Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.987943Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.990204Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.996670Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:54 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:54.998921Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.002083Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.002798Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.005428Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.006745Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.007711Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.007977Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.005464Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.009120Z lvl=info msg="Reading file" log_id=0WcHneRW000 engine=tsm1 service
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.039325Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.040960Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.058344Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.059732Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.070618Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.072421Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.076704Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.078335Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.088317Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.089626Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.099478Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.101179Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.104364Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.106930Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.126157Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.127827Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.132678Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.134310Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.138511Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.139997Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.155818Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.157503Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.161337Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.162980Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.171298Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.172730Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.195469Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.196986Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.198445Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.199579Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.201840Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.203283Z lvl=info msg="Opened file" log_id=0WcHneRW000 engine=tsm1 service=
Sep 16 07:40:55 raspberrypi influxd-systemd-start.sh[20240]: ts=2021-09-16T05:40:55.221512Z lvl=info msg="Opened shard" log_id=0WcHneRW000 service=store trace

I cannot find any help for this kind of error-code and I would be very happy to get a little bit assist of you guys :slight_smile:

Thanks a lot!

Regards
Berzi

2

When influxdb starts up it has to read in all its data files, so if you have a lot of historical data points this can take a long period of time. This startup process can also use a lot of memory relative to the steady state running memory usage.
Do you have a time out to service start which is killing the process before the database has opened ?
Is the process running out of memory and being killed ?

3

Thank you Neil_Benton for your reply.

The data is less than 100MB, the memory usage of the raspberry pi is less than 45% and the cpu usage is very low as well. Waiting for the process to come up has no hope, because I did it several times for more than a few days :-/ Having any timeout would be new to me, because I didn’t set up anything like that. Also I would say that the database is very small so this shouldn’t be the problem, should it?

4

Since your using systemd, I’m wondering if you are facing a similar issue as reported here: https://github.com/influxdata/influxdb/issues/21967

This may also be useful: Debian package release v1.8.7 fails to start with systemd · Issue #21933 · influxdata/influxdb · GitHub

Is any of that discussion helpful?

1 Like
5

Hey wbaker, wow thank you!
This one did it: https://github.com/influxdata/influxdb/issues/21967#issuecomment-889695265

Although they talk about vers. 1.8.7 and I’m already running 1.8.9 the issue is still not fixed with these two version steps.

Thanks a lot!