Somehow my complete Influx installation is gone. When I visit the GUI, I am shown the onboarding steps again. It feel like my data and setup is gone.
I was on holiday this week and I know there was a power outage. I have no idea if this is related to that or not.
Is there a guide that shows me what steps I should make to investigate what happened? I only checked the folder on the server and at least I can see my database in the data folder. So probably not everything is lost.
Would you be able to give me any pointers how to get this up and running again?
I managed to get the logs from journalctl and I can see when the power was gone, and how influx was rebooting and the power cam back. Obviously the shutdown was sudden and not “managed”.
But I can’t really tell from the log what went wrong?
-- Journal begins at Fri 2021-11-26 19:15:23 CET, ends at Sun 2023-03-12 20:34:11 CET. --
Mar 07 08:00:00 NodeRedServer influxd-systemd-start.sh[615]: ts=2023-03-07T07:00:00.106896Z lvl=info msg="Error exhausting result iterator" log_id=0bmiVunG000 service=task-executor error="runtime error @19:9-19:29: fieldsAsCols: specified row key column does not exist in table: _time" name=experimental-to13
Mar 07 08:04:34 NodeRedServer influxd-systemd-start.sh[615]: ts=2023-03-07T07:04:34.512373Z lvl=info msg="TSI log compaction (start)" log_id=0bmiVunG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=start
Mar 07 08:04:34 NodeRedServer influxd-systemd-start.sh[615]: ts=2023-03-07T07:04:34.514034Z lvl=info msg="Log file compacted" log_id=0bmiVunG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 elapsed=1ms bytes=596 kb_per_sec=345
Mar 07 08:04:34 NodeRedServer influxd-systemd-start.sh[615]: ts=2023-03-07T07:04:34.514089Z lvl=info msg="TSI log compaction (end)" log_id=0bmiVunG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=1.732ms
-- Boot 0d52a068fbb746f68cdb6a1b286bb823 --
Mar 07 11:59:53 NodeRedServer systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[661]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.265292Z lvl=info msg="Welcome to InfluxDB" log_id=0gQwUm5W000 version=v2.5.1 commit=5b6fdbf05d build_date=2022-11-02T18:06:28Z log_level=info
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.279098Z lvl=info msg="Resources opened" log_id=0gQwUm5W000 service=bolt path=/var/lib/influxdb/influxd.bolt
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.279197Z lvl=info msg="Resources opened" log_id=0gQwUm5W000 service=sqlite path=/var/lib/influxdb/influxd.sqlite
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.282742Z lvl=info msg="Bringing up metadata migrations" log_id=0gQwUm5W000 service="KV migrations" migration_count=20
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[813]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[832]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[645]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.583274Z lvl=info msg="Bringing up metadata migrations" log_id=0gQwUm5W000 service="SQL migrations" migration_count=8
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742278Z lvl=info msg="Using data dir" log_id=0gQwUm5W000 service=storage-engine service=store path=/var/lib/influxdb/engine/data
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742405Z lvl=info msg="Compaction settings" log_id=0gQwUm5W000 service=storage-engine service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742420Z lvl=info msg="Open store (start)" log_id=0gQwUm5W000 service=storage-engine service=store op_name=tsdb_open op_event=start
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742472Z lvl=info msg="Open store (end)" log_id=0gQwUm5W000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=0.053ms
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742496Z lvl=info msg="Starting retention policy enforcement service" log_id=0gQwUm5W000 service=retention check_interval=30m
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742505Z lvl=info msg="Starting precreation service" log_id=0gQwUm5W000 service=shard-precreation check_interval=10m advance_period=30m
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.748505Z lvl=info msg="Starting query controller" log_id=0gQwUm5W000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.754027Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0gQwUm5W000 max_select_point=0 max_select_series=0 max_select_buckets=0
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.788461Z lvl=info msg=Starting log_id=0gQwUm5W000 service=telemetry interval=8h
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.789434Z lvl=info msg=Listening log_id=0gQwUm5W000 service=tcp-listener transport=http addr=:8086 port=8086
Mar 07 11:59:55 NodeRedServer influxd-systemd-start.sh[645]: InfluxDB started
Mar 07 11:59:55 NodeRedServer systemd[1]: Started InfluxDB is an open-source, distributed, time series database.
Mar 07 12:00:04 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T11:00:04.361863Z lvl=info msg=Unauthorized log_id=0gQwUm5W000 error="authorization not found"
Mar 07 12:00:14 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T11:00:14.358014Z lvl=info msg=Unauthorized log_id=0gQwUm5W000 error="authorization not found"
Mar 07 12:00:24 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T11:00:24.358155Z lvl=info msg=Unauthorized log_id=0gQwUm5W000 error="authorization not found"
It looks like when your server is rebooted and InfluxDB started, it didn’t get the bolt file which stores the credentials bucket name, etc.
Please make sure your data, wal, and Influxd. bolt files are mounted properly.
also, please mention your InfluxDB version and your mount points for Influx configuration.
I can help better as I already face this issue multiple times.
I am using the following:
Influx CLI 2.5.0 (git: 3285a03) build_date: 2022-11-01T16:32:06Z
X-Influxdb-Build: OSS
X-Influxdb-Version: v2.5.1
Date: Thu, 16 Mar 2023 13:55:51 GMT
And this is the content of my /var/lib/influxdb
drwxr-xr-x 5 influxdb influxdb 4096 Dec 7 2021 data
drwxr-xr-x 4 influxdb influxdb 4096 Mar 7 11:59 engine
-rw------- 1 influxdb influxdb 65536 Mar 12 19:51 influxd.bolt
-rw-r--r-- 1 influxdb influxdb 6 Mar 12 19:51 influxd.pid
-rw-r--r-- 1 influxdb influxdb 110592 Mar 7 11:59 influxd.sqlite
drwxr-xr-x 2 influxdb influxdb 4096 Dec 8 2021 meta
drwx------ 5 influxdb influxdb 4096 Dec 7 2021 wal
Let me know if you need more info.
I suppose you have pasted content inside of your InfluxDB container.
Please paste the local storage point and docker-compose if you are using it.
No, I am not using docker. I installed it from command line, I followed this page for installation: https://citizix.com/how-to-install-and-configure-influxdb2-in-debian-11/
What is a local storage point?
I hope you can help me with this… I year worth of data I would rather not see to go to waste…
Did you sign up for after the system reboot?
If not then your old token and data wal will be safe then there is a chance you can recover your data.
Sign up where? Sign up for what? I don’t remember doing anything like that.
