Influx gone, back in onboarding steps

1

Somehow my complete Influx installation is gone. When I visit the GUI, I am shown the onboarding steps again. It feel like my data and setup is gone.

I was on holiday this week and I know there was a power outage. I have no idea if this is related to that or not.

Is there a guide that shows me what steps I should make to investigate what happened? I only checked the folder on the server and at least I can see my database in the data folder. So probably not everything is lost.

Would you be able to give me any pointers how to get this up and running again?

2

I managed to get the logs from journalctl and I can see when the power was gone, and how influx was rebooting and the power cam back. Obviously the shutdown was sudden and not “managed”.
But I can’t really tell from the log what went wrong?

-- Journal begins at Fri 2021-11-26 19:15:23 CET, ends at Sun 2023-03-12 20:34:11 CET. --
Mar 07 08:00:00 NodeRedServer influxd-systemd-start.sh[615]: ts=2023-03-07T07:00:00.106896Z lvl=info msg="Error exhausting result iterator" log_id=0bmiVunG000 service=task-executor error="runtime error @19:9-19:29: fieldsAsCols: specified row key column does not exist in table: _time" name=experimental-to13
Mar 07 08:04:34 NodeRedServer influxd-systemd-start.sh[615]: ts=2023-03-07T07:04:34.512373Z lvl=info msg="TSI log compaction (start)" log_id=0bmiVunG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=start
Mar 07 08:04:34 NodeRedServer influxd-systemd-start.sh[615]: ts=2023-03-07T07:04:34.514034Z lvl=info msg="Log file compacted" log_id=0bmiVunG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 elapsed=1ms bytes=596 kb_per_sec=345
Mar 07 08:04:34 NodeRedServer influxd-systemd-start.sh[615]: ts=2023-03-07T07:04:34.514089Z lvl=info msg="TSI log compaction (end)" log_id=0bmiVunG000 service=storage-engine index=tsi tsi1_partition=5 op_name=tsi1_compact_log_file tsi1_log_file_id=1 op_event=end op_elapsed=1.732ms
-- Boot 0d52a068fbb746f68cdb6a1b286bb823 --
Mar 07 11:59:53 NodeRedServer systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[661]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.265292Z lvl=info msg="Welcome to InfluxDB" log_id=0gQwUm5W000 version=v2.5.1 commit=5b6fdbf05d build_date=2022-11-02T18:06:28Z log_level=info
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.279098Z lvl=info msg="Resources opened" log_id=0gQwUm5W000 service=bolt path=/var/lib/influxdb/influxd.bolt
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.279197Z lvl=info msg="Resources opened" log_id=0gQwUm5W000 service=sqlite path=/var/lib/influxdb/influxd.sqlite
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.282742Z lvl=info msg="Bringing up metadata migrations" log_id=0gQwUm5W000 service="KV migrations" migration_count=20
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[813]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[832]: Command "print-config" is deprecated, use the influx-cli command server-config to display the configuration values from the running server
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[645]: InfluxDB API at http://localhost:8086/ready unavailable after 1 attempts...
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.583274Z lvl=info msg="Bringing up metadata migrations" log_id=0gQwUm5W000 service="SQL migrations" migration_count=8
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742278Z lvl=info msg="Using data dir" log_id=0gQwUm5W000 service=storage-engine service=store path=/var/lib/influxdb/engine/data
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742405Z lvl=info msg="Compaction settings" log_id=0gQwUm5W000 service=storage-engine service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742420Z lvl=info msg="Open store (start)" log_id=0gQwUm5W000 service=storage-engine service=store op_name=tsdb_open op_event=start
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742472Z lvl=info msg="Open store (end)" log_id=0gQwUm5W000 service=storage-engine service=store op_name=tsdb_open op_event=end op_elapsed=0.053ms
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742496Z lvl=info msg="Starting retention policy enforcement service" log_id=0gQwUm5W000 service=retention check_interval=30m
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.742505Z lvl=info msg="Starting precreation service" log_id=0gQwUm5W000 service=shard-precreation check_interval=10m advance_period=30m
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.748505Z lvl=info msg="Starting query controller" log_id=0gQwUm5W000 service=storage-reads concurrency_quota=1024 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=1024
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.754027Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0gQwUm5W000 max_select_point=0 max_select_series=0 max_select_buckets=0
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.788461Z lvl=info msg=Starting log_id=0gQwUm5W000 service=telemetry interval=8h
Mar 07 11:59:54 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T10:59:54.789434Z lvl=info msg=Listening log_id=0gQwUm5W000 service=tcp-listener transport=http addr=:8086 port=8086
Mar 07 11:59:55 NodeRedServer influxd-systemd-start.sh[645]: InfluxDB started
Mar 07 11:59:55 NodeRedServer systemd[1]: Started InfluxDB is an open-source, distributed, time series database.
Mar 07 12:00:04 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T11:00:04.361863Z lvl=info msg=Unauthorized log_id=0gQwUm5W000 error="authorization not found"
Mar 07 12:00:14 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T11:00:14.358014Z lvl=info msg=Unauthorized log_id=0gQwUm5W000 error="authorization not found"
Mar 07 12:00:24 NodeRedServer influxd-systemd-start.sh[659]: ts=2023-03-07T11:00:24.358155Z lvl=info msg=Unauthorized log_id=0gQwUm5W000 error="authorization not found"
3

It looks like when your server is rebooted and InfluxDB started, it didn’t get the bolt file which stores the credentials bucket name, etc.

Please make sure your data, wal, and Influxd. bolt files are mounted properly.

also, please mention your InfluxDB version and your mount points for Influx configuration.
I can help better as I already face this issue multiple times.

4

I am using the following:
Influx CLI 2.5.0 (git: 3285a03) build_date: 2022-11-01T16:32:06Z
X-Influxdb-Build: OSS
X-Influxdb-Version: v2.5.1
Date: Thu, 16 Mar 2023 13:55:51 GMT

And this is the content of my /var/lib/influxdb

drwxr-xr-x 5 influxdb influxdb   4096 Dec  7  2021 data
drwxr-xr-x 4 influxdb influxdb   4096 Mar  7 11:59 engine
-rw------- 1 influxdb influxdb  65536 Mar 12 19:51 influxd.bolt
-rw-r--r-- 1 influxdb influxdb      6 Mar 12 19:51 influxd.pid
-rw-r--r-- 1 influxdb influxdb 110592 Mar  7 11:59 influxd.sqlite
drwxr-xr-x 2 influxdb influxdb   4096 Dec  8  2021 meta
drwx------ 5 influxdb influxdb   4096 Dec  7  2021 wal

Let me know if you need more info.

5

I suppose you have pasted content inside of your InfluxDB container.
Please paste the local storage point and docker-compose if you are using it.

6

No, I am not using docker. I installed it from command line, I followed this page for installation: https://citizix.com/how-to-install-and-configure-influxdb2-in-debian-11/

8

What is a local storage point?

I hope you can help me with this… I year worth of data I would rather not see to go to waste…

9

Did you sign up for after the system reboot?
If not then your old token and data wal will be safe then there is a chance you can recover your data.

10

Sign up where? Sign up for what? I don’t remember doing anything like that.

11

i experienced the same after a power outage. Is there a way the restore the data into a new installation?
Influx CLI doesn’t work anymore because of athorization error and i am afraid to create a new one at the Onboarding page.
Is there a way to write the data from disk to line protocol and inject it to a new installation?
Best regards
Sebastian

12

I have given up after waiting for 2 weeks. Nobody was able to give me any advice. I backed up the data folder, but to be honest I don’t think I will be able to recover the data at all.

I also checked the backup options, but whatever I do I get a 401 Unauthorized access error message.

I am planning to create a node-red flow which runs on a monthly basis and exports my measurements into line protocol and stores it as a compressed text files. Those will be easy to restore to a new database. Of course this will require you to re-create the database, token for node-red, buckets and those are actually pretty quick. Maybe you can take a note of the steps in a text file or something. And you can easily inject the line protocol extracts in the GUI. And at least I am not relying on any “unknown” of influx stuff.

Also please note that I just found on some influx installation documentation that you should take a not of the root token that got generated in the installation process. I did not do that, and it looks like once you save the token, it is “gone”. So this could be another reason I cannot use the backup and restore function.

13

Also, I am not sure if this is related. But looking back my youtube videos, I had Influx 2.1.1, and now after the onboarding steps I noticed that the GUI has changes and when I checked the version I now have 2.5.1. I can’t tell if the power outage and the version update happened at the same time, and the loss of data has anything to do with the version upgrade. But who knows…

14

Hi there, I just watched your youtube video about this problem. I also followed a thread about missing data after a power outage. As a new user of Influxdb I was a bit shocked.
You mention you have a backup strategy using nodered to export the data via the line-protocol. Is that nodered flow available somewhere?

15

Thanks, I started working on it, but it is not complete.
I am also having issues with the batch data export. I give it a large text file with line protocol lines and it appears to process it, but gives a generic error message which does not indicate what the issue is and in which line the issue occurred. It looks like another half baked solution, which provides inconsistent user experience. When I have a time I will pick this up again and probably report it to Influx to get it fixed.

16

I had the same problem today after an unexpected reboot on a Mac mini.
The reason for the unexpected onboarding landing page was simple:
I used a different account this time. Installed with my account and restarted after a reboot as user root.
Since there are no configurations for influxdb for user root, it assumes this is a complete new environment.
Once I tried it with my user account, everything was fine again.
My upload scripts for sensor data no longer gets those responses:
HTTP response body: {“code”:“unauthorized”,“message”:“unauthorized access”}

17

I had a similar issue to nygma2004 and was able to get the original login and buckets back.
I found the influxdb directory - /var/lib/influxdb/
In the directory I had; data, engine, influxd.bolt, .influxdbv2, influxd.sqlite, meta and wal.
Inside .influxdbv2 it had; engine, influxd.bolt and influx.sqlite.
I stopped the influxdb service - sudo service influxdb stop
I then copied influxd.bolt and influx.sqlite from .influxdbv2 into /var/lib/influxdb/, replacing the files in there.
I started the influxdb service - sudo service influxdb start
I was then able to log into my influxdb via the GUI and access my orignal buckets.