Raspberry crash leaves data from older shards invisible

[Uncle_Jani]

Hi, I am quite new to Influxdb.

use 1.8.4 on Raspberry 2b with debian, default installation for logging sensor data of my sons science competition project. Approx. 7500 points daily. Use Grafana for data visualization.

Everything was fine until Raspberry crashed. After reboot, Influxdb does only show data from last shard, using CLI or Grafana doesnt matter.

Show shards lists all of them like this.
name: sensors
id database retention_policy shard_group start_time end_time expiry_time owners
– -------- ---------------- ----------- ---------- -------- ----------- ------
2 sensors autogen 2 2021-03-15T00:00:00Z 2021-03-22T00:00:00Z 2021-03-22T00:00:00Z
5 sensors autogen 5 2021-03-22T00:00:00Z 2021-03-29T00:00:00Z 2021-03-29T00:00:00Z
13 sensors autogen 13 2021-03-29T00:00:00Z 2021-04-05T00:00:00Z 2021-04-05T00:00:00Z
21 sensors autogen 21 2021-04-05T00:00:00Z 2021-04-12T00:00:00Z 2021-04-12T00:00:00Z
29 sensors autogen 29 2021-04-12T00:00:00Z 2021-04-19T00:00:00Z 2021-04-19T00:00:00Z
37 sensors autogen 37 2021-04-19T00:00:00Z 2021-04-26T00:00:00Z 2021-04-26T00:00:00Z
46 sensors autogen 46 2021-04-26T00:00:00Z 2021-05-03T00:00:00Z 2021-05-03T00:00:00Z
55 sensors autogen 55 2021-05-03T00:00:00Z 2021-05-10T00:00:00Z 2021-05-10T00:00:00Z

Have tried backup and restore to new db. Same picture there, newdb shows multiple shards but data can only be retrieved from the last one.

What can do the trick to make the data from all shards visible again? Or is the data completely lost?

Side note: when trying to backup one of the older shards, backup fails:

pi@raspberrypi:~ $ sudo influxd backup -database sensors -retention autogen -shard 46 /home/pi/influx-backup
2021/05/07 12:06:29 backing up metastore to /home/pi/influx-backup/meta.02
2021/05/07 12:06:29 backing up db=sensors rp=autogen shard=46 to /home/pi/influx-backup/sensors.autogen.00046.00 since 0001-01-01T00:00:00Z
2021/05/07 12:06:29 Download shard 46 failed copy backup to file: err=, n=0. Waiting 2s and retrying (0)…
2021/05/07 12:06:31 Download shard 46 failed copy backup to file: err=, n=0. Waiting 2s and retrying (1)…
2021/05/07 12:06:33 Download shard 46 failed copy backup to file: err=, n=0. Waiting 2s and retrying (2)…
2021/05/07 12:06:35 Download shard 46 failed copy backup to file: err=, n=0. Waiting 2s and retrying (3)…
2021/05/07 12:06:38 Download shard 46 failed copy backup to file: err=, n=0. Waiting 2s and retrying (4)…
2021/05/07 12:06:40 Download shard 46 failed copy backup to file: err=, n=0. Waiting 2s and retrying (5)…
2021/05/07 12:06:42 Download shard 46 failed copy backup to file: err=, n=0. Waiting 3.01s and retrying (6)…
2021/05/07 12:06:45 Download shard 46 failed copy backup to file: err=, n=0. Waiting 11.441s and retrying (7)…
2021/05/07 12:06:56 Download shard 46 failed copy backup to file: err=, n=0. Waiting 43.477s and retrying (8)…
2021/05/07 12:07:39 Download shard 46 failed copy backup to file: err=, n=0. Waiting 2m45.216s and retrying (9)…
2021/05/07 12:10:25 backup failed: copy backup to file: err=, n=0
backup: copy backup to file: err=, n=0

[Uncle_Jani]

Edit:
Backup Manifest contains only current “hot” shard. Only this TSM file in backup. But show shards shows all historic shards even after restore from this backup.

@Anaisdg u are one of the more active people from influx. If you dont mind, could you take a look at my issue? Thanks a lot.
Regards
Jan

[Anaisdg]

Hello @Uncle_Jani,
I’m not sure. What version of InfluxDB are you using and how are you writing data to InfluxDB?
Are you trying to perform a backup?

[MarcV]

Hi , @Uncle_Jani ,

that sounds like a very interesting project , I am sure it will give a lot of satisfaction doing this together with your son .
Maybe the competition is already finished , I hope he won .

At the time of your writing all shards had already expired except for the last shard. ( expiry_time )
That explains why you could only see data from the last shard .
The expiration of shards depends on your retention policy .
What is the output of show shards now ?
And the output of : show retention policies ?

best regards ,
Marc

[BienVu1309]

Hi, I faced the same issue that lost data after the reboot of Rasp (TinyCore OS). My system has local influxdb and it syncs to our cloud database ( MongoDB) by our program. Mine attach picture shows the green is MongoDB from the cloud and the red is from local Pi. The configuration also attaches here and nor retention policy. Hope to see your reply
InfluxDB version 1.8.4 and commit id : bc8ec4384

Configuration:

[meta]
dir = “/tmp/influxdb/meta”

[data]
dir = “/mnt/mmcblk0p2/ogcunit/data/tsm”

wal-dir = “/mnt/mmcblk0p2/ogcunit/data/wal”

series-id-set-cache-size = 100

[coordinator]

[retention]

[shard-precreation]

[monitor]
store-enabled = false

[http]

flux-enabled = false

[logging]

[subscriber]

[[graphite]]

[[collectd]]

[[opentsdb]]

[[udp]]

[continuous_queries]

[tls]

Regards,
BienVu