Data loss after db server reboot

vimal525 · June 4, 2019, 2:24am

Data is lost from measurements requestsRaw .Now it shows only two rows.Recently we have
server reboot.
How do I get my data.I don’t have backup of database jmeter3.

[root@hgbupap07vm03 /]# ls -al /etc/influxdb/influxdb.conf
-rw-r–r–. 1 root root 17269 Sep 17 2018 /etc/influxdb/influxdb.conf

[data]
dir = “/var/lib/influxdb/data”
index-version = “tsi1”
wal-dir = “/var/lib/influxdb/wal”
wal-fsync-delay = “0s”
query-log-enabled = true
cache-max-memory-size = 1073741824
cache-snapshot-memory-size = 26214400
cache-snapshot-write-cold-duration = “10m0s”
compact-full-write-cold-duration = “4h0m0s”
max-series-per-database = 1000000
max-values-per-tag = 100000
max-concurrent-compactions = 0
max-index-log-file-size = 1048576
trace-logging-enabled = false

[root@hgbupap07vm03 /]# influx -version
InfluxDB shell version: 1.5.2

use jmeter3
Using database jmeter3
show measurements
name: measurements
name

requestsRaw
testStartEnd
virtualUsers

select * from requestsRaw;
name: requestsRaw
time errorCount nodeName requestName responseTime testName threadName

1559297851134052732 0 WS OC_WS_CP_FetchHotelContext-WS 915 UIScripting S01_WS_OC_CreateProfile-Thread 1-1
1559297856044300918 0 WS OC_WS_CP_CreateProfileRQ-WS 4908 UIScripting S01_WS_OC_CreateProfile-Thread 1-1

name: jmeter3
id database retention_policy shard_group start_time end_time expiry_time owners

854 jmeter3 autogen 854 2019-02-11T00:00:00Z 2019-02-18T00:00:00Z 2019-02-18T00:00:00Z
880 jmeter3 autogen 880 2019-02-18T00:00:00Z 2019-02-25T00:00:00Z 2019-02-25T00:00:00Z
908 jmeter3 autogen 908 2019-02-25T00:00:00Z 2019-03-04T00:00:00Z 2019-03-04T00:00:00Z
936 jmeter3 autogen 936 2019-03-04T00:00:00Z 2019-03-11T00:00:00Z 2019-03-11T00:00:00Z
964 jmeter3 autogen 964 2019-03-11T00:00:00Z 2019-03-18T00:00:00Z 2019-03-18T00:00:00Z
992 jmeter3 autogen 992 2019-03-18T00:00:00Z 2019-03-25T00:00:00Z 2019-03-25T00:00:00Z
1020 jmeter3 autogen 1020 2019-03-25T00:00:00Z 2019-04-01T00:00:00Z 2019-04-01T00:00:00Z
1049 jmeter3 autogen 1049 2019-04-01T00:00:00Z 2019-04-08T00:00:00Z 2019-04-08T00:00:00Z
1078 jmeter3 autogen 1078 2019-04-08T00:00:00Z 2019-04-15T00:00:00Z 2019-04-15T00:00:00Z
1107 jmeter3 autogen 1107 2019-04-15T00:00:00Z 2019-04-22T00:00:00Z 2019-04-22T00:00:00Z
1136 jmeter3 autogen 1136 2019-04-22T00:00:00Z 2019-04-29T00:00:00Z 2019-04-29T00:00:00Z
1165 jmeter3 autogen 1165 2019-04-29T00:00:00Z 2019-05-06T00:00:00Z 2019-05-06T00:00:00Z
1194 jmeter3 autogen 1194 2019-05-06T00:00:00Z 2019-05-13T00:00:00Z 2019-05-13T00:00:00Z
1223 jmeter3 autogen 1223 2019-05-13T00:00:00Z 2019-05-20T00:00:00Z 2019-05-20T00:00:00Z
1252 jmeter3 autogen 1252 2019-05-20T00:00:00Z 2019-05-27T00:00:00Z 2019-05-27T00:00:00Z
1281 jmeter3 autogen 1281 2019-05-27T00:00:00Z 2019-06-03T00:00:00Z 2019-06-03T00:00:00Z
1311 jmeter3 autogen 1311 2019-06-03T00:00:00Z 2019-06-10T00:00:00Z 2019-06-10T00:00:00Z

vimal525 · June 4, 2019, 2:27am

one more information ,I need to add,Both these values are commented in my influxdb.conf file.
So does it impact or will it take default value.

cache-snapshot-memory-size = “25m”

cache-snapshot-write-cold-duration = “10m”

vimal525 · June 4, 2019, 2:36am

[root@hgbupap07vm03 jmeter3]# cd autogen
[root@hgbupap07vm03 autogen]# du -sch *
14M 1020
11M 1049
17M 1078
245M 1107
5.1M 1136
98M 1165
25M 1194
193M 1223
133M 1252
31M 1281
8.3M 854
25M 936
291M 964
307M 992
1.4G total
[root@hgbupap07vm03 autogen]# ls -lrt
total 0
drwxr-xr-x. 3 influxdb influxdb 65 Feb 14 20:41 854
drwxr-xr-x. 3 influxdb influxdb 65 Mar 8 20:34 936
drwxr-xr-x. 3 influxdb influxdb 65 Mar 16 08:27 964
drwxr-xr-x. 3 influxdb influxdb 65 Mar 21 02:53 992
drwxr-xr-x. 3 influxdb influxdb 65 Mar 31 02:18 1020
drwxr-xr-x. 3 influxdb influxdb 65 Apr 5 18:16 1049
drwxr-xr-x. 3 influxdb influxdb 65 Apr 12 21:58 1078
drwxr-xr-x. 3 influxdb influxdb 65 Apr 19 16:38 1107
drwxr-xr-x. 3 influxdb influxdb 65 Apr 26 18:12 1136
drwxr-xr-x. 3 influxdb influxdb 65 May 5 17:07 1165
drwxr-xr-x. 3 influxdb influxdb 65 May 11 06:03 1194
drwxr-xr-x. 3 influxdb influxdb 65 May 19 22:15 1223
drwxr-xr-x. 3 influxdb influxdb 65 May 24 22:31 1252
drwxr-xr-x. 3 influxdb influxdb 65 May 31 19:47 1281
[root@hgbupap07vm03 autogen]# du -sch 1281
31M 1281
31M total
[root@hgbupap07vm03 autogen]# cd 1281
[root@hgbupap07vm03 1281]# ls -lrt
total 31364
drwxr-xr-x. 10 influxdb influxdb 70 May 28 11:45 index
-rw-r–r--. 1 influxdb influxdb 251 May 28 11:45 fields.idx
-rw-r–r--. 1 influxdb influxdb 32109797 May 31 19:47 000000007-000000002.tsm
[root@hgbupap07vm03 1281]# pwd
/var/lib/influxdb/data/jmeter3/autogen/1281
[root@hgbupap07vm03 1281]# du -sch *
31M 000000007-000000002.tsm
4.0K fields.idx
32K index
31M total

MarcV · June 4, 2019, 7:55am

Hi @vimal525 welcome !
If they are commented default values are used. To change the values the lines must be uncommented,
Best regards

MarcV · June 4, 2019, 7:58am

Hi @vimal525,

Can you see which configuration file is used ? ( with a ps | grep - i influxd or systemctl status influxd)

vimal525 · June 5, 2019, 4:32am

Hi Marc,
Thanks for your reply.Please find below output from both commands.
[root@hgbupap07vm03 ~]# ps -ef|grep -i influxd
influxdb 948 1 6 May30 ? 09:07:39 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
root 29830 29793 0 10:00 pts/0 00:00:00 grep --color=auto -i influxd
[root@hgbupap07vm03 ~]# systemctl status influxd
● influxdb.service - InfluxDB is an open-source, distributed, time series database
Loaded: loaded (/usr/lib/systemd/system/influxdb.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2019-05-30 14:02:42 IST; 5 days ago
Docs: InfluxDB OSS 2.6 Documentation
Main PID: 948 (influxd)
CGroup: /system.slice/influxdb.service
└─948 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

Jun 05 09:29:55 hgbupap07vm03 influxd[948]: ts=2019-06-05T03:59:55.103549Z lvl=info msg=“Finished compacting files” log_id=0FiylWVG000 engine=t…65.788ms
Jun 05 09:29:55 hgbupap07vm03 influxd[948]: ts=2019-06-05T03:59:55.103565Z lvl=info msg=“TSM compaction (end)” log_id=0FiylWVG000 engine=tsm1 t…65.789ms
Jun 05 09:32:30 hgbupap07vm03 influxd[948]: ts=2019-06-05T04:02:30.499268Z lvl=info msg=“Cache snapshot (start)” log_id=0FiylWVG000 engine=tsm1…nt=start
Jun 05 09:32:30 hgbupap07vm03 influxd[948]: ts=2019-06-05T04:02:30.605058Z lvl=info msg=“Snapshot for path written” log_id=0FiylWVG000 engine=t…05.816ms
Jun 05 09:32:30 hgbupap07vm03 influxd[948]: ts=2019-06-05T04:02:30.605089Z lvl=info msg=“Cache snapshot (end)” log_id=0FiylWVG000 engine=tsm1 t…05.838ms
Jun 05 09:34:11 hgbupap07vm03 influxd[948]: ts=2019-06-05T04:04:11.958384Z lvl=info msg=“Retention policy deletion check (start)” log_id=0FiylW…nt=start
Jun 05 09:34:11 hgbupap07vm03 influxd[948]: ts=2019-06-05T04:04:11.958944Z lvl=info msg=“Retention policy deletion check (end)” log_id=0FiylWVG…=0.576ms
Jun 05 09:49:51 hgbupap07vm03 influxd[948]: ts=2019-06-05T04:19:51.499287Z lvl=info msg=“Cache snapshot (start)” log_id=0FiylWVG000 engine=tsm1…nt=start
Jun 05 09:49:51 hgbupap07vm03 influxd[948]: ts=2019-06-05T04:19:51.605564Z lvl=info msg=“Snapshot for path written” log_id=0FiylWVG000 engine=t…06.310ms
Jun 05 09:49:51 hgbupap07vm03 influxd[948]: ts=2019-06-05T04:19:51.605595Z lvl=info msg=“Cache snapshot (end)” log_id=0FiylWVG000 engine=tsm1 t…06.331ms
Hint: Some lines were ellipsized, use -l to show in full.

MarcV · June 5, 2019, 5:34am

Hi @vimal525,

Was the data lost right after the reboot ?
What is the retention policy ?
Was the retention policy changed ?
Best regards

vimal525 · June 5, 2019, 2:32pm

Please find below inline reply
Was the data lost right after the reboot ?—YES
What is the retention policy ?—default
Was the retention policy changed ?—Nope

show retention policies
name duration shardGroupDuration replicaN default

autogen 0s 168h0m0s 1 true

singhankur · June 6, 2019, 2:41am

I faced the same issue in this influx version.
I have changed the index version to index-version = “tsi1” and then restarted influx.
So whenever influx got automatically crashed because of long-running query it was unable to restore the data.
Update the influx version to 1.7 and migrate the old data to the new one then this problem got solved.

vimal525 · June 6, 2019, 3:06am

Thanks Ankur for your reply.I have two questions:

If I will change index-version = “inmem” with influx restart then my data will come?
Do you have steps of upgrading 1.5.2 to 1.7 and migrating the old data(If migrating old data involve some specific steps) ?
I have found below url for upgrading 1.4 to 1.7.Will it work for 1.5.2 to 1.7?
Upgrading to InfluxDB 1.7.x | InfluxDB OSS 1.7 Documentation

MarcV · June 6, 2019, 4:39am

@vimal525,

What is the output of :
influxd config | grep -i index ?

Can you copy your database to a test environment or make a backup ?

vimal525 · June 6, 2019, 4:50am

Below is output:
[gbuora@~]$ influxd config | grep -i index
Merging with configuration at: /etc/influxdb/influxdb.conf
index-version = “tsi1”
max-index-log-file-size = 1048576

Yes ,I can take backup of database.

MarcV · June 6, 2019, 5:59am

@vimal525,
Sorry for all these questions …
Do you remember how you changed from tsm to tsi1 ?
Have you lost data in the other measurements ?

vimal525 · June 6, 2019, 6:50am

There are 3 measurements in jmeter3 database. I think we lost data in 2 measurements named as requestsRaw and testStartEnd.But did not lost data in virtualUsers.
Apart from that there are other databases as well and their measurements are intact.

use jmeter3
Using database jmeter3

show measurements;
name: measurements
name

requestsRaw
testStartEnd
virtualUsers

select count(*) from requestsRaw;
name: requestsRaw
time count_errorCount count_nodeName count_responseTime count_testName count_threadName

0 2 2 2 2 2

select count(*) from testStartEnd;
name: testStartEnd
time count_testName

0 5

select count(*) from virtualUsers;
name: virtualUsers
time count_finishedThreads count_maxActiveThreads count_meanActiveThreads count_minActiveThreads count_startedThreads

0 13936 13936 13936 13936 13936

Do you remember how you changed from tsm to tsi1 ?
That I am not sure,Since installation is done by other person.Who already left organisation.
Influxdb is come with installation of Grafana version 5.1.3 .

vimal525 · June 12, 2019, 10:01am

Hi All,

I have found work around to get lost data.While this is not exact resolution of lost data.It’s just a work around.I thought to share with all of you ,So that it can help someone who faced same issue.

As earlier mentioned ,files(shards) are physically available on disk for my database but the data was not visible after server reboot.

If this is case then follow below steps to get your lost data

please take backup of your exisiting database with below commands

influxd backup -database -portable /Folder_where_you_want_to_store_backup/

Restore the database with new db name

influxd restore -db “your database name” -newdb “your new database name” -portable /Folder_where_you_stored_backup/

Now you can access the lost data from your new database.

Uncle_Jani · May 7, 2021, 5:45pm

Hi @vimal525 ,
I was hoping your workaround would help me out in my situation, but unfortunately it didnt.
See my topic.
So did you have any other resolutions for that problem later?
Kind regards
Uncle_Jani

Topic		Replies	Views
Recover single measurement data	3	4392	August 30, 2017
Lost measurements after restoring a database from backup influxdb , backup	2	1881	May 13, 2019
Data Lost After Reboot	0	595	October 27, 2019
Data lost from InfluxDB databases	9	9000	June 21, 2017
Data lost after upgrade? InfluxDB 2	5	1820	November 2, 2023

Data loss after db server reboot

cache-snapshot-memory-size = “25m”

cache-snapshot-write-cold-duration = “10m”

Related topics