I got this error:
2022/03/29 07:20:39 INFO: Backing up TSM for shard 4
2022/03/29 07:20:39 WARN: Shard 4 removed during backup
2022/03/29 07:20:39 INFO: Backing up TSM for shard 5
2022/03/29 07:20:39 WARN: Shard 5 removed during backup
2022/03/29 07:20:39 INFO: Backing up TSM for shard 7
2022/03/29 07:20:39 WARN: Shard 7 removed during backup
2022/03/29 07:20:39 INFO: Backing up TSM for shard 8
2022/03/29 07:20:39 INFO: Backing up TSM for shard 9
2022/03/29 07:20:39 INFO: Backing up TSM for shard 10
2022/03/29 07:20:39 INFO: Backing up TSM for shard 19
2022/03/29 07:20:39 INFO: Backing up TSM for shard 3
2022/03/29 07:20:39 INFO: Backing up TSM for shard 12
2022/03/29 07:20:39 INFO: Backing up TSM for shard 2
2022/03/29 07:20:39 INFO: Backing up TSM for shard 11
2022/03/29 07:20:39 INFO: Backing up TSM for shard 1
2022/03/29 07:20:46 INFO: Backing up TSM for shard 13
2022/03/29 07:20:47 INFO: Backing up TSM for shard 18
Is it all ok? I’m using v 2.1 with Docker.
@sineverba was this during a restore?
opened 11:24AM - 07 Oct 19 UTC
closed 04:26PM - 28 Jan 22 UTC
area/backup and restore
area/storage
1.x
<!--
Thank you for reporting a bug in InfluxDB.
* Please ask usage questi… ons on the Influx Community site.
* https://community.influxdata.com/
* Please add a :+1: or comment on a similar existing bug report instead of opening a new one.
* https://github.com/influxdata/influxdb/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+is%3Aclosed+sort%3Aupdated-desc+label%3Akind%2Fbug+
* Please check whether the bug can be reproduced with the latest release.
* The fastest way to fix a bug is to open a Pull Request.
* https://github.com/influxdata/influxdb/pulls
-->
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Backup database with:
`influxd backup -host ${INFLUXDB_HOST} -portable -database ${DB} ${DB_BACKUP_DIR}`
2. Upload files to the new host
3. Try to restore database with:
`influxd restore -portable /backup/f3-20191004_1215`
__Expected behavior:__
Should restore all shards without warnings.
__Actual behavior:__
A lot of messages like these:
```
ts=2019-10-07T10:11:47.626290Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="engine is closed"
ts=2019-10-07T10:11:47.626467Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=77
...
ts=2019-10-07T10:12:13.487297Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=36
ts=2019-10-07T10:12:13.490966Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="engine is closed"
```
__Environment info:__
* System info: Run `uname -srm` and copy the output here
source: `Linux 4.19.25-coreos x86_64`
target: `Linux 4.9.184-linuxkit x86_64`
* InfluxDB version: Run `influxd version` and copy the output here
source `InfluxDB v1.7.8 (git: 1.7 ff383cdc0420217e3460dabe17db54f8557d95b6)`
target: `InfluxDB v1.7.8 (git: 1.7 ff383cdc0420217e3460dabe17db54f8557d95b6)`
* Other relevant environment details: Container runtime, disk info, etc
Source is running as a docker container on c5.4xlarge AWS EC2 instance (16 CPU, 32GB RAM).
Target is running as a docker container on a local dev laptop (3 CPUs and 12GB of RAM).
__Config:__
Copy any non-default config values here or attach the full config as a gist or file.
Source modifications:
```
env:
- name: INFLUXDB_GRAPHITE_ENABLED
value: "true"
- name: INFLUXDB_UDP_ENABLED
value: "true"
- name: INFLUXDB_HTTP_FLUX_ENABLED
value: "true"
- name: INFLUXDB_DATA_INDEX_VERSION
value: "tsi1"
- name: INFLUXDB_COORDINATOR_QUERY_TIMEOUT
value: "10s"
- name: INFLUXDB_COORDINATOR_LOG_QUERIES_AFTER
value: "10s"
- name: INFLUXDB_BIND_ADDRESS
value: "0.0.0.0:8088"
```
Target runs as
__Logs:__
```
ts=2019-10-07T10:10:37.326721Z lvl=info msg="InfluxDB starting" log_id=0ILR1jZW000 version=1.7.8 branch=1.7 commit=ff383cdc0420217e3460dabe17db54f8557d95b6
ts=2019-10-07T10:10:37.326877Z lvl=info msg="Go runtime" log_id=0ILR1jZW000 version=go1.11 maxprocs=3
ts=2019-10-07T10:10:37.441600Z lvl=info msg="Using data dir" log_id=0ILR1jZW000 service=store path=/var/lib/influxdb/data
ts=2019-10-07T10:10:37.445285Z lvl=info msg="Compaction settings" log_id=0ILR1jZW000 service=store max_concurrent_compactions=1 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
ts=2019-10-07T10:10:37.445399Z lvl=info msg="Open store (start)" log_id=0ILR1jZW000 service=store trace_id=0ILR1k1G000 op_name=tsdb_open op_event=start
ts=2019-10-07T10:10:37.521508Z lvl=info msg="Reading file" log_id=0ILR1jZW000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/1/_00001.wal size=12927
ts=2019-10-07T10:10:37.533550Z lvl=info msg="Opened shard" log_id=0ILR1jZW000 service=store trace_id=0ILR1k1G000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/_internal/monitor/1 duration=29.037ms
ts=2019-10-07T10:10:37.533646Z lvl=info msg="Open store (end)" log_id=0ILR1jZW000 service=store trace_id=0ILR1k1G000 op_name=tsdb_open op_event=end op_elapsed=88.251ms
ts=2019-10-07T10:10:37.533832Z lvl=info msg="Opened service" log_id=0ILR1jZW000 service=subscriber
ts=2019-10-07T10:10:37.533870Z lvl=info msg="Starting monitor service" log_id=0ILR1jZW000 service=monitor
ts=2019-10-07T10:10:37.533880Z lvl=info msg="Registered diagnostics client" log_id=0ILR1jZW000 service=monitor name=build
ts=2019-10-07T10:10:37.533887Z lvl=info msg="Registered diagnostics client" log_id=0ILR1jZW000 service=monitor name=runtime
ts=2019-10-07T10:10:37.533921Z lvl=info msg="Registered diagnostics client" log_id=0ILR1jZW000 service=monitor name=network
ts=2019-10-07T10:10:37.533939Z lvl=info msg="Registered diagnostics client" log_id=0ILR1jZW000 service=monitor name=system
ts=2019-10-07T10:10:37.533956Z lvl=info msg="Starting precreation service" log_id=0ILR1jZW000 service=shard-precreation check_interval=10m advance_period=30m
ts=2019-10-07T10:10:37.533963Z lvl=info msg="Starting snapshot service" log_id=0ILR1jZW000 service=snapshot
ts=2019-10-07T10:10:37.533968Z lvl=info msg="Starting continuous query service" log_id=0ILR1jZW000 service=continuous_querier
ts=2019-10-07T10:10:37.533973Z lvl=info msg="Starting HTTP service" log_id=0ILR1jZW000 service=httpd authentication=false
ts=2019-10-07T10:10:37.533976Z lvl=info msg="opened HTTP access log" log_id=0ILR1jZW000 service=httpd path=stderr
ts=2019-10-07T10:10:37.534078Z lvl=info msg="Listening on HTTP" log_id=0ILR1jZW000 service=httpd addr=[::]:8086 https=false
ts=2019-10-07T10:10:37.534093Z lvl=info msg="Starting retention policy enforcement service" log_id=0ILR1jZW000 service=retention check_interval=30m
ts=2019-10-07T10:10:37.534260Z lvl=info msg="Storing statistics" log_id=0ILR1jZW000 service=monitor db_instance=_internal db_rp=monitor interval=10s
ts=2019-10-07T10:10:37.534506Z lvl=info msg="Listening for signals" log_id=0ILR1jZW000
ts=2019-10-07T10:10:37.534633Z lvl=info msg="Sending usage statistics to usage.influxdata.com" log_id=0ILR1jZW000
ts=2019-10-07T10:11:05.632784Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/32/000000015-000000002.tsm id=0 duration=15.251ms
ts=2019-10-07T10:11:47.626290Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="engine is closed"
ts=2019-10-07T10:11:47.626467Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=77
ts=2019-10-07T10:11:48.468494Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/77/000000193-000000002.tsm id=0 duration=711.420ms
ts=2019-10-07T10:11:53.062240Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/18/000000015-000000002.tsm id=0 duration=14.591ms
ts=2019-10-07T10:11:54.739358Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/31/000000016-000000002.tsm id=0 duration=25.957ms
ts=2019-10-07T10:11:59.444467Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/54/000000022-000000003.tsm id=0 duration=71.394ms
ts=2019-10-07T10:12:03.309322Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=87
ts=2019-10-07T10:12:04.234337Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/59/000000031-000000003.tsm id=0 duration=102.396ms
ts=2019-10-07T10:12:04.332932Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/87/000000156-000000002.tsm id=0 duration=851.227ms
ts=2019-10-07T10:12:11.866406Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=25
ts=2019-10-07T10:12:11.992990Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/25/000000071-000000002.tsm id=0 duration=83.623ms
ts=2019-10-07T10:12:13.487297Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=36
ts=2019-10-07T10:12:13.490966Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="engine is closed"
ts=2019-10-07T10:12:13.555212Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/36/000000018-000000002.tsm id=0 duration=31.417ms
ts=2019-10-07T10:12:14.521182Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=39
ts=2019-10-07T10:12:14.617060Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/39/000000015-000000002.tsm id=0 duration=27.660ms
ts=2019-10-07T10:12:41.253858Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/24/000000015-000000002.tsm id=0 duration=20.952ms
ts=2019-10-07T10:12:43.281293Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/44/000000015-000000003.tsm id=0 duration=165.497ms
ts=2019-10-07T10:12:45.426243Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/52/000000019-000000004.tsm id=0 duration=53.571ms
ts=2019-10-07T10:12:52.161687Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/85/000000168-000000002.tsm id=0 duration=613.393ms
ts=2019-10-07T10:12:58.773898Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/66/000000052-000000003.tsm id=0 duration=107.989ms
ts=2019-10-07T10:12:59.425764Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="shard is disabled"
ts=2019-10-07T10:13:06.526161Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/69/000000053-000000002.tsm id=0 duration=232.124ms
ts=2019-10-07T10:13:34.248077Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=3 trace_id=0ILRCXf0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/03 op_event=start
ts=2019-10-07T10:13:34.260067Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=4 trace_id=0ILRCXi0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/04 op_event=start
ts=2019-10-07T10:13:34.273958Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=0 trace_id=0ILRCXlG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/00 op_event=start
ts=2019-10-07T10:13:34.277939Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=5 trace_id=0ILRCXmG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/05 op_event=start
ts=2019-10-07T10:13:34.698322Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=1 trace_id=0ILRCZQW000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/01 op_event=start
ts=2019-10-07T10:13:34.714712Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=2 trace_id=0ILRCZUW000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/02 op_event=start
ts=2019-10-07T10:13:35.205503Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=7 trace_id=0ILRCaPG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/07 op_event=start
ts=2019-10-07T10:13:35.213474Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=6 trace_id=0ILRCaRG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/06 op_event=start
ts=2019-10-07T10:13:35.583694Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=3 trace_id=0ILRCXf0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/03 op_event=end op_elapsed=1335.642ms
ts=2019-10-07T10:13:35.813518Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=5 trace_id=0ILRCXmG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/05 op_event=end op_elapsed=1535.590ms
ts=2019-10-07T10:13:35.973803Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=0 trace_id=0ILRCXlG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/00 op_event=end op_elapsed=1699.868ms
ts=2019-10-07T10:13:36.035225Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=4 trace_id=0ILRCXi0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/04 op_event=end op_elapsed=1775.173ms
ts=2019-10-07T10:13:36.388081Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=2 trace_id=0ILRCZUW000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/02 op_event=end op_elapsed=1673.380ms
ts=2019-10-07T10:13:36.543421Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=1 trace_id=0ILRCZQW000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/01 op_event=end op_elapsed=1845.108ms
ts=2019-10-07T10:13:36.584074Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=6 trace_id=0ILRCaRG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/06 op_event=end op_elapsed=1370.679ms
ts=2019-10-07T10:13:36.840178Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=7 trace_id=0ILRCaPG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/07 op_event=end op_elapsed=1634.687ms
ts=2019-10-07T10:14:19.107136Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=92
ts=2019-10-07T10:14:23.798974Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/92/000000177-000000002.tsm id=0 duration=4162.136ms
ts=2019-10-07T10:14:53.514849Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/34/000000015-000000002.tsm id=0 duration=85.640ms
ts=2019-10-07T10:14:54.893778Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/42/000000015-000000003.tsm id=0 duration=59.970ms
ts=2019-10-07T10:14:55.743345Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/45/000000015-000000003.tsm id=0 duration=36.454ms
ts=2019-10-07T10:24:10.375622Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/47/000000015-000000003.tsm id=0 duration=86.457ms
ts=2019-10-07T10:24:22.543407Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/49/000000021-000000003.tsm id=0 duration=112.245ms
ts=2019-10-07T10:24:33.565916Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/57/000000022-000000003.tsm id=0 duration=82.496ms
ts=2019-10-07T10:24:53.595079Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/67/000000050-000000003.tsm id=0 duration=512.198ms
ts=2019-10-07T10:24:58.143150Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=97
ts=2019-10-07T10:24:59.770794Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/97/000000224-000000002.tsm id=0 duration=1073.836ms
ts=2019-10-07T10:25:22.507769Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/37/000000015-000000002.tsm id=0 duration=22.937ms
ts=2019-10-07T10:25:27.895507Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/53/000000020-000000003.tsm id=0 duration=65.498ms
ts=2019-10-07T10:25:41.887707Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/62/000000046-000000003.tsm id=0 duration=134.798ms
ts=2019-10-07T10:25:42.347925Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/90/000000159-000000002.tsm id=0 duration=762.383ms
ts=2019-10-07T10:25:47.661753Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=68
ts=2019-10-07T10:25:47.662854Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="shard is disabled"
ts=2019-10-07T10:25:47.863131Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/68/000000049-000000003.tsm id=0 duration=145.193ms
ts=2019-10-07T10:25:59.362085Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=72
ts=2019-10-07T10:25:59.848841Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/72/000000061-000000003.tsm id=0 duration=450.964ms
ts=2019-10-07T10:26:20.502486Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=75
ts=2019-10-07T10:26:20.970431Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/75/000000086-000000002.tsm id=0 duration=290.982ms
ts=2019-10-07T10:27:30.053974Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/51/000000017-000000004.tsm id=0 duration=218.161ms
ts=2019-10-07T10:28:10.876148Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/70/000000051-000000002.tsm id=0 duration=204.799ms
ts=2019-10-07T10:29:29.633009Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=79
ts=2019-10-07T10:29:30.738507Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/26/000000066-000000002.tsm id=0 duration=276.988ms
ts=2019-10-07T10:29:34.014130Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/35/000000015-000000002.tsm id=0 duration=189.369ms
ts=2019-10-07T10:29:35.025978Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/79/000000273-000000002.tsm id=0 duration=3411.019ms
ts=2019-10-07T10:29:41.269560Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/50/000000019-000000003.tsm id=0 duration=100.924ms
ts=2019-10-07T10:30:27.475770Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=5 trace_id=0ILSANYW000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/05 op_event=start
ts=2019-10-07T10:30:27.792054Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=4 trace_id=0ILSAOoG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/04 op_event=start
ts=2019-10-07T10:30:40.306896Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=2 trace_id=0ILSB9h0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/02 op_event=start
ts=2019-10-07T10:30:40.751459Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=0 trace_id=0ILSBBRW000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/00 op_event=start
ts=2019-10-07T10:30:40.777773Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=1 trace_id=0ILSBBY0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/01 op_event=start
ts=2019-10-07T10:30:40.929512Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=3 trace_id=0ILSBC8G000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/03 op_event=start
ts=2019-10-07T10:30:56.917830Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=7 trace_id=0ILSCAaG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/07 op_event=start
ts=2019-10-07T10:30:57.531913Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=6 trace_id=0ILSCCzl000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/06 op_event=start
ts=2019-10-07T10:31:13.816393Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=5 trace_id=0ILSANYW000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/05 op_event=end op_elapsed=46378.277ms
ts=2019-10-07T10:31:20.822998Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=4 trace_id=0ILSAOoG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/04 op_event=end op_elapsed=53099.895ms
ts=2019-10-07T10:31:23.358325Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=2 trace_id=0ILSB9h0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/02 op_event=end op_elapsed=43120.066ms
ts=2019-10-07T10:31:23.494619Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=3 trace_id=0ILSBC8G000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/03 op_event=end op_elapsed=42631.650ms
ts=2019-10-07T10:31:23.535097Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=7 trace_id=0ILSCAaG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/07 op_event=end op_elapsed=26651.612ms
ts=2019-10-07T10:31:23.732939Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=1 trace_id=0ILSBBY0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/01 op_event=end op_elapsed=43021.941ms
ts=2019-10-07T10:31:23.791646Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=6 trace_id=0ILSCCzl000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/06 op_event=end op_elapsed=26294.225ms
ts=2019-10-07T10:31:23.845948Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=0 trace_id=0ILSBBRW000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/00 op_event=end op_elapsed=43161.123ms
ts=2019-10-07T10:31:56.593108Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/82/000000179-000000002.tsm id=0 duration=3139.733ms
ts=2019-10-07T10:34:30.938811Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/80/000000239-000000002.tsm id=0 duration=3544.004ms
ts=2019-10-07T10:34:30.985692Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/19/000000004-000000002.tsm id=0 duration=80.263ms
ts=2019-10-07T10:34:37.605651Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/43/000000015-000000003.tsm id=0 duration=180.469ms
ts=2019-10-07T10:34:57.849726Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/56/000000020-000000003.tsm id=0 duration=256.403ms
ts=2019-10-07T10:35:14.138001Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="shard is disabled"
ts=2019-10-07T10:35:42.449184Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/64/000000053-000000002.tsm id=0 duration=608.478ms
ts=2019-10-07T10:35:52.011501Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/81/000000216-000000003.tsm id=0 duration=3813.281ms
ts=2019-10-07T10:38:43.710175Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000032-000000003.tsm id=0 duration=3819.941ms
ts=2019-10-07T10:38:45.133952Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000064-000000003.tsm id=1 duration=4551.274ms
ts=2019-10-07T10:38:45.531769Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000096-000000003.tsm id=2 duration=4172.539ms
ts=2019-10-07T10:38:46.185389Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000104-000000002.tsm id=3 duration=1498.636ms
ts=2019-10-07T10:38:46.812193Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000122-000000001.tsm id=6 duration=626.630ms
ts=2019-10-07T10:38:47.090880Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000121-000000002.tsm id=5 duration=1558.650ms
ts=2019-10-07T10:38:47.170856Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000123-000000001.tsm id=7 duration=358.130ms
ts=2019-10-07T10:38:47.190111Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000113-000000002.tsm id=4 duration=2055.730ms
ts=2019-10-07T10:38:47.594087Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000124-000000001.tsm id=8 duration=502.650ms
ts=2019-10-07T10:38:47.636348Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000125-000000001.tsm id=9 duration=464.799ms
ts=2019-10-07T10:38:47.648389Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000126-000000001.tsm id=10 duration=457.460ms
ts=2019-10-07T10:38:48.111279Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000129-000000001.tsm id=13 duration=432.041ms
ts=2019-10-07T10:38:48.112530Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000128-000000001.tsm id=12 duration=510.304ms
ts=2019-10-07T10:38:48.161430Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/99/000000127-000000001.tsm id=11 duration=605.858ms
ts=2019-10-07T10:39:40.826621Z lvl=info msg="TSM compaction (start)" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group op_event=start
ts=2019-10-07T10:39:40.827460Z lvl=info msg="Beginning compaction" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_files_n=8
ts=2019-10-07T10:39:40.827540Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000122-000000001.tsm
ts=2019-10-07T10:39:40.827586Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000123-000000001.tsm
ts=2019-10-07T10:39:40.827628Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000124-000000001.tsm
ts=2019-10-07T10:39:40.827672Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000125-000000001.tsm
ts=2019-10-07T10:39:40.827747Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_index=4 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000126-000000001.tsm
ts=2019-10-07T10:39:40.827853Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_index=5 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000127-000000001.tsm
ts=2019-10-07T10:39:40.827931Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_index=6 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000128-000000001.tsm
ts=2019-10-07T10:39:40.827977Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_index=7 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000129-000000001.tsm
ts=2019-10-07T10:40:13.094951Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/23/000000022-000000002.tsm id=0 duration=34.070ms
ts=2019-10-07T10:40:25.923091Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/38/000000015-000000002.tsm id=0 duration=208.706ms
ts=2019-10-07T10:40:28.784790Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/40/000000015-000000002.tsm id=0 duration=196.281ms
ts=2019-10-07T10:40:38.962713Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/60/000000033-000000003.tsm id=0 duration=166.063ms
ts=2019-10-07T10:40:57.705778Z lvl=info msg="Compacted file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000129-000000002.tsm.tmp
ts=2019-10-07T10:40:57.705941Z lvl=info msg="Finished compacting files" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2019-10-07T10:40:57.706043Z lvl=info msg="TSM compaction (end)" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=1 tsm1_strategy=level trace_id=0ILSh8yW000 op_name=tsm1_compact_group op_event=end op_elapsed=76986.053ms
ts=2019-10-07T10:40:58.342664Z lvl=info msg="TSM compaction (start)" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0ILSlsuW000 op_name=tsm1_compact_group op_event=start
ts=2019-10-07T10:40:58.365568Z lvl=info msg="Beginning compaction" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0ILSlsuW000 op_name=tsm1_compact_group tsm1_files_n=4
ts=2019-10-07T10:40:58.365777Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0ILSlsuW000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000104-000000002.tsm
ts=2019-10-07T10:40:58.365847Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0ILSlsuW000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000113-000000002.tsm
ts=2019-10-07T10:40:58.365895Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0ILSlsuW000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000121-000000002.tsm
ts=2019-10-07T10:40:58.365957Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0ILSlsuW000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000129-000000002.tsm
ts=2019-10-07T10:41:12.092017Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/65/000000050-000000002.tsm id=0 duration=778.758ms
ts=2019-10-07T10:42:03.149557Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=94
ts=2019-10-07T10:42:28.017224Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/94/000000241-000000002.tsm id=0 duration=20781.688ms
ts=2019-10-07T10:42:35.023943Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/71/000000055-000000002.tsm id=0 duration=3930.645ms
ts=2019-10-07T10:43:54.798325Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=74
ts=2019-10-07T10:43:55.401732Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="engine is closed"
ts=2019-10-07T10:43:56.870576Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/74/000000076-000000002.tsm id=0 duration=773.190ms
ts=2019-10-07T10:47:03.475081Z lvl=info msg="Compacted file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0ILSlsuW000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000129-000000003.tsm.tmp
ts=2019-10-07T10:47:03.477472Z lvl=info msg="Finished compacting files" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0ILSlsuW000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2019-10-07T10:47:03.477575Z lvl=info msg="TSM compaction (end)" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0ILSlsuW000 op_name=tsm1_compact_group op_event=end op_elapsed=365548.962ms
ts=2019-10-07T10:47:03.931690Z lvl=info msg="TSM compaction (start)" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=3 tsm1_strategy=level trace_id=0ILT7BzG000 op_name=tsm1_compact_group op_event=start
ts=2019-10-07T10:47:03.932098Z lvl=info msg="Beginning compaction" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=3 tsm1_strategy=level trace_id=0ILT7BzG000 op_name=tsm1_compact_group tsm1_files_n=4
ts=2019-10-07T10:47:03.932163Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=3 tsm1_strategy=level trace_id=0ILT7BzG000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000032-000000003.tsm
ts=2019-10-07T10:47:03.932257Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=3 tsm1_strategy=level trace_id=0ILT7BzG000 op_name=tsm1_compact_group tsm1_index=1 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000064-000000003.tsm
ts=2019-10-07T10:47:03.932295Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=3 tsm1_strategy=level trace_id=0ILT7BzG000 op_name=tsm1_compact_group tsm1_index=2 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000096-000000003.tsm
ts=2019-10-07T10:47:03.932328Z lvl=info msg="Compacting file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=3 tsm1_strategy=level trace_id=0ILT7BzG000 op_name=tsm1_compact_group tsm1_index=3 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000129-000000003.tsm
ts=2019-10-07T10:48:39.878451Z lvl=info msg="Cache snapshot (start)" log_id=0ILR1jZW000 engine=tsm1 trace_id=0ILTD2mG000 op_name=tsm1_cache_snapshot op_event=start
ts=2019-10-07T10:48:49.236982Z lvl=info msg="Snapshot for path written" log_id=0ILR1jZW000 engine=tsm1 trace_id=0ILTD2mG000 op_name=tsm1_cache_snapshot path=/var/lib/influxdb/data/_internal/monitor/1 duration=9393.894ms
ts=2019-10-07T10:48:49.240949Z lvl=info msg="Cache snapshot (end)" log_id=0ILR1jZW000 engine=tsm1 trace_id=0ILTD2mG000 op_name=tsm1_cache_snapshot op_event=end op_elapsed=9397.740ms
ts=2019-10-07T10:50:05.641388Z lvl=info msg="Retention policy deletion check (start)" log_id=0ILR1jZW000 service=retention trace_id=0ILTIHn0000 op_name=retention_delete_check op_event=start
ts=2019-10-07T10:50:10.514977Z lvl=info msg="Deleted shard group" log_id=0ILR1jZW000 service=retention trace_id=0ILTIHn0000 op_name=retention_delete_check db_instance=system db_shard_group=4 db_rp=default
ts=2019-10-07T10:51:55.417955Z lvl=info msg="Compacted file" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=3 tsm1_strategy=level trace_id=0ILT7BzG000 op_name=tsm1_compact_group tsm1_index=0 tsm1_file=/var/lib/influxdb/data/f3/default/99/000000129-000000004.tsm.tmp
ts=2019-10-07T10:51:55.418432Z lvl=info msg="Finished compacting files" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=3 tsm1_strategy=level trace_id=0ILT7BzG000 op_name=tsm1_compact_group tsm1_files_n=1
ts=2019-10-07T10:51:55.418670Z lvl=info msg="TSM compaction (end)" log_id=0ILR1jZW000 engine=tsm1 tsm1_level=3 tsm1_strategy=level trace_id=0ILT7BzG000 op_name=tsm1_compact_group op_event=end op_elapsed=291836.529ms
ts=2019-10-07T10:52:03.734994Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=98
ts=2019-10-07T10:52:08.409491Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/21/000000015-000000002.tsm id=0 duration=143.920ms
ts=2019-10-07T10:52:12.582001Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/98/000000214-000000002.tsm id=0 duration=4452.406ms
ts=2019-10-07T10:52:43.819580Z lvl=info msg="failed to store statistics" log_id=0ILR1jZW000 service=monitor error=timeout
ts=2019-10-07T10:52:48.111748Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/29/000000021-000000002.tsm id=0 duration=133.747ms
ts=2019-10-07T10:53:05.861305Z lvl=info msg="Deleted shard" log_id=0ILR1jZW000 service=retention trace_id=0ILTIHn0000 op_name=retention_delete_check db_instance=system db_shard_id=3 db_rp=default
ts=2019-10-07T10:53:08.224982Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/93/000000199-000000002.tsm id=0 duration=1922.044ms
ts=2019-10-07T10:53:09.636539Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/33/000000015-000000002.tsm id=0 duration=35.551ms
ts=2019-10-07T10:53:18.118175Z lvl=info msg="Deleted shard" log_id=0ILR1jZW000 service=retention trace_id=0ILTIHn0000 op_name=retention_delete_check db_instance=system db_shard_id=2 db_rp=default
ts=2019-10-07T10:53:27.387797Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=55
ts=2019-10-07T10:53:27.389693Z lvl=info msg="Deleted shard" log_id=0ILR1jZW000 service=retention trace_id=0ILTIHn0000 op_name=retention_delete_check db_instance=system db_shard_id=4 db_rp=default
ts=2019-10-07T10:53:27.486479Z lvl=info msg="Retention policy deletion check (end)" log_id=0ILR1jZW000 service=retention trace_id=0ILTIHn0000 op_name=retention_delete_check op_event=end op_elapsed=202085.246ms
ts=2019-10-07T10:53:28.234590Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/55/000000022-000000003.tsm id=0 duration=394.444ms
ts=2019-10-07T10:53:56.438611Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=63
ts=2019-10-07T10:53:56.454984Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=63
ts=2019-10-07T10:53:56.849842Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/63/000000051-000000003.tsm id=0 duration=233.415ms
ts=2019-10-07T10:54:45.139955Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=76
ts=2019-10-07T10:54:46.534079Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/76/000000116-000000002.tsm id=0 duration=1039.682ms
ts=2019-10-07T10:54:51.279843Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="shard is disabled"
ts=2019-10-07T10:57:59.213842Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/83/000000184-000000003.tsm id=0 duration=2830.564ms
ts=2019-10-07T10:58:57.384814Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/41/000000016-000000004.tsm id=0 duration=224.794ms
ts=2019-10-07T10:59:42.794754Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=96
ts=2019-10-07T11:00:15.373370Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/96/000000217-000000003.tsm id=0 duration=31764.263ms
ts=2019-10-07T11:00:32.080523Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/27/000000041-000000002.tsm id=0 duration=562.282ms
ts=2019-10-07T11:00:34.193742Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/30/000000016-000000004.tsm id=0 duration=206.995ms
ts=2019-10-07T11:00:41.660256Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=91
ts=2019-10-07T11:00:43.421495Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/91/000000162-000000002.tsm id=0 duration=1516.921ms
ts=2019-10-07T11:00:51.890567Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=61
ts=2019-10-07T11:00:51.891083Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="shard is disabled"
ts=2019-10-07T11:00:52.305088Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/61/000000040-000000003.tsm id=0 duration=244.684ms
ts=2019-10-07T11:04:21.408759Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=78
ts=2019-10-07T11:04:24.081603Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/78/000000287-000000003.tsm id=0 duration=1995.684ms
ts=2019-10-07T11:05:41.780354Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/20/000000001-000000001.tsm id=0 duration=49.499ms
ts=2019-10-07T11:05:55.775088Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/48/000000019-000000003.tsm id=0 duration=199.847ms
ts=2019-10-07T11:06:20.310040Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=95
ts=2019-10-07T11:06:23.886693Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/95/000000222-000000002.tsm id=0 duration=3322.943ms
ts=2019-10-07T11:08:09.857972Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=84
ts=2019-10-07T11:08:12.296110Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/84/000000175-000000002.tsm id=0 duration=1735.868ms
ts=2019-10-07T11:08:19.040798Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="shard is disabled"
ts=2019-10-07T11:09:12.548623Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/22/000000016-000000003.tsm id=0 duration=28.334ms
ts=2019-10-07T11:09:31.910457Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/28/000000027-000000002.tsm id=0 duration=180.718ms
ts=2019-10-07T11:09:36.239139Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/46/000000016-000000003.tsm id=0 duration=82.460ms
ts=2019-10-07T11:09:49.551457Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/58/000000029-000000003.tsm id=0 duration=320.920ms
ts=2019-10-07T11:10:00.512808Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="shard is disabled" db_shard_id=89
ts=2019-10-07T11:10:02.706390Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/89/000000157-000000002.tsm id=0 duration=1540.961ms
ts=2019-10-07T11:10:48.079293Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/73/000000065-000000002.tsm id=0 duration=4334.105ms
ts=2019-10-07T11:13:15.350633Z lvl=warn msg="Error while freeing cold shard resources" log_id=0ILR1jZW000 service=store error="engine is closed" db_shard_id=86
ts=2019-10-07T11:13:17.707782Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/86/000000167-000000002.tsm id=0 duration=1972.604ms
ts=2019-10-07T11:14:28.389411Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=4 trace_id=0ILUgZ_l000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/04 op_event=start
ts=2019-10-07T11:14:29.168725Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=5 trace_id=0ILUgbgl000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/05 op_event=start
ts=2019-10-07T11:14:34.999786Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=1 trace_id=0ILUgyTl000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/01 op_event=start
ts=2019-10-07T11:14:36.210421Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=5 trace_id=0ILUgbgl000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/05 op_event=end op_elapsed=7042.665ms
ts=2019-10-07T11:14:36.635915Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=4 trace_id=0ILUgZ_l000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/04 op_event=end op_elapsed=8246.708ms
ts=2019-10-07T11:14:38.811171Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=3 trace_id=0ILUhCMW000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/03 op_event=start
ts=2019-10-07T11:14:41.137000Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=0 trace_id=0ILUhLS0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/00 op_event=start
ts=2019-10-07T11:14:42.765176Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=7 trace_id=0ILUhRoG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/07 op_event=start
ts=2019-10-07T11:14:42.768817Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=2 trace_id=0ILUhRol000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/02 op_event=start
ts=2019-10-07T11:14:44.115141Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=1 trace_id=0ILUgyTl000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/01 op_event=end op_elapsed=9115.524ms
ts=2019-10-07T11:14:51.588801Z lvl=info msg="Series partition compaction (start)" log_id=0ILR1jZW000 partition=6 trace_id=0ILUhzH0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/06 op_event=start
ts=2019-10-07T11:14:53.812450Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=3 trace_id=0ILUhCMW000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/03 op_event=end op_elapsed=15021.644ms
ts=2019-10-07T11:14:56.548087Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=0 trace_id=0ILUhLS0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/00 op_event=end op_elapsed=15438.183ms
ts=2019-10-07T11:14:57.908406Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=2 trace_id=0ILUhRol000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/02 op_event=end op_elapsed=15167.683ms
ts=2019-10-07T11:15:00.862080Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=7 trace_id=0ILUhRoG000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/07 op_event=end op_elapsed=18123.933ms
ts=2019-10-07T11:15:02.952306Z lvl=info msg="Series partition compaction (end)" log_id=0ILR1jZW000 partition=6 trace_id=0ILUhzH0000 op_name=series_partition_compaction path=/var/lib/influxdb/data/f3/_series/06 op_event=end op_elapsed=11363.563ms
ts=2019-10-07T11:15:07.782025Z lvl=info msg="Cannot retrieve measurement cardinality" log_id=0ILR1jZW000 service=store db_instance=f3 error="shard is disabled"
ts=2019-10-07T11:15:09.388679Z lvl=info msg="Opened file" log_id=0ILR1jZW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/f3/default/88/000000149-000000002.tsm id=0 duration=1331.381ms
```
It is during backup.
I use Influx via Docker.
Some day ago I tried to restore backup on aother server, seems all ok. But, when I do backup, I got all that errors.
In next days I will try another restore on another server, just to try…
Hi,
cguy
January 26, 2023, 11:42pm
5
@HannesKannes Did you find a solution? This happens for me as well:
bash-5.1# influx backup /influxdb2-backup/backup --bucket MyBucket -t MySeCrEtToKeN!
2023/01/26 23:33:24 INFO: Downloading metadata snapshot
2023/01/26 23:33:24 INFO: Backing up TSM for shard 1
2023/01/26 23:33:24 WARN: Shard 1 removed during backup
2023/01/26 23:33:24 INFO: Backing up TSM for shard 2
2023/01/26 23:33:24 WARN: Shard 2 removed during backup
2023/01/26 23:33:24 INFO: Backing up TSM for shard 3
2023/01/26 23:33:24 WARN: Shard 3 removed during backup
2023/01/26 23:33:24 INFO: Backing up TSM for shard 4
2023/01/26 23:33:24 WARN: Shard 4 removed during backup
2023/01/26 23:33:24 INFO: Backing up TSM for shard 5
2023/01/26 23:33:24 WARN: Shard 5 removed during backup
2023/01/26 23:33:24 INFO: Backing up TSM for shard 6
2023/01/26 23:33:24 WARN: Shard 6 removed during backup
2023/01/26 23:33:24 INFO: Backing up TSM for shard 7
2023/01/26 23:33:50 INFO: Backing up TSM for shard 8
And then when restoring from that backup, there is data missing.
This is running influxdb 2.3.0:
InfluxDB v2.3.0+SNAPSHOT.090f681737 Server: 090f681 Frontend: a2bd1f3
MzazM
March 6, 2023, 1:37pm
6
I got the same today (using 2.1.1) . I wonder how to check which data is contained inside of shards that are being removed. I am hoping that that those shard have been deleted because, for example, they fall outside of the Retention policy. How to check this?
MzazM
March 6, 2023, 1:56pm
7
from my manifest file and the content in the backup folder, it seems that all the shards necessary have been backupped.
