Index Inconsistency

Store
1

Hi all. When I run influx_inspect verify-seriesfile, I get quite a few “index inconsistency” errors like the lines pasted below.

Can anyone suggest how to repair these?

2019-01-19T02:16:26.589793Z error Index inconsistency {“log_id”: “0D4z9zJl000”, “path”: “/var/lib/influxdb/data/_internal/_series”, “partition”: “00”, “id”: 1, “got_offset”: 5, “expected_offset”: 0}
2019-01-19T02:16:26.726176Z error Index inconsistency {“log_id”: “0D4z9zJl000”, “path”: “/var/lib/influxdb/data/_kapacitor/_series”, “partition”: “00”, “id”: 25, “got_offset”: 712, “expected_offset”: 0}
2019-01-19T02:16:26.726919Z error Index inconsistency {“log_id”: “0D4z9zJl000”, “path”: “/var/lib/influxdb/data/_kapacitor/_series”, “partition”: “01”, “id”: 34, “got_offset”: 949, “expected_offset”: 0}
2019-01-19T02:16:26.891847Z error Index inconsistency {“log_id”: “0D4z9zJl000”, “path”: “/var/lib/influxdb/data//_series”, “partition”: “01”, “id”: 26, “got_offset”: 303, “expected_offset”: 0}
2019-01-19T02:16:29.106959Z error Index inconsistency {“log_id”: “0D4z9zJl000”, “path”: “/var/lib/influxdb/data//_series”, “partition”: “01”, “id”: 42194, “got_id”: 2302570, “expected_id”: 0}

2

What version of InfluxDB are you running?

3

Influxdb 1.7.2 (Docker container)

4

Is there any recommended way to fix that error? I see the same problem in my database, too.

# influx_inspect verify-seriesfile -dir /var/lib/influxdb/data/
2019-04-07T17:40:23.161548Z     error   Index inconsistency     {"log_id": "0EfE1klG000", "path": "/var/lib/influxdb/data/_internal/_series", "partition": "03", "id": 4, "got_offset": 5, "expected_offset": 0}
2019-04-07T17:40:23.163579Z     error   Index inconsistency     {"log_id": "0EfE1klG000", "path": "/var/lib/influxdb/data/_internal/_series", "partition": "01", "id": 10, "got_offset": 63, "expected_offset": 0}
2019-04-07T17:40:23.167433Z     error   Index inconsistency     {"log_id": "0EfE1klG000", "path": "/var/lib/influxdb/data/_internal/_series", "partition": "02", "id": 11, "got_offset": 44, "expected_offset": 0}
2019-04-07T17:40:23.169752Z     error   Index inconsistency     {"log_id": "0EfE1klG000", "path": "/var/lib/influxdb/data/_internal/_series", "partition": "05", "id": 14, "got_offset": 62, "expected_offset": 0}
2019-04-07T17:40:23.173533Z     error   Index inconsistency     {"log_id": "0EfE1klG000", "path": "/var/lib/influxdb/data/_internal/_series", "partition": "04", "id": 5, "got_offset": 5, "expected_offset": 0}
2019-04-07T17:40:23.210691Z     error   Index inconsistency     {"log_id": "0EfE1klG000", "path": "/var/lib/influxdb/data/telegraf/_series", "partition": "00", "id": 409, "got_offset": 11818, "expected_offset": 0}
2019-04-07T17:40:23.213619Z     error   Index inconsistency     {"log_id": "0EfE1klG000", "path": "/var/lib/influxdb/data/telegraf/_series", "partition": "04", "id": 605, "got_offset": 16270, "expected_offset": 0}
5

Hello,

I have the same problem here.
How can we fix index inconsistency errors ?

root@runtsdb001:~# sudo -u influxdb influx_inspect verify-seriesfile -dir /var/lib/influxdb/data/
2019-04-17T07:20:15.140328Z error Index inconsistency {“log_id”: “0ErYX830000”, “path”: “/var/lib/influxdb/data/***/_series”, “partition”: “00”, “id”: 2297, “got_offset”: 35451, “expected_offset”: 0}
2019-04-17T07:20:15.141302Z error Index inconsistency {“log_id”: “0ErYX830000”, “path”: “/var/lib/influxdb/data/***/_series”, “partition”: “01”, “id”: 2226, “got_offset”: 33497, “expected_offset”: 0}
2019-04-17T07:20:15.151288Z error Index inconsistency {“log_id”: “0ErYX830000”, “path”: “/var/lib/influxdb/data/***/_series”, “partition”: “04”, “id”: 5421, “got_offset”: 74567, “expected_offset”: 0}
2019-04-17T07:20:15.151322Z error Index inconsistency {“log_id”: “0ErYX830000”, “path”: “/var/lib/influxdb/data/***/_series”, “partition”: “06”, “id”: 6407, “got_offset”: 83999, “expected_offset”: 0}
2019-04-17T07:20:15.308222Z error Index inconsistency {“log_id”: “0ErYX830000”, “path”: “/var/lib/influxdb/data/***/_series”, “partition”: “01”, “id”: 1338, “got_offset”: 17126, “expected_offset”: 0}
2019-04-17T07:20:15.361514Z error Index inconsistency {“log_id”: “0ErYX830000”, “path”: “/var/lib/influxdb/data/_internal/_series”, “partition”: “05”, “id”: 6, “got_offset”: 5, “expected_offset”: 0}
2019-04-17T07:20:15.446883Z error Index inconsistency {“log_id”: “0ErYX830000”, “path”: “/var/lib/influxdb/data/poc/_series”, “partition”: “01”, “id”: 738, “got_offset”: 8375, “expected_offset”: 0}
2019-04-17T07:20:15.507189Z error Index inconsistency {“log_id”: “0ErYX830000”, “path”: “/var/lib/influxdb/data/test/_series”, “partition”: “06”, “id”: 71, “got_offset”: 442, “expected_offset”: 0}

6

We’re looking into this now. I’ll post an update when I know more!

Edit:

We have a fix in the works for this issue (fix(influx_inspect): verify-seriesfile with deleted index entries by zeebo · Pull Request #13458 · influxdata/influxdb · GitHub). Essentially, the errors you’re seeing are a symptom of hitting a bug in the verify-seriesfile, but they’re not real errors.

7

We are still using v. 1.7.9 and having this issue.
Is there any way to understand if we are hitting the bug reported here or facing a real issue?
Has the bug been fixed only in v 1.8?