Is a missing series file a problem?

I thought I’d check the sanity of my database… running “influx_inspect -verify” was slow but did not report any problems, however:

> sudo influx_inspect verify-seriesfile -dir /var/lib/influxdb/
    2019-08-12T15:47:03.293723Z     error   Series file does not exist      {"log_id": "0HDd1H~G000", "path": "/var/lib/influxdb/data/_series"}
    2019-08-12T15:47:03.298789Z     error   Series file does not exist      {"log_id": "0HDd1H~G000", "path": "/var/lib/influxdb/meta/_series"}
    2019-08-12T15:47:03.298825Z     error   Series file does not exist      {"log_id": "0HDd1H~G000", "path": "/var/lib/influxdb/wal/_series"}

Is this why startup says it fails?

> sudo service influxdb start
Starting influxdb...
influxdb process was unable to start [ FAILED ]

But the log file disagrees… I’ve removed “Opened file” and “shard” log entries.

ts=2019-08-12T13:22:36.180352Z lvl=info msg="Signal received, initializing clean shutdown..." log_id=0HDUclgW000
ts=2019-08-12T13:22:36.180429Z lvl=info msg="Waiting for clean shutdown..." log_id=0HDUclgW000
ts=2019-08-12T13:22:36.180557Z lvl=info msg="Listener closed" log_id=0HDUclgW000 service=snapshot
ts=2019-08-12T13:22:36.180512Z lvl=info msg="Shutting down monitor service" log_id=0HDUclgW000 service=monitor
ts=2019-08-12T13:22:36.180616Z lvl=info msg="Terminating storage of statistics" log_id=0HDUclgW000 service=monitor
ts=2019-08-12T13:22:36.180681Z lvl=info msg="Terminating continuous query service" log_id=0HDUclgW000 service=continuous_querier
ts=2019-08-12T13:22:36.180768Z lvl=info msg="Closing retention policy enforcement service" log_id=0HDUclgW000 service=retention
ts=2019-08-12T13:23:06.180646Z lvl=info msg="Time limit reached, initializing hard shutdown" log_id=0HDUclgW000
ts=2019-08-12T15:51:23.695751Z lvl=info msg="InfluxDB starting" log_id=0HDdHBBl000 version=1.7.6 branch=1.7 commit=01c8dd416270f424ab0c40f9291e269ac6921964
ts=2019-08-12T15:51:23.695805Z lvl=info msg="Go runtime" log_id=0HDdHBBl000 version=go1.11 maxprocs=12
ts=2019-08-12T15:51:23.797100Z lvl=info msg="Using data dir" log_id=0HDdHBBl000 service=store path=/var/lib/influxdb/data
ts=2019-08-12T15:51:23.797157Z lvl=info msg="Compaction settings" log_id=0HDdHBBl000 service=store max_concurrent_compactions=6 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
ts=2019-08-12T15:51:23.797177Z lvl=info msg="Open store (start)" log_id=0HDdHBBl000 service=store trace_id=0HDdHBaG000 op_name=tsdb_open op_event=start
ts=2019-08-12T15:51:24.769362Z lvl=info msg="Reading file" log_id=0HDdHBBl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/627/_00075.wal size=7335186
ts=2019-08-12T15:51:27.164307Z lvl=info msg="Reading file" log_id=0HDdHBBl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/collectd/autogen/626/_00157.wal size=10503321
ts=2019-08-12T15:51:27.300815Z lvl=info msg="Reading file" log_id=0HDdHBBl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/collectd/autogen/91/_00004.wal size=657
ts=2019-08-12T15:51:28.015468Z lvl=info msg="Reading file" log_id=0HDdHBBl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/collectd/autogen/626/_00158.wal size=839971
ts=2019-08-12T15:51:28.327586Z lvl=info msg="Open store (end)" log_id=0HDdHBBl000 service=store trace_id=0HDdHBaG000 op_name=tsdb_open op_event=end op_elapsed=4530.406ms
ts=2019-08-12T15:51:28.327703Z lvl=info msg="Opened service" log_id=0HDdHBBl000 service=subscriber
ts=2019-08-12T15:51:28.327719Z lvl=info msg="Starting monitor service" log_id=0HDdHBBl000 service=monitor
ts=2019-08-12T15:51:28.327734Z lvl=info msg="Registered diagnostics client" log_id=0HDdHBBl000 service=monitor name=build
ts=2019-08-12T15:51:28.327745Z lvl=info msg="Registered diagnostics client" log_id=0HDdHBBl000 service=monitor name=runtime
ts=2019-08-12T15:51:28.327763Z lvl=info msg="Registered diagnostics client" log_id=0HDdHBBl000 service=monitor name=network
ts=2019-08-12T15:51:28.327775Z lvl=info msg="Registered diagnostics client" log_id=0HDdHBBl000 service=monitor name=system
ts=2019-08-12T15:51:28.327824Z lvl=info msg="Starting snapshot service" log_id=0HDdHBBl000 service=snapshot
ts=2019-08-12T15:51:28.327838Z lvl=info msg="Starting continuous query service" log_id=0HDdHBBl000 service=continuous_querier
ts=2019-08-12T15:51:28.327860Z lvl=info msg="Starting HTTP service" log_id=0HDdHBBl000 service=httpd authentication=false
ts=2019-08-12T15:51:28.327871Z lvl=info msg="opened HTTP access log" log_id=0HDdHBBl000 service=httpd path=stderr
ts=2019-08-12T15:51:28.327893Z lvl=info msg="Storing statistics" log_id=0HDdHBBl000 service=monitor db_instance=_internal db_rp=monitor interval=10s
ts=2019-08-12T15:51:28.328049Z lvl=info msg="Listening on HTTP" log_id=0HDdHBBl000 service=httpd addr=[::]:8086 https=false
ts=2019-08-12T15:51:28.328094Z lvl=info msg="Starting retention policy enforcement service" log_id=0HDdHBBl000 service=retention check_interval=30m
ts=2019-08-12T15:51:28.328117Z lvl=info msg="Starting collectd service" log_id=0HDdHBBl000 service=collectd
ts=2019-08-12T15:51:28.328281Z lvl=info msg="Loading types from file" log_id=0HDdHBBl000 service=collectd path=/usr/share/collectd/softel_types.db
ts=2019-08-12T15:51:28.329132Z lvl=info msg="Loading types from file" log_id=0HDdHBBl000 service=collectd path=/usr/share/collectd/types.db
ts=2019-08-12T15:51:28.330794Z lvl=info msg="Loading types from file" log_id=0HDdHBBl000 service=collectd path=/usr/share/collectd/vmware-types.db
ts=2019-08-12T15:51:28.332279Z lvl=info msg="Loading types from file" log_id=0HDdHBBl000 service=collectd path=/usr/share/collectd/zfs.db
ts=2019-08-12T15:51:28.332567Z lvl=info msg="Listening on UDP" log_id=0HDdHBBl000 service=collectd addr=10.1.1.1:25827
ts=2019-08-12T15:51:28.332949Z lvl=info msg="Listening for signals" log_id=0HDdHBBl000

I don’t understand if this is good or bad…

> sudo influx_inspect reporttsi -db-path /var/lib/influxdb/
reporttsi: open /var/lib/influxdb/wal/collectd/autogen/546/index: no such file or directory
> sudo influx_inspect reporttsi -db-path /var/lib/influxdb/
reporttsi: open /var/lib/influxdb/wal/_internal/monitor/623/index: no such file or directory
> sudo influx_inspect reporttsi -db-path /var/lib/influxdb/
reporttsi: open /var/lib/influxdb/wal/_internal/monitor/623/index: no such file or directory
> sudo influx_inspect reporttsi -db-path /var/lib/influxdb/
reporttsi: open /var/lib/influxdb/wal/collectd/autogen/259/index: no such file or directory

Hi @dataMechanic , thanks for sticking around :wink:

missing series files will be recreated when influxdb is restarted ,

your command

influx_inspect verify-seriesfile -dir /var/lib/influxdb/

should be

influx_inspect verify-seriesfile -dir /var/lib/influxdb/data

ps : is your database up and running or does it fail to start ( sudo service influxdb status ) ?
do you have errors in /var/log/messages or journalctl ?

best regards ,

When I point it at the data directory:

influx_inspect verify-seriesfile -dir /var/lib/influxdb/data/
2019-08-13T17:02:34.089914Z	error	Index inconsistency	{"log_id": "0HEzkGH0000", "path": "/var/lib/influxdb/data/_internal/_series", "partition": "01", "id": 18, "got_offset": 172, "expected_offset": 0}
2019-08-13T17:02:34.089504Z	error	Index inconsistency	{"log_id": "0HEzkGH0000", "path": "/var/lib/influxdb/data/_internal/_series", "partition": "00", "id": 17, "got_offset": 151, "expected_offset": 0}
2019-08-13T17:02:34.089987Z	error	Index inconsistency	{"log_id": "0HEzkGH0000", "path": "/var/lib/influxdb/data/_internal/_series", "partition": "02", "id": 3, "got_offset": 5, "expected_offset": 0}
2019-08-13T17:02:34.148585Z	error	Index inconsistency	{"log_id": "0HEzkGH0000", "path": "/var/lib/influxdb/data/collectd/_series", "partition": "04", "id": 333, "got_offset": 3935, "expected_offset": 0}
2019-08-13T17:02:34.148700Z	error	Index inconsistency	{"log_id": "0HEzkGH0000", "path": "/var/lib/influxdb/data/collectd/_series", "partition": "02", "id": 259, "got_offset": 3090, "expected_offset": 0}
2019-08-13T17:02:34.148819Z	error	Index inconsistency	{"log_id": "0HEzkGH0000", "path": "/var/lib/influxdb/data/collectd/_series", "partition": "00", "id": 257, "got_offset": 3052, "expected_offset": 0}

How do I fix that?

hi @dataMechanic,

Because influx_inspect -verify did not report any problems ,

And if you are not able to start your database , you can fix that as follows

1.Stop the database( if not already so)

2.rm -fR /var/lib/influxdb/data/_internal/_series
rm -fR /var/lib/influxdb/data/collectd/_series

3.Start the database

The series will be recreated during startup

Ps if you are hitting the bug mentioned in your other post ,
Chances are that this will not fix your problem , and yet I am optimistic .
But then again in that case you have no problem.

Some people don’t believe in the magic of latest versions , but mostly the latest versions are better because people are working hard to add bug fixes in the latest versions.( in order of priority)

Probably the bug mentioned in your other post has a low priority because it is not a show stopper and will not make a ship sink :wink:
In case you hit the bug , i can advise wat mysql , postgresql , Oracle , just to name a few , would advise : ignore the error.
Best regards