I thought I’d check the sanity of my database… running “influx_inspect -verify” was slow but did not report any problems, however:
> sudo influx_inspect verify-seriesfile -dir /var/lib/influxdb/
2019-08-12T15:47:03.293723Z error Series file does not exist {"log_id": "0HDd1H~G000", "path": "/var/lib/influxdb/data/_series"}
2019-08-12T15:47:03.298789Z error Series file does not exist {"log_id": "0HDd1H~G000", "path": "/var/lib/influxdb/meta/_series"}
2019-08-12T15:47:03.298825Z error Series file does not exist {"log_id": "0HDd1H~G000", "path": "/var/lib/influxdb/wal/_series"}
Is this why startup says it fails?
> sudo service influxdb start
Starting influxdb...
influxdb process was unable to start [ FAILED ]
But the log file disagrees… I’ve removed “Opened file” and “shard” log entries.
ts=2019-08-12T13:22:36.180352Z lvl=info msg="Signal received, initializing clean shutdown..." log_id=0HDUclgW000
ts=2019-08-12T13:22:36.180429Z lvl=info msg="Waiting for clean shutdown..." log_id=0HDUclgW000
ts=2019-08-12T13:22:36.180557Z lvl=info msg="Listener closed" log_id=0HDUclgW000 service=snapshot
ts=2019-08-12T13:22:36.180512Z lvl=info msg="Shutting down monitor service" log_id=0HDUclgW000 service=monitor
ts=2019-08-12T13:22:36.180616Z lvl=info msg="Terminating storage of statistics" log_id=0HDUclgW000 service=monitor
ts=2019-08-12T13:22:36.180681Z lvl=info msg="Terminating continuous query service" log_id=0HDUclgW000 service=continuous_querier
ts=2019-08-12T13:22:36.180768Z lvl=info msg="Closing retention policy enforcement service" log_id=0HDUclgW000 service=retention
ts=2019-08-12T13:23:06.180646Z lvl=info msg="Time limit reached, initializing hard shutdown" log_id=0HDUclgW000
ts=2019-08-12T15:51:23.695751Z lvl=info msg="InfluxDB starting" log_id=0HDdHBBl000 version=1.7.6 branch=1.7 commit=01c8dd416270f424ab0c40f9291e269ac6921964
ts=2019-08-12T15:51:23.695805Z lvl=info msg="Go runtime" log_id=0HDdHBBl000 version=go1.11 maxprocs=12
ts=2019-08-12T15:51:23.797100Z lvl=info msg="Using data dir" log_id=0HDdHBBl000 service=store path=/var/lib/influxdb/data
ts=2019-08-12T15:51:23.797157Z lvl=info msg="Compaction settings" log_id=0HDdHBBl000 service=store max_concurrent_compactions=6 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
ts=2019-08-12T15:51:23.797177Z lvl=info msg="Open store (start)" log_id=0HDdHBBl000 service=store trace_id=0HDdHBaG000 op_name=tsdb_open op_event=start
ts=2019-08-12T15:51:24.769362Z lvl=info msg="Reading file" log_id=0HDdHBBl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/627/_00075.wal size=7335186
ts=2019-08-12T15:51:27.164307Z lvl=info msg="Reading file" log_id=0HDdHBBl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/collectd/autogen/626/_00157.wal size=10503321
ts=2019-08-12T15:51:27.300815Z lvl=info msg="Reading file" log_id=0HDdHBBl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/collectd/autogen/91/_00004.wal size=657
ts=2019-08-12T15:51:28.015468Z lvl=info msg="Reading file" log_id=0HDdHBBl000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/collectd/autogen/626/_00158.wal size=839971
ts=2019-08-12T15:51:28.327586Z lvl=info msg="Open store (end)" log_id=0HDdHBBl000 service=store trace_id=0HDdHBaG000 op_name=tsdb_open op_event=end op_elapsed=4530.406ms
ts=2019-08-12T15:51:28.327703Z lvl=info msg="Opened service" log_id=0HDdHBBl000 service=subscriber
ts=2019-08-12T15:51:28.327719Z lvl=info msg="Starting monitor service" log_id=0HDdHBBl000 service=monitor
ts=2019-08-12T15:51:28.327734Z lvl=info msg="Registered diagnostics client" log_id=0HDdHBBl000 service=monitor name=build
ts=2019-08-12T15:51:28.327745Z lvl=info msg="Registered diagnostics client" log_id=0HDdHBBl000 service=monitor name=runtime
ts=2019-08-12T15:51:28.327763Z lvl=info msg="Registered diagnostics client" log_id=0HDdHBBl000 service=monitor name=network
ts=2019-08-12T15:51:28.327775Z lvl=info msg="Registered diagnostics client" log_id=0HDdHBBl000 service=monitor name=system
ts=2019-08-12T15:51:28.327824Z lvl=info msg="Starting snapshot service" log_id=0HDdHBBl000 service=snapshot
ts=2019-08-12T15:51:28.327838Z lvl=info msg="Starting continuous query service" log_id=0HDdHBBl000 service=continuous_querier
ts=2019-08-12T15:51:28.327860Z lvl=info msg="Starting HTTP service" log_id=0HDdHBBl000 service=httpd authentication=false
ts=2019-08-12T15:51:28.327871Z lvl=info msg="opened HTTP access log" log_id=0HDdHBBl000 service=httpd path=stderr
ts=2019-08-12T15:51:28.327893Z lvl=info msg="Storing statistics" log_id=0HDdHBBl000 service=monitor db_instance=_internal db_rp=monitor interval=10s
ts=2019-08-12T15:51:28.328049Z lvl=info msg="Listening on HTTP" log_id=0HDdHBBl000 service=httpd addr=[::]:8086 https=false
ts=2019-08-12T15:51:28.328094Z lvl=info msg="Starting retention policy enforcement service" log_id=0HDdHBBl000 service=retention check_interval=30m
ts=2019-08-12T15:51:28.328117Z lvl=info msg="Starting collectd service" log_id=0HDdHBBl000 service=collectd
ts=2019-08-12T15:51:28.328281Z lvl=info msg="Loading types from file" log_id=0HDdHBBl000 service=collectd path=/usr/share/collectd/softel_types.db
ts=2019-08-12T15:51:28.329132Z lvl=info msg="Loading types from file" log_id=0HDdHBBl000 service=collectd path=/usr/share/collectd/types.db
ts=2019-08-12T15:51:28.330794Z lvl=info msg="Loading types from file" log_id=0HDdHBBl000 service=collectd path=/usr/share/collectd/vmware-types.db
ts=2019-08-12T15:51:28.332279Z lvl=info msg="Loading types from file" log_id=0HDdHBBl000 service=collectd path=/usr/share/collectd/zfs.db
ts=2019-08-12T15:51:28.332567Z lvl=info msg="Listening on UDP" log_id=0HDdHBBl000 service=collectd addr=10.1.1.1:25827
ts=2019-08-12T15:51:28.332949Z lvl=info msg="Listening for signals" log_id=0HDdHBBl000
I don’t understand if this is good or bad…
> sudo influx_inspect reporttsi -db-path /var/lib/influxdb/
reporttsi: open /var/lib/influxdb/wal/collectd/autogen/546/index: no such file or directory
> sudo influx_inspect reporttsi -db-path /var/lib/influxdb/
reporttsi: open /var/lib/influxdb/wal/_internal/monitor/623/index: no such file or directory
> sudo influx_inspect reporttsi -db-path /var/lib/influxdb/
reporttsi: open /var/lib/influxdb/wal/_internal/monitor/623/index: no such file or directory
> sudo influx_inspect reporttsi -db-path /var/lib/influxdb/
reporttsi: open /var/lib/influxdb/wal/collectd/autogen/259/index: no such file or directory