I recently started using the TICK stack to monitor a set of servers, as part of my testing i wanted to simulate restoring the database from a backup. I cloned the VM that runs InfluxDB and attempted to restore both the meta data and the database. During the restore neither the metadata restore or the db restore output any errors during the restore, however upon starting InfluxDB it seems that my hosts can no longer write or view new metrics in the database. (Utilizing Grafana and Chronograf, both can see the old metrics in the database so viewing old data at least works.) The Telegraf clients don’t output any errors about writing the cluster and when viewing the log from tail -f /var/log/syslog. I can see that the cluster is receiving data and is also receiving the request for viewing metrics.
(Here is 1 log cycle):
Jan 31 15:34:20 debian-monitoring influxd[1469]: [httpd] 10.0.0.232 - burning-monitoring [31/Jan/2018:15:34:20 -0700] "POST /write?consistency=any&db=telegraf_burning HTTP/1.1" 204 0 "-" "-" e10b61c4-06d6-11e8-820b-000000000000 8253
Jan 31 15:34:20 debian-monitoring influxd[1469]: [httpd] 127.0.0.1 - burning-monitoring [31/Jan/2018:15:34:20 -0700] "POST /write?consistency=any&db=telegraf_burning HTTP/1.1" 204 0 "-" "-" e14bf1b5-06d6-11e8-820c-000000000000 24629
Jan 31 15:34:22 debian-monitoring chronograf[390]: time="2018-01-31T15:34:22-07:00" level=info msg="Response: OK" component=server method=GET remote_addr="10.0.0.232:58312" response_time="100.824µs" status=200
Jan 31 15:34:22 debian-monitoring influxd[1469]: [query] 2018/01/31 15:34:22 SHOW DATABASES
Jan 31 15:34:22 debian-monitoring influxd[1469]: [httpd] ::1 - admin [31/Jan/2018:15:34:22 -0700] "POST /query?db=&epoch=ms&q=SHOW+DATABASES&rp= HTTP/1.1" 200 142 "-" "Go-http-client/1.1" e28bf51d-06d6-11e8-820d-000000000000 1144
Jan 31 15:34:22 debian-monitoring chronograf[390]: time="2018-01-31T15:34:22-07:00" level=info msg="Response: OK" component=server method=POST remote_addr="10.0.0.232:58312" response_time=2.555145ms status=200
Information on how I am backing up the databases and restoring them:
Backup commands used:
influxd backup /backup/influxdb/metastore/"$(/bin/date +\%Y-\%m-\%d)"
influxd backup --database telegraf_burning /backup/influxdb/telegraf-burning/"$(/bin/date +\%Y-\%m-\%d)"
Restore commands used:
systemctl stop influxdb.service
influxd restore -metadir /var/lib/influxdb/meta /backup/influxdb/metastore/2018-01-31
influxd restore -database telegraf_burning -datadir /var/lib/influxdb/data /backup/influxdb/telegraf-burning/2018-01-31
chown -R influxdb:influxdb /var/lib/influxdb
systemctl start influxdb.service
Now for the head scratcher, when I tested the above restore process on a clean install to simulate rebuilding the entire host from database backups the database works completely fine. Hosts are able to write to it, i can see the metrics with Grafana and Chronograf, all of the previous data is still intact.
The InfluxDB config files are identical with each other since they are both from the same template file. (Hooray configuration management).
Both systems are on InluxDB version 1.0.2 on Debian 9.
EDIT Upgrading the host to InfluxDB version 1.4.3 fixed the problem.
