The disk on the AWS server hosting our influxd filled up and I’m now trying to resurrect the system. Since the web interface doesn’t connect, I’ve logged into the server directly and I’m trying to communicate with influxd via the command line.
what I’ve tried
web interface
Attempting to connect via the web interface gives a “refused to connect”. So instead, I’ve logged into the server directly.
influx command line interface
I can see that there is an influxd running on the system:
My attempt to use the influx app doesn’t work (although the .conf file does specify port 8086):
# ./influx
Failed to connect to http://localhost:8086
Please check your connection settings and ensure 'influxd' is running.
curl
cURL fares better, but I’m troubled by the empty response – there should be more than zero databases in the system. Am I structuring the query properly?
$ curl -G "http://localhost:8086/query" -u $user:$pass --trace-ascii /tmp/dump.txt --data-urlencode "q=SHOW DATABASES"
== Info: Hostname was NOT found in DNS cache
== Info: Trying ::1...
== Info: Connected to localhost (::1) port 8086 (#0)
== Info: Server auth using Basic with user 'aquadyne'
=> Send header, 169 bytes (0xa9)
0000: GET /query?q=SHOW%20DATABASES HTTP/1.1
0028: Authorization: Basic YXF1YWR5bmU6ZWpsb0tMeThkSUsxMExxdHdkeW5SeA=
0068: =
006b: User-Agent: curl/7.35.0
0084: Host: localhost:8086
009a: Accept: */*
00a7:
== Info: Empty reply from server
== Info: Connection #0 to host localhost left intact
curl: (52) Empty reply from server
my real goal
Ultimately, I’d like to prune old entries from the database(s) – whoever set up the system didn’t implement any retention policy, which is one reason the disk filled up.
But to do that, I need to perform basic queries to the db. Is there something obvious I’m missing?
I realized that the server uses docker containers, so the following command succeeded in getting me into the CLI:
$ docker exec -i rancher_influxdb_1 influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 1.0.2
InfluxDB shell version: 1.0.2
>
I can sympathize with you on the "disk full’ stuff and how hard it can be to manage. Mine happened on my local machine (Pro Tip: don’t set nanosecond accuracy µSec data gathering in your telegraf.conf file or your disk will fill up PRONTO! ).
Here’s what I ultimately had to do, which may not be the ‘recommended’ way, but with a disk at 100%, sometimes you have to take drastic measures.
If you can, try using the CLI DROP SHARD command to drop older shards from the database. This should clean things up for you.
In my case, with my disk at 100% and InfluxDB running, I couldn’t do that. I had to take much more drastic measures. THESE CAN RUIN THINGS!
First, I stopped InfluxDB and Telegraf to make sure I wasn’t adding to the problem I was trying to solve. Next, I went into /var/lib/influxdb/data/telegraf/autogen – that was the offending database in my case – and ran ls -l so I could see which shards were the oldest. I then deleted those shards. (I also deleted a few of the newest ones because of that little mishap mentioned above.)
Once my disk was at a reasonable %free, I restarted everything and was able to continue with the CLI and DROP SHARD commands to fix things up.
This is potentially the best news I’ve heard all day!
Are you saying that I can delete shard files at the unix filesystem level (i.e. rm ...)? Or are you only using ls -l to find the oldest shards and then calling drop shard ... within the influx app?
I am saying that if possible, use the DROP SHARD on the oldest shards. If that is not possible, you can delete the files directly at the unix level with rm -f ...BUT keep in mind my caveat above. InfluxDB should not be running at the time, and it has the potential to do Very Bad Things™ to your system. I have done it successfully, but with such drastic sledgehammer tactics there is always room for calamity.
