InfluxDB Output Error: Response Error: Status Code [500], expected [204] any ideas what this means?

1

Hi all,

I am recieving this error:

2018-07-05T08:13:10Z E! InfluxDB Output Error: Response Error: Status Code [500], expected [204], [engine: error writing WAL entry: write /var/lib/influxdb/wal/telegraf/autogen/221/_00001.wal: no space left on device]

I am consuming all the space in this area it looks like, but what woudl cause this and how do I remedy it? I am collecting snmp information from cisco switches using the following MIBS

RFC1213-MIB::sysName.0
IF-MIB::ifXTable
IF-MIB::ifName

I am pretty new to all this and my influxdb and telegraf tend to use most of the standard configs. I guess my question is am i collecting too much information? should the sampling rate be changed or should I ask for more space?

I am not sure how im filling it up so quick as im trying to capture from maybe 20 switches

If someone can explain whats going on that would be great!

Thanks

Rob

2

Assuming you are running influx and telegraf on linux.

First, run:

sudo df -h

or

sudo df -h /var/lib/influxdb

to see disk usage

In the telegraf config on the server look for the section

# # Send telegraf metrics to file(s)
# [[outputs.file]]
#   ## Files to write to, "stdout" is a specially handled file.
#   files = ["stdout", "/tmp/metrics.out"]
#
#   ## Data format to output.
#   ## Each data format has it's own unique set of configuration options, read
#   ## more about them here:
#   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
#   data_format = "influx"

If it isn’t commented out, then do so. This file can grow quite large. I would recommend deleting it unless you need it.

What retention policy are you using?

If you have gone through the set up without changing much i suspect you will be using autogen which i think comes with an infinite retention policy, which as you can imagine means all your data is still stored there.

You can check from the InfluxDB CLI:

influx -username -password

from there

USE < DATABASE NAME >

SHOW RETENTION POLICIES

Unless you have created a new RP already there should just be ‘autogen’ and the ‘default’ column would be true.

You can create a new RP for 30 days or a duration of your choosing and apply it as the default RP. Then all new metrics will go in there and only be stored for as long as you specify.

The following link should help explain how to change the retention policy.

Retention Policy Management

Then, you will have to either move your old data or delete it.

If it is important that you retain all the information you collect then the option is more disk.

Hope that helps

1 Like
3

Philb,

thanks for this its really helpful!

I am using the default retention policy so I will look to change that.

thanks again

Rob

4

No probs Roberto. I ran in to the same issue when i started out with Influx. I’d overlooked the retention policies and shard durations until it came back to bite me in the backside.

PhilB