I am running (Telegraf 1.1.0) , (Influxdb 1.1.0) and Grafana 3.0 (soon will run kapacitor) on a RHEL 7.0 server , and I ran into a 100% disk utilization issue because of the file /var/log/messages .
Seems like telegraf , influxdb and grafana are all logging their events to this file, resulting in a large file size quickly.
How do I direct the logs somewhere else ?
And what is the difference between info logged in /var/log/messages and /var/log/influxdb/ for example?
Thank you
@randomeleven Hello and welcome!
This is a new one for me. To my knowlage InfluxDB, Telegraf and Grafana all log to /var/log/{{ .product }}. What kind of information is in the /var/log/messages file?
@jackzampolin The file contains event logs by influxdb, telegraf and grafana, like when a batch of points has been been written by telegraf, a point is arriving to influxdb via http or typical grafana query activity. I’ll give an example:
Apr 20 10:03:32 my-server-host-name influxd: [query] 2017/04/20 10:03:32 SELECT mean(value) FROM db.rp.measure WHERE time > now() - 1h GROUP BY time(30s)
Apr 20 10:03:32 my-server-host-name influxd: [query] 2017/04/20 10:03:32 SELECT mean(value2) FROM db.rp.measure WHERE time > now() - 1h GROUP BY time(30s)
Apr 20 10:03:32 my-server-host-name influxd: [httpd] ::1 - influx_user [20/Apr/2017:10:03:32 +0400] "GET /query?db=db&epoch=ms&q=SELECT+mean....etc
Apr 20 10:03:32 my-server-host-name influxd: [httpd] ::1 - influx_user [20/Apr/2017:10:03:32 +0400] "GET /query?db=db&epoch=ms&q=SELECT+mean%......etc
Apr 20 10:03:37 my-server-host-name influxd: [httpd] X.X.X.X - influx_user2 [20/Apr/2017:10:03:37 +0400] "POST /write?db=db&p=%5BREDACTED%5D&precision=ms&rp=rp&u=influx_user2 HTTP/1.1" 204 0 "-" "python-requests/2.11.0" 1864a5ab-258f-11e7-8309-000000000000 18886
Apr 20 10:03:41 my-server-host-name telegraf: 2017/04/20 10:03:41 I! Output [influxdb] buffer fullness: 30 / 10000 metrics. Total gathered metrics: 35263075. Total dropped metrics: 0.
Apr 20 10:03:41 my-server-host-name telegraf: 2017/04/20 10:03:41 I! Output [influxdb] wrote batch of 30 metrics in 223.127µs
Apr 20 10:04:06 my-server-host-name influxd: [httpd] X.X.X.X - influx_user2 [20/Apr/2017:10:04:06 +0400] "POST /write?db=db&p=%5BREDACTED%5D&precision=ms&rp=rp&u=influx_user2 HTTP/1.1" 204 0 "-" "python-requests/2.4.3 CPython/2.7.9 Linux/3.16.0-4-amd64" 29b25ca8-258f-11e7-83fc-000000000000 8499
Apr 20 10:04:06 my-server-host-name influxd: [httpd] X.X.X.X - influx_user2 [20/Apr/2017:10:04:06 +0400] "POST /write?db=db&p=%5BREDACTED%5D&precision=ms&rp=rp&u=influx_user2 HTTP/1.1" 204 0 "-" "python-requests/2.4.3 CPython/2.7.9 Linux/3.16.0-4-amd64" 29b33168-258f-11e7-83fd-000000000000 10269
......
......
more of the same
What’s alarming is that the file is growing in size by about 0.5GB / 24 hours.
I am thinking about purging these logs after having directed them away from /var/log/messages
@randomeleven You can reduce the frequency on both the influxdb and telegraf instances:
# /etc/influxdb/influxdb.conf
[httpd]
log-enabled = false
# /etc/telegraf/telegraf.conf
[agent]
quiet = true
This also sounds like a bug. I’ve never seen these logs going to /var/log/messages. Would you mind opening an issue about that on both influxdb and telegraf?
