We are currently trying to use Telegraf tail input plugin to collect Nginx access logs in Linux VM and convert to metrics to be consumed by downstream components based on InfluxDb line protocol. Nginx will put access logs in a specific location, where tail plugin can read logs line by line. However, when service needs to restart maybe due to a deployment, tail plugin will read all logs again which causes huge duplicates. Is there a way that it can keep a persistent record of last read position? Or what is the suggested way to tail logs in this scenario?
I was thinking logparser plugin has been deprecated and moved to tail plugin. Basically we are trying to figure out if there is a way to read logs line by line and keep going forward regardless of service restart.
If you are using
from_beginning = true you will probably want to disable it, this way Telegraf will pick up from the end of the logfile when restarted. It isn’t currently possible to store the offsets when restarting, though it is something we would like to add, but you can often avoid restarting Telegraf by using SIGHUP to reload which does keep track of the offsets.