The way I see it, I can use one of file, tail, or multifile.
If I use tail, how would that work if I have to restart the Telegraf agent at any point? Will it pick up where it left off, or will it simply start tailing again (and missing the part during the restart)?
If I use file or multifile, I see that it reads the entire file contents every time? Does this have any performance issues with large files like access logs? What about duplicate entries, is it smart enough just enter new data?
And finally, how do either of them work with logrotate?
You will want to use the tail plugin, the other two plugins don’t work well for log data because they parse the full file each interval.
When restarting Telegraf you can set the from_beginning option to true and the file will be reparsed from the start of the file, or if this is false the parsing will start at the end of the file and would potentially miss new lines added during the restart. If you do use from_beginning, it will rely on the property of InfluxDB to only record a single value per measurement+tags+field+timestamp to avoid duplicates, but it still can be quite expensive if you have a large log file. Instead of restarting, you can reload Telegraf by sending a SIGHUP and it will pick up at the position it left off when from_beginning = false.
The plugin only follows files that matches the files pattern, if the file is renamed so that it no longer matches the pattern then it will stop reading the file as soon as it notices the rename. This means it is possible for a rotated file to be removed before it is fully read, we have an open issue for this. You might be able to list both the current and last file to avoid this, for example:
