Output single file per tag

Telegraf
1

Hello,

My telegraf is feeded by an nginx access log. Now, I need to store access log per hostname. I have the hostname in a field, but can’t anticipate the list of all hostnames.

Is there a way to use output.file with a field/tag as “variable” on the “files” parameter?

Thanks in advance.

2

If you don’t already know the hostnames, then no. Your best option would be to use the outptus.exec output to write your own script that writes the output to the specific filename based on the host tag.

3

Thank you for your help. I’ve written a short LUA script to achieve different file output per field/tag, and I piped JSON to this LUA script using outputs like this:

[[outputs.execd]]
  command = ["/usr/bin/lua", "/etc/telegraf/telegraf.d/splitlog.lua"]
  data_format = "json"
  namepass = ["accesslog"]

The LUA script (very minimalist):

#!/usr/bin/lua

local cjson = require "cjson"

local hostname = "_"

while true
do
    local line = io.read()
    local data = cjson.decode(line)
    if data.fields.host then
        hostname = data.fields.host
    end

    local filepath = "/var/log/nginx/accesslogs/" .. hostname .. ".json"
    local file = io.open(filepath, "a")
    file:write(line .. "\n")
    file:close()
end

This seems to work fine even on high loads that [[outputs.mongodb]] hasn’t been able to handle (probably more about MongoDB performance than telegraf issue).

1 Like