[[inputs.logparser]] has no output

Telegraf
1

I’ve been running in circles with this for a while and I can’t seem to find anything on forums that help much.

I’m trying to parse through the /var/log/auth.log file to extract different auth logging. Elastic even has a Blog post about this: Grokking the Linux authorization logs | Elastic Blog

Everything seems to work in https://grokdebug.herokuapp.com/ but when I input everything into telegraf I receive no results. Anybody have any ideas?

Confirmed telegraf as access to the adm group which has access to the logs.

logparser config setting for telegraf:

[[inputs.logparser]]
  files = ["/var/log/auth.log"]
  from_beginning = false
  watch_method = "inotify"
  [inputs.logparser.grok]
    patterns = ['''
        %{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sshd(?:\[%{POSINT:system.auth.pid}\])?: %{DATA:system.auth.ssh.event} user %{DATA:system.auth.user} from %{IPORHOST:system.auth.ip}
    ''']
    measurement = "auth_log"

Example Logs:

Apr 10 05:11:57 localhost sshd[22041]: Invalid user frank from 172.31.14.87
Apr 10 05:11:57 localhost sshd[22041]: input_userauth_request: invalid user frank [preauth]
Apr 10 05:17:38 localhost sshd[33668]: Invalid user frank from 172.31.14.87
Apr 10 05:17:38 localhost sshd[33668]: input_userauth_request: invalid user frank [preauth]
Apr 10 05:18:36 localhost sshd[35700]: Invalid user frank from 172.31.14.87
Apr 10 05:18:36 localhost sshd[35700]: input_userauth_request: invalid user frank [preauth]

2

I tried your pattern and I think it is a bug, can you open a new issue on the Telegraf github page? I just started looking into it and it may help to enter the pattern on a single line:

patterns = ['''%{SYSLOGTIMESTAMP:syst...''']
3

Thanks @daniel for the quick reply!

I updated my patterns to include it all in one line and still no luck. I’ve opened a new Issue here: Logparser captures cannot contain non-word characters · Issue #4003 · influxdata/telegraf · GitHub

Let me know if I missed anything. And your help is greatly appreciated.

[[inputs.logparser]]
    files = ["/var/log/auth.log"]
    from_beginning = false
    watch_method = "inotify"
    [inputs.logparser.grok]
      patterns = ['''%{SYSLOGTIMESTAMP:system.auth.timestamp} %{SYSLOGHOST:system.auth.hostname} sshd(?:\[%{POSINT:system.auth.pid}\])?: %{DATA:system.auth.ssh.event} user %{DATA:system.auth.user} from %{IPORHOST:system.auth.ip}''']
      measurement = "auth_log"
4

There also seems to be an issue with using a . in the field name, can you try with fields like: %{SYSLOGTIMESTAMP:system_auth_timestamp}

5

Yes! That did it!

Thank you @daniel!

So it seems there are two issues at play? One with putting the pattern on it’s own line? And another with periods in the field name?

I can try a couple of different scenario’s tomorrow now knowing this (sitting in CET timezone here).

Working config:

[[inputs.logparser]]
    files = ["/var/log/auth.log"]
    from_beginning = false
    watch_method = "inotify"
    [inputs.logparser.grok]
      patterns = ['''%{SYSLOGTIMESTAMP:system_auth_timestamp} %{SYSLOGHOST:system_auth_hostname} sshd(?:\[%{POSINT:system_auth_pid}\])?: %{DATA:system_auth_ssh_event} user %{DATA:system_auth_user} from %{IPORHOST:system_auth_ip}''']
      measurement = "auth_log"
6

Yes there are two issues, the newline pattern bug should be easy to fix so I’ll add to 1.6. It also looks like support for periods was added recently to the grok library, so I’ll look into including this in the next feature release (1.7).

7

This issue specifically tracks the newline in the patterns string: