Telegraf: Ruby on Rails Log Parse

Telegraf
,
1

Hi, so I’m having trouble getting some Ruby on Rails logs into InfluxDB using Telegraf. At one point I had something working for the first portion of log entry… and when I tried to add to it, it broke… and I’ve never since been able to get it back to even working for the piece that did work before.

Here’s an example line from the log:
I, [2017-08-14T13:47:07.245469 #2180] INFO -- : Completed 500 Internal Server Error in 43ms (Views: 0.3ms | ActiveRecord: 8.6ms)

Here’s the config I thought worked at first to just capture the Loglevel/severity, Timestamp, and PID…

[[inputs.logparser]]
   files = ["/opt/applications/core/current/log/staging.log"]
   [inputs.logparser.grok]
     patterns = ["%{CUSTOM_CORE_LOG}"]
     measurement = "app_core_log"
     custom_patterns = '''
       CUSTOM_CORE_LOG %{LOGLEVEL:sev:tag}, \[%{TIMESTAMP_ISO8601:timestamp:ts-"2017-08-14T13:47:07.201998"} \#%{POSINT:pid:int}\]
     '''

So in my mind, that should match I, [2017-08-14T13:47:07.245469 #2180], resulting in sev=“I”, ts=2017-08-14T13:47:07.245469, and pid=2180.

I’m at the point where I can’t even just match the stinkin’ Loglevel… so even a patter of simply %{LOGLEVEL:sev:tag} doesn’t work for me right now. Even if I don’t make sev a tag.

I need to parse through the whole line, but if I could just get going with at least the first three components, I can hopefully get the rest working… Any help would be greatly appreciated! I’m sure it’s something stupid…

Thanks!

2

I see a couple issues, the first is that LOGLEVEL can not parse a single letter level. You could define a custom pattern for it and assign it to the tag, or it might be nicer to capture the longer INFO after the date for this, which is what I did in the example below.

The next issue is that in the date you need to use the “reference date” which is defined as exactly Mon Jan 2 15:04:05 -0700 MST 2006.

I believe this will parse up to through the INFO token, hope it is helpful and good luck with the rest of the pattern:

[inputs.logparser.grok]
  patterns = ["%{CUSTOM_CORE_LOG}"]
  custom_patterns = '''
    CUSTOM_CORE_LOG %{LEVEL}, \[%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02T15:04:05.000000"} #%{POSINT:pid:int}\] %{LOGLEVEL:sev:tag}
    LEVEL (?:[IWDE])
  '''
3

Ahhhhhh didn’t realize a specific “reference” date had to be used. I have made it past those “woes”… thanks so much!!!