From Laravel to Chronograph Log Viewer - Date Parsing Issue

InfluxDB 2
, , , ,
1

Hello there,

I’m trying to use the Log Viewer from Chronograph to view my custom app logs.
I’m aware that the datas sent to InfluxDb needs to be in syslog format to be shown in the UI.

The logs are written in a specific parsable format by our Laravel app, and look like this :

syslog,appname=Laravel,facility=local,host=myserver,hostname=myserver.domain.tld,severity=ERROR,facility_code=1,message=" - Log automatically generated by artisan command - RND/5438",procid=8164,severity_code=0,timestamp=1552407620.135900010,version=1
syslog,appname=Laravel,facility=local,host=myserver,hostname=myserver.domain.tld,severity=WARNING,facility_code=1,message=" - Log automatically generated by artisan command - RND/7010",procid=8164,severity_code=0,timestamp=1552407620.135900020,version=1
syslog,appname=Laravel,facility=local,host=myserver,hostname=myserver.domain.tld,severity=NOTICE,facility_code=1,message=" - Log automatically generated by artisan command - RND/3720",procid=8164,severity_code=0,timestamp=1552407620.135900030,version=1
syslog,appname=Laravel,facility=local,host=myserver,hostname=myserver.domain.tld,severity=INFO,facility_code=1,message=" - Log automatically generated by artisan command - RND/7325",procid=8164,severity_code=0,timestamp=1552407620.135900040,version=1
syslog,appname=Laravel,facility=local,host=myserver,hostname=myserver.domain.tld,severity=DEBUG,facility_code=1,message=" - Log automatically generated by artisan command - RND/6260",procid=8164,severity_code=0,timestamp=1552407620.135900050,version=1

These logs are parsed by Telegraf, thanks to this configuration :

[[inputs.logparser]]
  files = [ "/path/to/my/log/file.log" ]
  from_beginning = false

  [inputs.logparser.grok]
    patterns = [ '%{CHANNEL},appname=%{APPNAME},facility=%{FACILITY},host=%{HOST},hostname=%{HOST_NAME},severity=%{SEVERITY},facility_code=%{FACILITY_CODE},message="%{MESSAGE}",procid=%{PROCID},severity_code=%{SEVERITY_CODE},timestamp=%{TSTAMP},version=%{VERSION}' ]
    measurement = "syslog"
    custom_patterns = '''
      APPNAME %{DATA:appname}
      CHANNEL %{DATA:channel}
      FACILITY %{DATA:facility}
      FACILITY_CODE %{NUMBER:facility_code}
      HOST %{DATA:host}
      HOST_NAME %{DATA:hostname}
      MESSAGE %{DATA:message}
      PROCID %{DATA:procid}
      SEVERITY %{DATA:severity}
      SEVERITY_CODE %{NUMBER:severity_code}
      TSTAMP %{DATA:timestamp:ts-epoch}
      VERSION %{NUMBER:version}
    '''
    timezone = "Europe/Paris"

When I load some new lines into my log file, I see them processed in the telegraf logs and no errors showing.

When I check datas sent to InfluxDb, into “telegraf.syslog” measurement set I see this :

> select time,appname,channel,facility,message,procid,severity,severity_code,timestamp,version from syslog
name: syslog
time                appname channel facility message                                                      procid severity  severity_code timestamp version
----                ------- ------- -------- -------                                                      ------ --------  ------------- --------- -------
1552409073203600000 Laravel syslog  local     - Log automatically generated by artisan command - RND/9846 9512   EMERGENCY 0                       1
1552409073203600001 Laravel syslog  local     - Log automatically generated by artisan command - RND/2824 9512   ALERT     0                       1
1552409073203600002 Laravel syslog  local     - Log automatically generated by artisan command - RND/5117 9512   CRITICAL  0                       1
1552409073203600003 Laravel syslog  local     - Log automatically generated by artisan command - RND/6222 9512   ERROR     0                       1
1552409073203600004 Laravel syslog  local     - Log automatically generated by artisan command - RND/1447 9512   WARNING   0                       1
1552409073203600005 Laravel syslog  local     - Log automatically generated by artisan command - RND/192  9512   NOTICE    0                       1
1552409073203600006 Laravel syslog  local     - Log automatically generated by artisan command - RND/8061 9512   INFO      0                       1
1552409073203600007 Laravel syslog  local     - Log automatically generated by artisan command - RND/7288 9512   DEBUG     0                       1

The timestamp column is empty.

I can see those lines in Chronograf’s Log Viewer with “Timestamp” set to “1970-01-01 01:00:00”.

30
301492×404 87.7 KB

Can you tell me what I am doing wrong ? Struggling with this since this morning, I’ve tried many formats implemented (timestamp) but none worked.

Original inspiration : Writing Logs Directly to InfluxDB - DZone
Documentation used for Grok : docs.influxdata.com-ARCHIVE/grok.md at master · influxdata/docs.influxdata.com-ARCHIVE · GitHub

Thanks in advance for your help

3

Hi , it seems to be an open issue …
there is a workaround in this link …
Epoch with decimal

In combination with the default
unique_timestamp = “auto”
It should work for you ?

4

Hello MarcV,

Thanks for your reply. I’ve tried the workaround presented in the issue.

Now my configuration looks like this :

[[inputs.logparser]]
  files = [ "/path/to/my/log/file.log" ]
  from_beginning = false

  [inputs.logparser.grok]
    patterns = [ '%{CHANNEL},appname=%{APPNAME},facility=%{FACILITY},host=%{HOST},hostname=%{HOST_NAME},severity=%{SEVERITY},facility_code=%{FACILITY_CODE},message="%{MESSAGE}",procid=%{PROCID},severity_code=%{SEVERITY_CODE},timestamp=%{TSTAMP},version=%{VERSION}' ]
    measurement = "syslog"
    custom_patterns = '''
      APPNAME %{DATA:appname}
      CHANNEL %{DATA:channel}
      FACILITY %{DATA:facility}
      FACILITY_CODE %{NUMBER:facility_code}
      HOST %{DATA:host}
      HOST_NAME %{DATA:hostname}
      MESSAGE %{DATA:message}
      PROCID %{DATA:procid}
      SEVERITY %{DATA:severity}
      SEVERITY_CODE %{NUMBER:severity_code}
      TSTAMP %{NUMBER:timestamp:ts-epoch}.%{NUMBER}
      VERSION %{NUMBER:version}
    '''
    timezone = "Europe/Paris"
    unique_timestamp = "auto"

I generated more logs, looking same as in my original message.
Once processed, we can see the timestamp is still empty.

> select time,appname,channel,facility,message,procid,severity,severity_code,timestamp,version from syslog
name: syslog
time                appname channel facility message                                                      procid severity  severity_code timestamp version
----                ------- ------- -------- -------                                                      ------ --------  ------------- --------- -------
1552466276811300000 Laravel syslog  local     - Log automatically generated by artisan command - RND/1313 61609  EMERGENCY 0                       1
1552466276811300001 Laravel syslog  local     - Log automatically generated by artisan command - RND/4375 61609  ALERT     0                       1
1552466276811300002 Laravel syslog  local     - Log automatically generated by artisan command - RND/1200 61609  CRITICAL  0                       1
1552466276811300003 Laravel syslog  local     - Log automatically generated by artisan command - RND/1506 61609  ERROR     0                       1
1552466276811300004 Laravel syslog  local     - Log automatically generated by artisan command - RND/751  61609  WARNING   0                       1
1552466276811300005 Laravel syslog  local     - Log automatically generated by artisan command - RND/7899 61609  NOTICE    0                       1
1552466276811300006 Laravel syslog  local     - Log automatically generated by artisan command - RND/6686 61609  INFO      0                       1
1552466276811300007 Laravel syslog  local     - Log automatically generated by artisan command - RND/8318 61609  DEBUG     0                       1

It seems that this workaround is not working.

Do you have any more ideas about this?

5

I think you have to set that workaround in the patterns ?

6

What do you mean ?

patterns = [ '....%{TSTAMP}....' ]
custom_patterns = TSTAMP %{NUMBER:timestamp:ts-epoch}.%{NUMBER}

On my opinion, it should be working. Do you think I have to move this rule in the main pattern like this ?

patterns = [ '....%{TSTAMP}.%{NUMBER}....']
custom_patterns = TSTAMP %{NUMBER:timestamp:ts-epoch}