Hi guys, the first post for me here 
I am having trouble ingesting a log using Telegraf logparse, specifically regarding the date, it is something like MM/dd/yyyy hh:mm:ss AM/PM.
I can not make it work with any of the supposed ways, any help, please? The idea is to use that date as a timestamp.
Hi @jorgedlcruz welcome !
Can you share a part of your logfile ?
Best regards
Hello, thanks for replying, sure:
5/31/2019 11:40:26 AM 9 (5176) Root folder opened successfully
5/31/2019 11:40:26 AM 9 (5176) Exchange Web Services initialized.
5/31/2019 11:40:26 AM 9 (5176) Counting items...
5/31/2019 11:40:26 AM 9 (5176) found 7 item(s).
5/31/2019 11:40:26 AM 9 (5176) Restoring items...
5/31/2019 11:40:26 AM 9 (5176) Initializing Exchange Web Services...Jonge , I will have a look in one hour or so
can you post your telegraf config for the logparse part ?
Hello, sure, I’ve tried with this:
[inputs.logparser.grok]
patterns = ['%{DATESTAMP:timestamp} %{WORD} %{SPACE} %{NUMBER} \(%{NUMBER}\) %{GREEDYDATA:message}']
And more sophisticated ways like:
%{DATE_US:timestamp} %{TIME:time} %{WORD} %{SPACE} %{NUMBER} \(%{NUMBER}\) \s*Creating %{GREEDYDATA} %{NUMBER:bytesrestored}
Mixing both DATEUS and time on one custom pattern, etc, but nothing, have you seen this before? It is the date with I have problems, the rest is fine. I couldn’t find a way to parse the whole date on a string Telegraf will understand I’ve tried with the ts-custom but nothing.
the idea is to use that date as the timestamp of the datapoints … ? or as an extra field ?
Hello,
Thanks for replying, timestamp so I can later on combine them and show them with Grafana, by using the timestamp on the dashboard on the top right. I can get it as a string, or as a tag, but then I will lose that ability to group them by time, etc.
Plus, because not really proper timestamp, every time I restart telegraf it does duplicates all.
Jorge ,
so far I have found this :
this is my inputs.logparser
[[inputs.logparser]]
files = ["/media/sf_software/influxdb/telegraf/parser.test"]
[inputs.logparser.grok]
patterns = ["%{DATESTAMP_AMPM:timestamp} value=%{NUMBER:value:int}"]
custom_pattern_files = ["/media/sf_software/influxdb/telegraf/patterns.conf"]
measurement = "parser"
unique_timestamp = "auto"
as you can see I used custom_patterns_file but you can also use
[inputs.logparser.grok]
patterns = ["%{DATESTAMP_AMPM:timestamp} value=%{NUMBER:value:int}"]
custom_patterns = "DATESTAMP_AMPM %{DATESTAMP} (AM|PM)"
, the content of my custom_patterns_file is :
DATESTAMP_AMPM %{DATESTAMP} (AM|PM)
with that combination I succeeded to parse the whole date for following input :
5/31/2019 11:40:26 AM value=10
and the output is :
[root@earthserver telegraf]# echo > /tmp/telegraf.out ; systemctl restart telegraf ; sleep 2 ; tail -f /tmp/telegraf.out
parser,path=/media/sf_software/influxdb/telegraf/parser.test timestamp="5/31/2019 11:40:26 AM",value=10i 1560694961314402659Hello,
Top work, thank you for that! So I am saving now the timestamp properly as string it seems, as I have the telegraf time when the metric was introduced and this timestamp. Is this correct?
How can I know use the timestamp on Grafana, or Chronograf to order by it?
Could be related to the timezone parsing? Do we need to put something like :tz-custom after the timestamp, and if so, what exactly for this case? I have seen telegraf/plugins/inputs/logparser at master · influxdata/telegraf · GitHub but that doesn’t mention this specific use case.
This is the result of telegraf if I try with:
[inputs.logparser.grok]
patterns = ['%{DATESTAMP_AMPM:timestamp:ts} %{SPACE} %{NUMBER} \(%{NUMBER}\) Action started:%{GREEDYDATA:restaction}']
custom_patterns = "DATESTAMP_AMPM %{DATESTAMP} (AM|PM)"
And this the output:
2019-06-16T16:24:09Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"veeamvbo3", Flush Interval:10s
2019-06-16T16:24:09Z E! Error parsing timestamp [4/5/2019 12:22:15 PM], could not find any suitable time layouts.
2019-06-16T16:24:09Z E! Error parsing timestamp [4/8/2019 9:20:28 AM], could not find any suitable time layouts.
2019-06-16T16:24:09Z E! Error parsing timestamp [4/8/2019 9:21:43 AM], could not find any suitable time layouts.
2019-06-16T16:24:09Z E! Error parsing timestamp [4/11/2019 2:29:07 AM], could not find any suitable time layouts.
2019-06-16T16:24:09Z E! Error parsing timestamp [4/30/2019 4:52:44 AM], could not find any suitable time layouts.
2019-06-16T16:24:09Z E! Error parsing timestamp [5/1/2019 3:41:36 AM], could not find any suitable time layouts.I dont know if it is related to the tz … I will try …
is it possible to change the format of the logged dates in your logfile ?
Hello, I wish it was that simple, I’ve already contacted the vendor but that will be difficult to change.
Can you think of something extra or any way I can use that data, if not by time, which ideal, at least maybe as a tag will be more flexible?
you could use the field as a tag by adding the tag modifier but that will increase your cardinality
patterns = ["%{DATESTAMP_AMPM:mytimestamp:tag} value=%{NUMBER:value:int}"]
I believe this will work:
%{DATESTAMP_AMPM:timestamp:ts-"1/2/2006 03:04:05 PM"}
The :ts-"blah" suffix describes to Go how to interprete the string matched in the %{DATESTAMP_AMPM} pattern. It is a little strange to work with on first use, the way it works is that you put the reference date, defined to be “Mon Jan 2 15:04:05 MST 2006” into the format of the date in your input file.
If the timestamps are not in UTC time, you will need to use the timezone option too, since your date string does not contain zone information.
Hello, thank you for the reply, it didn’t work, this is the output:
2019-06-18T07:43:14Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"veeamvbo3", Flush Interval:10s
2019-06-18T07:43:14Z E! Error parsing 4/8/2019 9:20:28 AM to time layout [1/2/2006 03:04:05 PM]: parsing time "4/8/2019 9:20:28 AM" as "1/2/2006 03:04:05 PM": cannot parse "9:20:28 AM" as "03"
2019-06-18T07:43:14Z E! Error parsing 4/8/2019 9:21:43 AM to time layout [1/2/2006 03:04:05 PM]: parsing time "4/8/2019 9:21:43 AM" as "1/2/2006 03:04:05 PM": cannot parse "9:21:43 AM" as "03"
2019-06-18T07:43:14Z E! Error parsing 4/11/2019 2:29:07 AM to time layout [1/2/2006 03:04:05 PM]: parsing time "4/11/2019 2:29:07 AM" as "1/2/2006 03:04:05 PM": cannot parse "2:29:07 AM" as "03"
2019-06-18T07:43:14Z E! Error parsing 4/30/2019 4:52:44 AM to time layout [1/2/2006 03:04:05 PM]: parsing time "4/30/2019 4:52:44 AM" as "1/2/2006 03:04:05 PM": cannot parse "4:52:44 AM" as "03"Looks like we just need to remove the leading 0 from the hour:
%{DATESTAMP_AMPM:timestamp:ts-"1/2/2006 3:04:05 PM"}
Hi Daniel, you are awesome! It is working now, and the final v0.1 of the project can be found here if interested - GitHub - jorgedlcruz/veeam-backup-for-microsoft365-grafana-logs-security-performance: A Grafana Dashboard for Veeam Backup for Microsoft Office Advanced Security and Performance Monitoring I am sure it is just the start of parsing logs and do awesome stuff with telegraf. Appreciated!
