I need to parse a MySQL slow log message with a Telegraf and then pass it to Prometheus.
an example of such message is:
# Time: 2021-04-01T13:26:56.734727Z
# User@Host: root[root] @ localhost [] Id: 3
# Query_time: 3.001243 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0
SET timestamp=1617283616;
select sleep(3);
To design a grok pattern that should handle this multiline log entry I followed the advice from the Telegraf’s documentation and created pattern via the online grok expression designer:
https://grokdebug.herokuapp.com/
# Time: %{TIMESTAMP_ISO8601}\n# User@Host: %{NOTSPACE} @ %{HOSTNAME} %{SPACE}\[\]%{SPACE}Id:%{SPACE}%{INT}\n# Query_time:%{SPACE}%{BASE16FLOAT}%{SPACE}Lock_time: %{BASE16FLOAT}%{SPACE}Rows_sent:%{SPACE}%{BASE10NUM}%{SPACE}Rows_examined: %{BASE10NUM}%{SPACE}\nSET timestamp=%{BASE10NUM:timestamp};\n%{GREEDYDATA:query}
However it turned out that Telegraf can not parse the grok regular expression above as a single line, they designed a so named
inputs.tail.multiline
feature to handle such situations, please also see the reference below:
How should I describe the expression below in the context of inputs.tail.multiline plugin?
###############################################################################
# SERVICE INPUT PLUGINS #
###############################################################################
# # Parse the new lines appended to a file
[[inputs.tail]]
files = ["/var/log/mysql/mysql-slow.log"]
from_beginning = true
watch_method = "inotify"
max_undelivered_lines = 1000
character_encoding = "utf-8"
name_override = "mysql_slow_log"
data_format = "grok"
grok_patterns = ["^# Time: %{TIMESTAMP_ISO8601:time}$"]
[inputs.tail.multiline]
pattern = "^%{GREEDYDATA:logmessage}"
match_which_line = "previous"
This part of the configuration can parse only the very first line of the log entry that consists from 5 lines and produces the output:
mysql_slow_log,host=c1-master1,path=/var/log/mysql/mysql-slow.log time="2021-04-01T13:26:56.734727Z" 1617345403983043021
How should I modify the configuration above to parse all 5 lines from the MySQL slow log entry?
Thank you.