After a little crying I managed to come up with the grok_pattern for it as follows:
%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} %{WORD:ceph}-%{WORD:instance}\[%{POSINT:pid}\]: %{GREEDYDATA:message}
This seems to work for the log in a grok debugger online as I can see the output as:
[
{
"timestamp": "Jul 29 19:18:31",
"hostname": "apollo3",
"ceph": "ceph",
"instance": "osd",
"pid": 1770536,
"message": "_get_class not permitted to load sdk"
}
]
for the following log line:
Jul 29 19:18:31 apollo ceph-osd[1770536]: _get_class not permitted to load sdk
However, when I use the same grok_patter in my telegraf.conf like:
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = "0s"
debug = true
quiet = false
[[inputs.tail]]
files = ["/var/log/messages"]
from_beginning = true
watch_method = "inotify"
data_format = "grok"
grok_patterns = ["%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} %{WORD:ceph}-%{WORD:instance}\[%{POSINT:pid}\]: %{GREEDYDATA:message}"]
[[outputs.file]]
files = ["stdout"]
data_format = "prometheus"
I get a syntax error:
2024-08-01T07:26:16Z E! error loading config file /etc/telegraf/telegraf.conf: error parsing data: line 17: invalid TOML syntax
But if I replace the backslash \ with \\, then there is no syntax error but the log which I want to be read does not get read. As shown below:
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = "0s"
debug = true
quiet = false
[[inputs.tail]]
files = ["/var/log/messages"]
from_beginning = true
watch_method = "inotify"
data_format = "grok"
grok_patterns = ["%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} %{WORD:ceph}-%{WORD:instance}\\[%{POSINT:pid}\\]: %{GREEDYDATA:message}"]
[[outputs.file]]
files = ["stdout"]
data_format = "prometheus"
the output is:
root@test telegraf]# telegraf --config /etc/telegraf/telegraf.conf -test
2024-08-01T07:35:35Z I! Loading config: /etc/telegraf/telegraf.conf
2024-08-01T07:35:35Z I! Starting Telegraf 1.31.2 brought to you by InfluxData the makers of InfluxDB
2024-08-01T07:35:35Z I! Available plugins: 234 inputs, 9 aggregators, 32 processors, 26 parsers, 60 outputs, 6 secret-stores
2024-08-01T07:35:35Z I! Loaded inputs: tail
2024-08-01T07:35:35Z I! Loaded aggregators:
2024-08-01T07:35:35Z I! Loaded processors:
2024-08-01T07:35:35Z I! Loaded secretstores:
2024-08-01T07:35:35Z W! Outputs are not used in testing mode!
2024-08-01T07:35:35Z I! Tags enabled: host=apollo3.procan.local
2024-08-01T07:35:35Z D! [agent] Initializing plugins
2024-08-01T07:35:35Z D! [agent] Starting service inputs
2024-08-01T07:35:35Z D! [inputs.tail] Tail added for "/var/log/messages"
2024-08-01T07:35:35Z D! [agent] Stopping service inputs
2024-08-01T07:35:35Z D! [inputs.tail] Tail removed for "/var/log/messages"
> tail,host=apollo.local,path=/var/log/messages ceph="ceph",hostname="apollo",instance="mon",message="mon.apollo@3(peon) e10 handle_command mon_command({\"prefix\": \"status\"} v 0) v1",pid="2343148",timestamp="Jul 28 03:33:01" 1722497735536110962
2024-08-01T07:35:35Z D! [agent] Input channel closed
2024-08-01T07:35:35Z D! [agent] Stopped Successfully
> tail,host=apollo.local,path=/var/log/messages ceph="ceph",hostname="apollo",instance="mon",message="log_channel(audit) log [DBG] : from='client.? 192.168.x.x:0/2107390132' entity='client.admin' cmd=[{\"prefix\": \"status\"}]: dispatch",pid="2343148",timestamp="Jul 28 03:33:01" 1722497735536155572
So as we can see the output does not have the one I actually want. What am I missing?
