Telegraf tail plugin to prometheus server

Telegraf
,
1

Hi,

I am trying to use telegraf tail plugin to read log files and expose that on a port and prometheus to read from there.
However when I execute the conf file I do not see any of the log data rather see a generic output as attached in image1.Below is the conf file I am using.
Please help me understand where am I missing.
P.S:grok debugger and the TOML syntax are verified in the online tools.

[[inputs.tail]]
## file(s) to tail:
files = ["/home/test.log"]
from_beginning = true
  #name of the "Metric" (which I want to see in prometheus)
  name_override = "test"
  grok_patterns = ["%{CUSTOM_LOG}"]
  grok_custom_patterns = '''
CUSTOM_LOG %{DATE:date} %{TIME:time} %{WORD:level}  \[org.%{DATA.org}\] %{GREEDYDATA:message}
'''
 data_format = "grok"

  [[outputs.prometheus_client]]
  ## Address to listen on.
  listen = ":9001"
  metric_version = 2
   ## Path to publish the metrics on.
      path = "/metrics"
  #[outputs.prometheus_client.tagpass]

Thanks in advance!!

image1
image1965×244 38.5 KB

2

First, forget about Prometheus and check if the input plugin is working.
Add the following output plugin and check if the desired output is generated.

# file output only for debugging
[[outputs.file]]
  files = ["debug.out"]
  influx_sort_fields = true
3

Hi Franky,

Thank you for the response.
I tried your suggestion.However the debug.out is empty file.
I also tried the custom pattern with %{GREEDYDATA} alone hoping log file to be read irrespective of the pattern.
That didn’t work either.

4

Then i suspect that the grok pattern is not working.
Can you provide a sample line/snippet of your log file?

5

HI Frank,

The grok pattern works with
grok_custom_patterns = ‘’’
CUSTOM_LOG %{TIMESTAMP_ISO8601:logdate}
‘’’
Testing with debug.out passes the data.(debug.out contains the parsed log)
But when I use the prometheus output there are no metrics on port.
[[outputs.prometheus_client]]

Address to listen on.

listen = “localhost:9023”

Path to publish the metrics on.

  path = "/metrics"

The output of “wget http://localhost:9023/metrics” is same as the image attached before.

TIA

6

If you have data in the debug.out file, but not via the outputs.prometheus_client plugin, one more thing comes to mind:

If you only have strings as metrics, the Prometheus plugin will ignore that, quote from the doc:

Note: String fields are ignored and do not produce Prometheus metrics.

But for that, we would have to see some sample data from the log, as I already wrote.

7

Here is the sample log lines.

2021-07-19 16:45:02,556 INFO  [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl    
2021-07-19 16:45:02,557 INFO  [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed connection: RemotingConnectionImpl
2021-07-19 16:45:02,557 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to /11.11.111.11:9999 has been detected: readAddress(..) failed: Connection reset by peer [code=GENERIC_EXCEPTION]

The goal is to pick the WARNING logs and send them to prometheus as a metric.

8

This is one possible working solution.

[[inputs.tail]]
  files = ["prometheus.input"]
  from_beginning = true
  name_override = "prometheus"
  grok_patterns =  ['%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05.000"} %{LOGLEVEL:level}  \[%{DATA:url:string}\] AMQ%{NUMBER:amq:int}: %{GREEDYDATA:message}']
  grok_timezone = "UTC"  # or "Local"
  data_format = "grok"
  tagexclude = ["path"]  # we dont need the file path as tag?

[[processors.converter]]  # convert log level to tag
  [processors.converter.fields]
    tag = ["level"]

[[outputs.prometheus_client]]
  # visit http://localhost:9001/metrics
  listen = ":9001"
  metric_version = 2
  path = "/metrics"
  [outputs.prometheus_client.tagpass]  # only pass points with certain tag values
    level = ["WARN", "ERROR", "CRITICAL"]  # adjust the logging keywords if necessary

[[outputs.file]]  # file output only for debugging
  files = ["prometheus.out"]
  influx_sort_fields = true

Besides, i am not a monitoring expert, but I think Prometheus may not be the best tool here, as it is designed more for metrics and less for logging messages.

9

Thank you for the code @Franky1 .
I executed this and the output on http://localhost:9001/metrics is stating amq as untyped.(Attached output image)
I also tried changing the tag from “level” to “amq” and passing the tag for " amq " like below,

 [outputs.prometheus_client.tagpass]  # only pass points with certain tag values
    amq = ["212037"] type or paste code here

However this amq filter does not give any output in http://localhost:9001/metrics and reading all the log lines to prometheus.out.

Is there a way where I can use amq as a filter and send it to prometheus.
I did try “amq” as float than int & that also seems not working.
Note:telegraf version is 1.14.0.

amq_untyped
amq_untyped1903×105 38.2 KB

TIA