# Telegraf tail plugin to prometheus server

**URL:** <https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883>\
**Category:** Telegraf\
**Tags:** telegraf, prometheus\
**Created:** [July 24, 2021, 5:26pm UTC](https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883 "2021-07-24T17:26:57Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![kavya](https://avatars.discourse-cdn.com/v4/letter/k/e480ec/32.png) [@kavya](https://community.influxdata.com/u/kavya)\
**Post date:** [July 24, 2021, 5:26pm UTC](https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883/1 "2021-07-24T17:26:57Z")

</div>

Hi,

I am trying to use telegraf tail plugin to read log files and expose that on a port and prometheus to read from there.  
However when I execute the conf file I do not see any of the log data rather see a generic output as attached in image1.Below is the conf file I am using.  
Please help me understand where am I missing.  
P.S:grok debugger and the TOML syntax are verified in the online tools.

```auto
[[inputs.tail]]
## file(s) to tail:
files = ["/home/test.log"]
from_beginning = true
  #name of the "Metric" (which I want to see in prometheus)
  name_override = "test"
  grok_patterns = ["%{CUSTOM_LOG}"]
  grok_custom_patterns = '''
CUSTOM_LOG %{DATE:date} %{TIME:time} %{WORD:level} \[org.%{DATA.org}\] %{GREEDYDATA:message}
'''
 data_format = "grok"

  [[outputs.prometheus_client]]
  ## Address to listen on.
  listen = ":9001"
  metric_version = 2
   ## Path to publish the metrics on.
      path = "/metrics"
  #[outputs.prometheus_client.tagpass]

```

Thanks in advance!!

 ![image1](https://us1.discourse-cdn.com/flex023/uploads/influxdata/original/2X/4/442e4b6198fef5fbe7b6fe50b02952470e299fac.png)

---

<div class="post-metadata">

**Author:** ![Franky1](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/franky1/32/6666_2.png) [@Franky1](https://community.influxdata.com/u/Franky1)\
**Post date:** [July 26, 2021, 8:30am UTC](https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883/2 "2021-07-26T08:30:47Z")

</div>

First, forget about Prometheus and check if the input plugin is working.  
Add the following output plugin and check if the desired output is generated.

```auto
# file output only for debugging
[[outputs.file]]
  files = ["debug.out"]
  influx_sort_fields = true

```

---

<div class="post-metadata">

**Author:** ![kavya](https://avatars.discourse-cdn.com/v4/letter/k/e480ec/32.png) [@kavya](https://community.influxdata.com/u/kavya)\
**Post date:** [July 26, 2021, 8:51am UTC](https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883/3 "2021-07-26T08:51:05Z")

</div>

Hi Franky,

Thank you for the response.  
I tried your suggestion.However the debug.out is empty file.  
I also tried the custom pattern with %{GREEDYDATA} alone hoping log file to be read irrespective of the pattern.  
That didn’t work either.

---

<div class="post-metadata">

**Author:** ![Franky1](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/franky1/32/6666_2.png) [@Franky1](https://community.influxdata.com/u/Franky1)\
**Post date:** [July 26, 2021, 9:08am UTC](https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883/4 "2021-07-26T09:08:01Z")

</div>

> [@kavya](#):
>
> However the debug.out is empty file.

Then i suspect that the grok pattern is not working.  
Can you provide a sample line/snippet of your log file?

---

<div class="post-metadata">

**Author:** ![kavya](https://avatars.discourse-cdn.com/v4/letter/k/e480ec/32.png) [@kavya](https://community.influxdata.com/u/kavya)\
**Post date:** [July 26, 2021, 8:08pm UTC](https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883/5 "2021-07-26T20:08:49Z")

</div>

HI Frank,

The grok pattern works with  
grok\_custom\_patterns = ‘’’  
CUSTOM\_LOG %{TIMESTAMP\_ISO8601:logdate}  
‘’’  
Testing with debug.out passes the data.(debug.out contains the parsed log)  
But when I use the prometheus output there are no metrics on port.  
[[outputs.prometheus\_client]]

## Address to listen on.

listen = “localhost:9023”

## Path to publish the metrics on.

```
  path = "/metrics"

```

The output of “wget [http://localhost:9023/metrics](http://localhost:9023/metrics)” is same as the image attached before.

TIA

---

<div class="post-metadata">

**Author:** ![Franky1](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/franky1/32/6666_2.png) [@Franky1](https://community.influxdata.com/u/Franky1)\
**Post date:** [July 27, 2021, 1:29pm UTC](https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883/6 "2021-07-27T13:29:22Z")

</div>

If you have data in the `debug.out` file, but _not_ via the `outputs.prometheus_client` plugin, one more thing comes to mind:

If you only have **strings** as metrics, the Prometheus plugin will ignore that, quote from the doc:

> **Note:** String fields are ignored and do not produce Prometheus metrics.

But for that, we would have to see some sample data from the log, as I already wrote.

---

<div class="post-metadata">

**Author:** ![kavya](https://avatars.discourse-cdn.com/v4/letter/k/e480ec/32.png) [@kavya](https://community.influxdata.com/u/kavya)\
**Post date:** [July 28, 2021, 7:24am UTC](https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883/7 "2021-07-28T07:24:57Z")

</div>

Here is the sample log lines.

```auto
2021-07-19 16:45:02,556 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841000: created connection: RemotingConnectionImpl    
2021-07-19 16:45:02,557 INFO [org.apache.activemq.artemis.core.server.plugin.impl] AMQ841001: destroyed connection: RemotingConnectionImpl
2021-07-19 16:45:02,557 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to /11.11.111.11:9999 has been detected: readAddress(..) failed: Connection reset by peer [code=GENERIC_EXCEPTION]

```

The goal is to pick the WARNING logs and send them to prometheus as a metric.

---

<div class="post-metadata">

**Author:** ![Franky1](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/franky1/32/6666_2.png) [@Franky1](https://community.influxdata.com/u/Franky1)\
**Post date:** [July 28, 2021, 10:00am UTC](https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883/8 "2021-07-28T10:00:49Z")

</div>

This is one possible working solution.

```auto
[[inputs.tail]]
  files = ["prometheus.input"]
  from_beginning = true
  name_override = "prometheus"
  grok_patterns = ['%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05.000"} %{LOGLEVEL:level} \[%{DATA:url:string}\] AMQ%{NUMBER:amq:int}: %{GREEDYDATA:message}']
  grok_timezone = "UTC" # or "Local"
  data_format = "grok"
  tagexclude = ["path"] # we dont need the file path as tag?

[[processors.converter]] # convert log level to tag
  [processors.converter.fields]
    tag = ["level"]

[[outputs.prometheus_client]]
  # visit http://localhost:9001/metrics
  listen = ":9001"
  metric_version = 2
  path = "/metrics"
  [outputs.prometheus_client.tagpass] # only pass points with certain tag values
    level = ["WARN", "ERROR", "CRITICAL"] # adjust the logging keywords if necessary

[[outputs.file]] # file output only for debugging
  files = ["prometheus.out"]
  influx_sort_fields = true

```

* * *

Besides, i am not a monitoring expert, but I think Prometheus may not be the best tool here, as it is designed more for metrics and less for logging messages.

---

<div class="post-metadata">

**Author:** ![kavya](https://avatars.discourse-cdn.com/v4/letter/k/e480ec/32.png) [@kavya](https://community.influxdata.com/u/kavya)\
**Post date:** [July 28, 2021, 9:05pm UTC](https://community.influxdata.com/t/telegraf-tail-plugin-to-prometheus-server/20883/9 "2021-07-28T21:05:52Z")

</div>

Thank you for the code @Franky1 .  
I executed this and the output on [http://localhost:9001/metrics](http://localhost:9001/metrics) is stating amq as untyped.(Attached output image)  
I also tried changing the tag from “level” to “amq” and passing the tag for " amq " like below,

```auto
 [outputs.prometheus_client.tagpass] # only pass points with certain tag values
    amq = ["212037"] type or paste code here

```

However this amq filter does not give any output in [http://localhost:9001/metrics](http://localhost:9001/metrics) and reading all the log lines to prometheus.out.

Is there a way where I can use amq as a filter and send it to prometheus.  
I did try “amq” as float than int & that also seems not working.  
Note:telegraf version is 1.14.0.

 ![amq_untyped](https://us1.discourse-cdn.com/flex023/uploads/influxdata/original/2X/f/f47f2b98eb913f89100c3fda45ffa66deacd359f.png)

TIA