Telegraf logparser input prometheus_client output

1

I’m trying to use the logparser input and the prometheus_client output.
I have set up logparser and cpu as input, prometheus_client and file as output.

I get log and cpu metrics in the file output, but only cpu data in the prometheus client output.

Please help!

running on “Ubuntu 14.04.5 LTS”

created by puppet using mod ‘datacentred-telegraf’, ‘1.5.0’ in Puppetfile and following hieradata:-

telegraf::inputs: 
  cpu:
    percpu: true
    totalcpu: true
    collect_cpu_time: false
    report_active: false  
telegraf::plugin_inputs:
  'logparser':
    plugin_type: 'logparser'
    options:
      files: 
        - '/srv/log/rus/**.log'
      from_beginning: true
    single_section:      
      logparser.grok:
        measurement: 'log_data'
        patterns:
          - '%%{}{ALTERNATIVES}'
          - '%%{}{UPGRADES}'  
          - '%%{}{CLOUD_INIT}'
          - '%%{}{SYSLOG_STYLE}'
          - '%%{}{DPKG}'
        custom_patterns: |-
          ""
          TIMESTAMP_ALT %%{}{YEAR}-%%{}{MONTHNUM}-%%{}{MONTHDAY} %%{}{HOUR}:%%{}{MINUTE}:%%{}{SECOND}:
          ALTERNATIVES %%{}{WORD:command} %%{}{TIMESTAMP_ALT:timestamp} %%{}{GREEDYDATA:message}
          UPGRADES %%{}{TIMESTAMP_ISO8601:timestamp} %%{}{LOGLEVEL:loglevel} %%{}{GREEDYDATA:message}
          PROGNAME [0-9a-zA-Z_]+\\.py
          CLOUD_INIT %%{}{TIMESTAMP_ISO8601:timestamp} - %%{}{PROGNAME:program}\\[%%{}{LOGLEVEL:loglevel}\\]: %%{}{GREEDYDATA:message}
          SYSLOG_STYLE %%{}{SYSLOGBASE} %%{}{GREEDYDATA:message}
          DPKG %%{}{TIMESTAMP_ISO8601:timestamp} %%{}{GREEDYDATA:message}
          ""
telegraf::outputs:
  prometheus_client:
    listen: ':9273'
    expiration_interval: '0'
    collectors_exclude: 
      - 'gocollector'
      - 'process'
  file:
    files:
      - '/var/log/telegraf/metrics.txt'

with the node.pp containing:-

  class { 'telegraf': }

  $_input = hiera_hash('telegraf::plugin_inputs',{})
  if $_input != '{}'
  {
    create_resources('telegraf::input',$_input)
  }

/etc/telegraf/telegraf.cong:-

# Telegraf Configuration
#
# THIS FILE IS MANAGED BY PUPPET
#
[global_tags]

[agent]
  hostname = "platform-production-log1"
  omit_hostname = false
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  logfile = ""
  debug = false
  quiet = false

#
# OUTPUTS:
#
[[outputs.file]]
  files = ["/var/log/telegraf/metrics.txt"]
[[outputs.prometheus_client]]
  collectors_exclude = ["gocollector", "process"]
  expiration_interval = "0"
  listen = ":9273"

#
# INPUTS:
#
[[inputs.cpu]]
  collect_cpu_time = false
  percpu = true
  report_active = false
  totalcpu = true

/etc/telegraf/telegraf.d/logparser.conf:-

[[inputs.logparser]]
  files = ["/srv/log/rus/**.log"]
  from_beginning = true
[inputs.logparser.grok]
  custom_patterns = """
TIMESTAMP_ALT %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}:
ALTERNATIVES %{WORD:command} %{TIMESTAMP_ALT:timestamp} %{GREEDYDATA:message}
UPGRADES %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}
PROGNAME [0-9a-zA-Z_]+\\.py
CLOUD_INIT %{TIMESTAMP_ISO8601:timestamp} - %{PROGNAME:program}\\[%{LOGLEVEL:loglevel}\\]: %{GREEDYDATA:message}
SYSLOG_STYLE %{SYSLOGBASE} %{GREEDYDATA:message}
DPKG %{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:message}
"""
  measurement = "log_data"
  patterns = ["%{ALTERNATIVES}", "%{UPGRADES}", "%{CLOUD_INIT}", "%{SYSLOG_STYLE}", "%{DPKG}"]
2

I think this may be because the prometheus format does not have a string value type, only numeric values, and your grok pattern is only capturing strings.

1 Like
3

I’ve since implemented a processor regex plugin to add count=1 to the metric and it all now works.

4

For anyone else with a similar issue … this required:-

Install go, clone the influxdb/telegraf repo, add the regex plugin, make it (in an environment running the same OS as the destination environment), and put the output telegraf executable there. This is being built in travis and the executable deployed out to an s3 bucket.

The script used to launch puppet pulls the telegraf executable into the destination machine prior to the puppet run and the datacentred/telegraf puppet module is told not to manage the installation, just the config files.

Then modify the datacentred/telegraf puppet module to support processor plugin types, and add the relevant hiera.

5

That was it! Hell of a parser…