Telegraf Tail file that contains json data to Influx

1

How can I extract the json data into influx by tailing a file that has the JSON data?

EG:

My logs are like this:

2020-12-01T18:34:06+02:00 10.132.92.22 {"wfd_successful_hits_sec": "0", "sql_hits_sec_max": "0", "timestamp": "2020/12/01 18:34:01", "connection_sec_max": "1922", "http_hits_sec_max": "1106", "http_hits_sec": "106", "wfd_successful_hits_sec_max": "0", "sql_hits_sec": "0", "sql_audit_phase2_events_sec_max": "0", "hdfs_hits_sec": "0", "connection_sec": "26"}

Is there a way to just extract the JSON part and send it to influx?

I know the grok pattern (\{.*\})$ will extract the JSON part.

My config looks like this:

[global_tags]
[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = false
[[inputs.tail]]
  files = ["/opt/soc/perf_stat/host*log"]
  data_format = "json"
[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = "TOKEN"
  organization = "ORG"
  bucket = "performance_stats
2

@vantoryc You’re headed down the right path. Rather than using the tail input, you can use the file input. Here’s a guide for using Telegraf to read JSON data from a file and write it to InfluxDB: JSON input data format | Telegraf 1.16 Documentation

3

@scott So this is part of a complex problem I’m trying to solve. We are getting what are basically appliance stats over syslog.

Due to some limitations we are not able to read directly from syslog but have to read it from the file where the data is stored.

At the date/time I posted this I was trying to solve above, but I figured I can alter the syslog server to save the rawmsg starting from the opening { so I did manage to get the above solved but I am now on a different set of problems which warrants a new post IMO but I’ll provide a brief overview below.

These are the logs we are getting:

The imperva_* are the measurements name that I want recorded as the appliance manufacturer has provided us with prebuilt Grafana dashboard we would like to use.

image
image849×853 13.2 KB