Metrics data not sending to influxdb

Telegraf
1

Hi All,

I have installed metricbeat (Version 6.3.0) in my ubuntu machine and sending data to logstash output and I am running logstash (5.6.10 Version) to send metricbeat data to influxdb and below is my logstash configuration,

Installed the logstash-output-influxdb plugin as like below
./logstash-plugin install --version 5.0.3 logstash-output-influxdb

Logstash configuration

stash configuration

input {
  beats {
    port => 5044
  }
}

output {
  stdout {codec => rubydebug}
  influxdb {
    host => "localhost"
    port => 8086
    user => "admin"
    password => "password"
    db => "metrics"
    codec => "json"
    use_event_fields_for_data_points => true
    exclude_fields => ["@timestamp", "@version", "sequence", "message", "type"]
    data_points => {
    }
  }
}

However I am able to run logstash successfully but metrics data were not sending to influxdb and below is the exception i am getting in console,

[2018-07-03T09:17:59,344][WARN ][logstash.outputs.influxdb] Non recoverable exception while writing to InfluxDB {:exception=>#<InfluxDB::Error: {"error":"unable to parse 'logstash,beats_input_raw_event=true,host={\"name\"\\=\u003e\"localhost\"} system={\"memory\"=\u003e{\"hugepages\"=\u003e{\"total\"=\u003e0, \"default_size\"=\u003e2097152, \"surplus\"=\u003e0, \"reserved\"=\u003e0, \"used\"=\u003e{\"pct\"=\u003e0, \"bytes\"=\u003e0}, \"free\"=\u003e0}, \"actual\"=\u003e{\"free\"=\u003e4079476736, \"used\"=\u003e{\"pct\"=\u003e0.567, \"bytes\"=\u003e5342081024}}, \"total\"=\u003e9421557760, \"swap\"=\u003e{\"total\"=\u003e0, \"used\"=\u003e{\"pct\"=\u003e0, \"bytes\"=\u003e0}, \"free\"=\u003e0}, \"used\"=\u003e{\"pct\"=\u003e0.6941, \"bytes\"=\u003e6539534336}, \"free\"=\u003e2882023424}},beat={\"name\"=\u003e\"localhost\", \"hostname\"=\u003e\"localhost\", \"version\"=\u003e\"6.3.0\"},@version=\"1\",metricset={\"name\"=\u003e\"memory\", \"rtt\"=\u003e326, \"module\"=\u003e\"system\"} 1530609478299': invalid boolean"}

Please correct me if my configuration is wrong and help me to resolve this issue.

Regards,
Ganeshbabu R

2

Hi!

I’m not sure of this, because my Logstash input is not Beats (it’s Redis), but maybe I can help you :slight_smile:

Input:
If you are recieving a json message from Beats, then you probably need to put codec => "json" in the input and not in the output.
By default, Beats codec is “plain” (codec => "plain"). Source.

Filter:
If codec => "json" in the input doesn’t work try to use json filter plugin to extract your fields. Or one of these.

Output:
If you are using use_event_fields_for_data_points option in InfluxDB output, you don’t need data_points option.
Btw, its not required, but you are not specifying a measurement in InfluxDB, so by default your data is stored in a measurement named “Logstash”. Source.

If any of this works, I’d recommend you to use the File output plugin to watch what is going on and what are your event fields

Regards,
Ilyasbel

3

Hi @Ilyasbel

Thanks for your response…

Yes I am receiving a json message from beats but the metricbeat generates data with nested fields and below is the sample data which I got fr,

{
  "@timestamp": "2018-07-09T10:35:19.887Z",
  "system": {
    "process": {
      "memory": {
        "share": 0,
        "rss": {
          "pct": 0,
          "bytes": 0
        },
        "size": 0
      },
      "pgid": 0,
      "name": "khungtaskd",
      "cpu": {
        "start_time": "2018-07-03T04:55:39.000Z",
        "total": {
          "pct": 0,
          "value": 530,
          "norm": {
            "pct": 0
          }
        }
      },
      "pid": 23,
      "state": "sleeping",
      "fd": {
        "limit": {
          "soft": 1024,
          "hard": 4096
        },
        "open": 0
      },
      "username": "root",
      "ppid": 2
    }
  },
  "host": {
    "name": "elasticsearch"
  },
  "beat": {
    "name": "elasticsearch",
    "hostname": "elasticsearch",
    "version": "6.3.0"
  },
  "@version": "1",
  "metricset": {
    "name": "process",
    "rtt": 70229,
    "module": "system"
  },
  "fields": {
    "env": "development"
  },
  "tags": [
    "beats_input_raw_event"
  ]
}

Yes the data was not sending to influx if I given the codec => “json” in the input and also the metricbeat will generate the lot of fields and it captures all the metrics like cpu, process,memory,filesystem,network etc… In that case its difficult to write the grok pattern to extract the fields from the message, since the structure of the data might varies depends on the time.

Please let me know your thoughts on this.

Yes first thing I installed file output plugin in logstash and saw the messages which is receiving from the metricbeat and the event fields are mentioned above.

Regards,
Ganeshbabu R

4

Hi @babu.ganesh0708,

First of all, I need to know if your input is been interpreting like a JSON by Logstash or not.

If it is not been reading like a JSON, all your data is in a field named “message” and you need to extract the information with a filter (probably with json filter plugin and not with grok). This means that codec => "json" in beats input pluging is not working.

If codec => "json" in beats input pluging is working properly, you will have a event field for each top level field in the JSON (i.e. @timestamp, system, host, beat, @version…), so maybe you are not accessing properly to the nested fields:

  1. If you are using use_event_fields_for_data_points then you are only accesing to the top level fields.
  2. If you are using data_points maybe you are trying this: data_points => { "hostname" => "%{[hostname]}" }, instead of this: data_points => { "hostname" => "%{[beat][hostname]}" }

More information about accesing event fields.

I’d recomend you to use the file output pluging to practice how to acces to the nested fields in your message and check if your json message from beats is been interpreting as a JSON by Logstash or not.

5

Hi @Ilyasbel

Thanks for your response and I have few more clarifications,

Yes I am getting metricbeat data as JSON by logstash

Yes you are right I am not accessing the nested fields properly

If I set the use_event_fields_for_data_points as true then the top level fields also not sending to influxdb so then I used the data_points to send it correctly.

Yes you are right I am not accessing event nested fields correctly so I read the documentation and then I am able to send data to influxdb successfully.

This is my input JSON by logstash

{
	"@timestamp": "2018-07-09T10:23:55.780Z",
	"system": {
		"fsstat": {
			"total_size": {
				"total": 10340831232,
				"used": 5776019456,
				"free": 4564811776
			},
			"count": 2,
			"total_files": 1280000
		}
	},
	"host": {
		"name": "elasticsearch"
	},
	"beat": {
		"name": "elasticsearch",
		"hostname": "elasticsearch",
		"version": "6.3.0"
	},
	"@version": "1",
	"metricset": {
		"name": "fsstat",
		"rtt": 382,
		"module": "system"
	},
	"fields": {
		"env": "development"
	},
	"tags": ["beats_input_raw_event"]
}

Since metricbeat will generate data based on different modules like cpu, memory, process etc… For each module it has different set of fields and Do you think adding all the event fields in the data points is the right idea?

Please let me know your thoughts.

Thanks,
Ganeshbabu R

6

Hi @babu.ganesh0708,

That’s weird… Maybe Logstash have problems to cast the top level fields to string because they have quotation marks :roll_eyes:

Cool!! I’m glad I helped you :smiley:

Honestly, I have no idea. You should ask someone else with more experience, I started using Logstash a month ago, and i’m not an expert :sweat_smile:

Regards,
Ilyasbel