How to send Metricbeat data to InfluxDB via Kafka?

influxdata
influxdb
#1

Hi,
I’ve been trying to send Metricbeat data to influx using the flow: Metricbeat -> Kafka -> Logstash -> InfluxDB but have been unable to do so. Can someone suggest a better way? Also, please point out if I’ve made any mistakes. Please find the modifications I’ve made in config files:

metricbeat.yml–
output.kafka:
hosts: “localhost:9092”
topic: “test”
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000

kafkainflux.conf file which I created in conf.d in Logstash to make a pipeline:
input {
kafka {
topics => “test”
codec => “json”
}
}
output {
stdout {codec => rubydebug}
influxdb {
use_event_fields_for_data_points => true
data_points => {}
host => “localhost”
db => “xyz”
exclude_fields => ["@timestamp", “@Version”, “sequence”, “message”, “type”,“path”]
use_event_fields_for_data_points => true
port => 8086
codec => “json”
}
}

The error message in Logstash logs is:

invalid boolean\nunable to parse ‘logstash,host={“name”\=\u003e"instance-1"} system={“process”=\u003e{“fd”=\u003e{“limit”=\u003e{“soft”=\u003e1024, “hard”=\u003e4096}, “open”=\u003e9}, “cpu”=\u003e{“total”=\u003e{“norm”=\u003e{“pct”=\u003e0.002}, “value”=\u003e3420, “pct”=\u003e0.002}, “start_time”=\u003e"2018-07-03T08:57:32.000Z"}, “username”=\u003e"syslog", “memory”=\u003e{“rss”=\u003e{“pct”=\u003e0.0187, “bytes”=\u003e72462336}, “size”=\u003e405155840, “share”=\u003e819200}, “cgroup”=\u003e{“blkio”=\u003e{“id”=\u003e"rsyslog.service", “total”=\u003e{“ios”=\u003e3170, “bytes”=\u003e149295104}, “path”=\u003e"/system.slice/rsyslog.service"}, “id”=\u003e"rsyslog.service", “cpu”=\u003e{“rt”=\u003e{“period”=\u003e{“us”=\u003e0}, “runtime”=\u003e{“us”=\u003e0}}, “stats”=\u003e{“throttled”=\u003e{“periods”=\u003e0, “ns”=\u003e0}, “periods”=\u003e0}, “id”=\u003e"rsyslog.service", “cfs”=\u003e{“period”=\u003e{“us”=\u003e100000}, “quota”=\u003e{“us”=\u003e0}, “shares”=\u003e1024}, “path”=\u003e"/system.slice/rsyslog.service"}, “memory”=\u003e{“kmem”=\u003e{“limit”=\u003e{“bytes”=\u003e9223372036854771712}, “usage”=\u003e{“max”=\u003e{“bytes”=\u003e487424}, “bytes”=\u003e487424}, “failures”=\u003e0}, “stats”=\u003e{“hierarchical_memsw_limit”=\u003e{“bytes”=\u003e0}, “major_page_faults”=\u003e1542, “hierarchical_memory_limit”=\u003e{“bytes”=\u003e9223372036854771712}, “inactive_anon”=\u003e{“bytes”=\u003e4096}, “swap”=\u003e{“bytes”=\u003e0}, “page_faults”=\u003e21537, “rss_huge”=\u003e{“bytes”=\u003e0}, “active_file”=\u003e{“bytes”=\u003e12288}, “pages_in”=\u003e58650, “cache”=\u003e{“bytes”=\u003e18477056}, “unevictable”=\u003e{“bytes”=\u003e0}, “inactive_file”=\u003e{“bytes”=\u003e18460672}, “pages_out”=\u003e36554, “rss”=\u003e{“bytes”=\u003e72028160}, “active_anon”=\u003e{“bytes”=\u003e72028160}, “mapped_file”=\u003e{“bytes”=\u003e532480}}, “id”=\u003e"rsyslog.service", “kmem_tcp”=\u003e{“limit”=\u003e{“bytes”=\u003e9223372036854771712}, “usage”=\u003e{“max”=\u003e{“bytes”=\u003e0}, “bytes”=\u003e0}, “failures”=\u003e0}, “memsw”=\u003e{“limit”=\u003e{“bytes”=\u003e0}, “usage”=\u003e{“max”=\u003e{“bytes”=\u003e0}, “bytes”=\u003e0}, “failures”=\u003e0}, “mem”=\u003e{“limit”=\u003e{“bytes”=\u003e9223372036854771712}, “usage”=\u003e{“max”=\u003e{“bytes”=\u003e91082752}, “bytes”=\u003e90992640}, “failures”=\u003e0}, “path”=\u003e"/system.slice/rsyslog.service"}, “cpuacct”=\u003e{“stats”=\u003e{“user”=\u003e{“ns”=\u003e1320000000}, “system”=\u003e{“ns”=\u003e1500000000}}, “id”=\u003e"rsyslog.service", “total”=\u003e{“ns”=\u003e3428309693}, “percpu”=\u003e{“1”=\u003e3428309693}, “path”=\u003e"/system.slice/rsyslog.service"}, “path”=\u003e"/system.slice/rsyslog.service"}, “name”=\u003e"rsyslogd", “cmdline”=\u003e"/usr/sbin/rsyslogd -n", “pgid”=\u003e1402, “cwd”=\u003e"/", “state”=\u003e"sleeping", “ppid”=\u003e1, “pid”=\u003e1402}},beat={“name”=\u003e"instance-1", “hostname”=\u003e"instance-1", “version”=\u003e"6.3.0"},@version=“1”,metricset={“rtt”=\u003e44364, “name”=\u003e"process", “module”=\u003e"system"} 1530707948784’: invalid boolean"}