Import Collectd data to influxdb just when needed

telegraf
influxdb
collectd

#1

Hi,

We are facing and issue while trying to insert data to influxdb to be picked up from grafana. As of now there a test going on where we are inserting directly from collectd from a host to and influxdb db, the thing is that this is getting pretty huge and this is only information from 1 single host, we have hundreds of hosts in our PROD environment which is our main goal.

What we are trying to do is to insert only the data whenever we need to investigate an outage, for that we are planning to do it from the RAW files that collectd is creating right now. We came across a solution that is there but for NMON which does exactly what we need but for NMON files but I honestly don’t know anything about Golang which is the language they used for that, or at least I think so.

I found in this community that there is a PARSER plugin that is part of telegraf, but what I can’t find is how can I use it to take the information from a RAW file given by collectd and insert it into a influxdb.

I would really appreciate if someone can give me an advice on this matter.

Thanks in Advance.


#2

You can use telegraf to consume the file by configuring the collectd parser as documented in the collectd readme.


#3

Hi,

I’ve Tried what you suggested but now im gettin this error when trying to run a test, the database gets created in influxdb, but nothing is inserter.

2019-02-21T05:32:00Z I! Loaded outputs: influxdb
2019-02-21T05:32:00Z I! Tags enabled: host=mxmcamon05
2019-02-21T05:32:00Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:“mxmcamon05”, Flush Interval:10s
2019-02-21T05:32:00Z D! [agent] Connecting outputs
2019-02-21T05:32:00Z D! [agent] Attempting connection to output: influxdb
2019-02-21T05:32:00Z D! [agent] Successfully connected to output: influxdb
2019-02-21T05:32:00Z D! [agent] Starting service inputs
2019-02-21T05:32:10Z I! ignoring field of type 0x1f8b
2019-02-21T05:32:10Z I! ignoring field of type 0xf356
2019-02-21T05:32:10Z I! ignoring field of type 0xdcbe
2019-02-21T05:32:10Z I! ignoring field of type 0xdac4
2019-02-21T05:32:10Z I! ignoring field of type 0x20fb
2019-02-21T05:32:10Z I! ignoring field of type 0x123c
2019-02-21T05:32:10Z I! ignoring field of type 0x94ea
2019-02-21T05:32:10Z I! ignoring field of type 0x924b
2019-02-21T05:32:10Z I! ignoring field of type 0x79d2
2019-02-21T05:32:10Z I! ignoring field of type 0x41c0
2019-02-21T05:32:10Z I! ignoring field of type 0x3691
2019-02-21T05:32:10Z I! ignoring field of type 0x7e7
2019-02-21T05:32:10Z I! ignoring field of type 0x87fc
2019-02-21T05:32:10Z I! ignoring field of type 0x8b33
2019-02-21T05:32:10Z I! ignoring field of type 0x66cd
2019-02-21T05:32:10Z I! ignoring field of type 0xfc20
2019-02-21T05:32:10Z I! ignoring field of type 0x504
2019-02-21T05:32:10Z I! ignoring field of type 0x8af1
2019-02-21T05:32:10Z I! ignoring field of type 0x115b
2019-02-21T05:32:10Z I! ignoring field of type 0x8944
2019-02-21T05:32:10Z I! ignoring field of type 0xd215
2019-02-21T05:32:10Z I! ignoring field of type 0x6a39
2019-02-21T05:32:10Z I! ignoring field of type 0xaeb6
2019-02-21T05:32:10Z E! [inputs.file]: Error in plugin: Collectd parser error: invalid length 61516

And this is my config file:

[global_tags]
[agent]
interval = “10s”
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 100000
collection_jitter = “0s”
flush_interval = “10s”
flush_jitter = “0s”
precision = “”
debug = true
quiet = false
logfile = “/var/lib/zabbix/UnixMonitoring/telegraf/telegraf/var/log/telegraf/telegraf.log”
hostname = “”
omit_hostname = false
[[outputs.influxdb]]
urls = [“http://localhost:8096”]
database = “telegraf”
skip_database_creation = false
[[inputs.file]]
files = ["/var/lib/zabbix/collectlfiles/**.gz"]
data_format = “collectd”
collectd_typesdb = ["/usr/share/collectd/types.db"]
collectd_parse_multivalue = “split”

influxdb version 1.7.1
telegraf version 1.9.4

Thank you for your help.