Logstash cannot connect to InfluxDB

I’m guessing the database statistics does not exist. influxdb returns a 404 on /write when the given database name doesn’t match.

$ curl -i -XPOST localhost:8086/write?db=does_not_exist --data-binary "m n=1i"
HTTP/1.1 404 Not Found
Content-Type: application/json
Request-Id: bb2b3873-4623-11e7-8196-000000000000
X-Influxdb-Version: unknown
Date: Wed, 31 May 2017 17:08:13 GMT
Content-Length: 51

{"error":"database not found: \"does_not_exist\""}

I created database “logstash” and updated logstash.conf to send data into that database:

input {
  beats {
    port => 5044
  }
}

filter {
}

output {
  influxdb {
    codec => json
    data_points => {}
    host => "influxdb"
    db => "logstash"
  }
  stdout {codec => json}
}

But I still get errors:

[httpd] xxx.xx.x.xx - - [31/May/2017:17:22:38 +0000] "POST /write?db=logstash&precision=ms&rp=autogen HTTP/1.1" 400 4171 "-" "Ruby" bea8251e-4625-11e7-83d6-000000000000 3890

I think you don’t want the codec => json line inside the influxdb block. We cut over from JSON to line protocol prior to the 1.0 release.

Unfortunately, that didn’t help.

To diagnose further, we’ll need to see either what Logstash is sending in the request body, or the response that InfluxDB is sending back. A 400 status code can mean many different things.

1 Like

I am confused how I can do that.
Could a reason be in Logstash InfluxDB output plugin itself? But in that case it wouldn’t try to write data to InfluxDB at all.

I have figured out Logstash is not able to write Metricbeat data to InfluxDB due to invalid field format.
Any ideas what could be wrong in field format?

"2017-06-06T17:29:41.676000+0000", :message=>"Error writing to InfluxDB", :response=>#<Manticore::Response:0x2dea8269 
@message="Bad Request", @callback_result="{\"error\":\"unable to parse 'logstash  1496770179090': 
invalid field format\\nunable to parse 'logstash  1496770179090': invalid field format\\nunable to parse 'logstash  1496770179091': 
invalid field format\\nunable to parse 'logstash  1496770179091': invalid field format\\nunable to parse 'logstash  1496770179091': 
invalid field format\\nunable to parse 'logstash  1496770179091': invalid field format\\nunable to parse 'logstash  1496770179091': 
...

{"@timestamp":"2017-06-06T17:29:39.091Z","type":"metricsets","system":{"process":{"ppid":2,"memory":{"size":0,"rss":{"bytes":0,"pct":0.000000},"share":0},
"state":"sleeping","fd":{"open":0,"limit":{"hard":4096,"soft":1024}},"pid":809,"username":"root","cpu":{"total":{"pct":0.000000},
"start_time":"2016-02-28T21:38:32.000Z"},"pgid":0,"name":"jbd2/dm-2-8"}},"metricset":{"module":"system","name":"process","rtt":106444},
"beat":{"name":"host_name","hostname":"host_name","version":"5.3.0"},"@version":"1",
"host":"host_name","tags":["beats_input_raw_event"]}{"type":"metricsets","system":{"process":{"name":"ext4-dio-unwrit","pgid":0,
"cpu":{"total":{"pct":0.000000},"start_time":"2016-02-28T21:38:32.000Z"},"fd":{"open":0,"limit":{"soft":1024,"hard":4096}},"ppid":2,"state":"sleeping",
"memory":{"size":0,"rss":{"bytes":0,"pct":0.000000},"share":0},"username":"root","pid":810}},"metricset":{"module":"system","name":"process","rtt":106444},
"beat":{"version":"5.3.0","name":"host_name","hostname":"host_name"},"@timestamp":"2017-06-06T17:29:39.091Z",
"@version":"1","host":"host_name","tags":["beats_input_raw_event"]}{"@timestamp":"2017-06-06T17:29:39.091Z","type":"metricsets",
"system":{"process":{"username":"root","memory":{"share":0,"size":0,"rss":{"bytes":0,"pct":0.000000}},"cpu":{"total":{"pct":0.000000},
"start_time":"2016-02-28T21:38:32.000Z"},"pid":849,"ppid":2,"fd":{"open":0,"limit":{"hard":4096,"soft":1024}},"pgid":0,"name":"kauditd","state":"sleeping"}},
"metricset":{"rtt":106444,"module":"system","name":"process"},"beat":{"name":"host_name","hostname":"host_name","version":"5.3.0"},
"@version":"1","host":"host_name","tags":["beats_input_raw_event"]}{"beat":{"version":"5.3.0","name":"host_name",
"hostname":"host_name"},"@timestamp":"2017-06-06T17:29:39.091Z","type":"metricsets","system":{"process":{"state":"sleeping","memory":
...

logstash 1496770179091 has two spaces between the measurement name and the timestamp. Normal formatting would look like logstash x=123,y=456 1496770179091 but it looks like the fields were output as an empty string for some reason.

I don’t know enough about Logstash or Metricbeat to debug further. Maybe there’s a way to opt in to emitting certain fields?

I am not sure how I can figure that out.
Btw, the same configuration works good when writing data to Elasticsearch but fails when trying to write to InfluxDB.

If nobody else here knows an answer, you may be able to get a quicker solution from the maintainers of Logstash’s influxdb output plugin. InfluxDB seems to be working correctly; the output plugin is emitting malformed output.

If you find out a solution, please post back to this thread for anyone else who may experience the same issue.