Logstash cannot connect to InfluxDB

I am trying to send data from Logstash to InxluxDB but Logstash is not able to connect:

[2017-05-18T16:39:37.652000 #1] ERROR -- InfluxDB: Failed to contact host server_name:8086: #<SocketError: initialize: name or service not known> - retrying in 1s.
[2017-05-18T16:39:38.676000 #1] ERROR -- InfluxDB: Failed to contact host server_name:8086: #<SocketError: initialize: name or service not known> - retrying in 2s.

This is influxdb.conf

[reporting]
  reporting-disabled = true

[meta]
  dir = "/var/lib/influxdb/meta"

[data]
  dir = "/var/lib/influxdb/data"
  engine = "tsm1"
  wal-dir = "/var/lib/influxdb/wal"

[retention]
  enabled = true
  check-interval = "60m"

[shard-precreation]
  enabled = true
  check-interval = "10m"
  advance-period = "30m"

[admin]
  enabled = true
  bind-address = ":8083"
  https-enabled = false

[monitor]
  store-enabled = false
  store-database = "_internal" # The destination database for recorded statistics
  store-interval = "10s" # The interval at which to record statistics

[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = false
  log-enabled = true
  write-tracing = false
  pprof-enabled = false
  https-enabled = false
  max-row-limit = 10000
  realm = "InfluxDB"

And this is logstash.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => ...
    ssl_key => ...
  }

output {
  influxdb {
    data_points => {}
    host => "server_name:8086"
  }
}

Did I miss something in config files?
Also, it’s not clear what is the name for my InfluxDB database?

Hello! The server_name in the logstash config would need to be the domain or ip for the InfluxDB server. for example if it was available at localhost:8086 the config would be as follows:

output {
  influxdb {
    data_points => {}
    host => "localhost:8086"
  }
}

Hi. Thanks.
I tried it with domain name and I also tried it with localhost:8086 but still getting the same errors:

[2017-05-18T18:18:05.083000 #1] ERROR -- InfluxDB: Failed to contact host localhost:8086: #<SocketError: initialize: name or service not known> - retrying in 2s.

@Steven_1 Where are you running? Sometimes if you are in an environment like AWS there can be firewall rules that prevent communication between hosts. Can you give a more complete description of how you have these pieces deployed? Also can you share the influxdb logs as well?

InfluxDB logs look good:

[I] 2017-05-18T16:13:55Z InfluxDB starting, version 1.2.4, branch master, commit 77909d7c7826afe597b12d957996d6e16cd1afaa
[I] 2017-05-18T16:13:55Z Go version go1.7.4, GOMAXPROCS set to 1
[I] 2017-05-18T16:13:55Z Using configuration at: /etc/influxdb/influxdb.conf
[I] 2017-05-18T16:13:55Z Using data dir: /var/lib/influxdb/data service=store
[I] 2017-05-18T16:13:55Z /var/lib/influxdb/data/_internal/monitor/1/000000001-000000001.tsm (#0) opened in 2.064758ms engine=tsm1 service=filestore
[I] 2017-05-18T16:13:55Z reading file /var/lib/influxdb/wal/_internal/monitor/1/_00006.wal, size 0 engine=tsm1 service=cacheloader
[I] 2017-05-18T16:13:55Z /var/lib/influxdb/data/_internal/monitor/1 database index loaded in 2.229183ms service=shard
[I] 2017-05-18T16:13:55Z /var/lib/influxdb/data/_internal/monitor/1 opened in 17.699596ms service=store
[I] 2017-05-18T16:13:55Z opened service service=subscriber
[I] 2017-05-18T16:13:55Z Starting monitor system service=monitor
[I] 2017-05-18T16:13:55Z 'build' registered for diagnostics monitoring service=monitor
[I] 2017-05-18T16:13:55Z 'runtime' registered for diagnostics monitoring service=monitor
[I] 2017-05-18T16:13:55Z 'network' registered for diagnostics monitoring service=monitor
[I] 2017-05-18T16:13:55Z 'system' registered for diagnostics monitoring service=monitor
[I] 2017-05-18T16:13:55Z Starting precreation service with check interval of 10m0s, advance period of 30m0s service=shard-precreation
[I] 2017-05-18T16:13:55Z Starting snapshot service service=snapshot
[I] 2017-05-18T16:13:55Z Starting admin service service=admin
[I] 2017-05-18T16:13:55Z DEPRECATED: This plugin is deprecated as of 1.1.0 and will be removed in a future release service=admin
[I] 2017-05-18T16:13:55Z Listening on HTTP: [::]:8083 service=admin
[I] 2017-05-18T16:13:55Z Starting continuous query service service=continuous_querier
[I] 2017-05-18T16:13:55Z Starting HTTP service service=httpd
[I] 2017-05-18T16:13:55Z Authentication enabled:false service=httpd
[I] 2017-05-18T16:13:55Z Listening on HTTP:[::]:8086 service=httpd
[I] 2017-05-18T16:13:55Z Starting retention policy enforcement service with check interval of 1h0m0s service=retention
[I] 2017-05-18T16:13:55Z Listening for signals
[I] 2017-05-18T16:13:55Z Sending usage statistics to usage.influxdata.com
[I] 2017-05-18T16:19:00Z SHOW DATABASES service=query
[httpd] 10.133.224.235 - - [18/May/2017:16:19:00 +0000] "GET /query?db=&q=SHOW+DATABASES HTTP/1.1" 200 115 "server:8083/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" b3deecce-3be5-11e7-8001-000000000000 8627
[I] 2017-05-18T16:19:08Z SHOW STATS service=query
[httpd] 10.133.224.235 - - [18/May/2017:17:19:41 +0000] "GET /query?db=_internal&q=SHOW+STATS HTTP/1.1" 200 978 "server:8083/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 2de748ec-3bee-11e7-8005-000000000000 2676
[I] 2017-05-18T18:13:55Z retention policy shard deletion check commencing service=retention

I am running both InfluxDB and Logstash with Docker on the same server:

docker run -d --name influxdb -p 8086:8086 -p 8083:8083 \
    -v /space/influxdb:/var/lib/influxdb \
    -v /space/config/influxdb.conf:/etc/influxdb/influxdb.conf \
    influxdb

docker run -d --name logstash--link influxdb:influxdb -p 5044:5044 \
    -v /opt/logstash/pipeline/:/usr/share/logstash/pipeline/ \
    logstash

Maybe problem with config files?

@Steven_1 Have you tried:

output {
  influxdb {
    data_points => {}
    host => "influxdb:8086"
  }
}

My knowledge of docker networking says that should work…

Same result:

[2017-05-18T23:50:32.560000 #1] ERROR -- InfluxDB: Failed to contact host influxdb:8086: #<SocketError: initialize: name or service not known> - retrying in 1s.
[2017-05-18T23:50:33.570000 #1] ERROR -- InfluxDB: Failed to contact host influxdb:8086: #<SocketError: initialize: name or service not known> - retrying in 2s.
[2017-05-18T23:50:35.573000 #1] ERROR -- InfluxDB: Failed to contact host influxdb:8086: #<SocketError: initialize: name or service not known> - retrying in 4s.
[2017-05-18T23:50:39,577][WARN ][logstash.outputs.influxdb] Connection Error while writing to InfluxDB {:exception=>#<InfluxDB::ConnectionError: Tried 3 times to reconnect but failed.>}

@Steven_1 Acording to the documentation here you just need the server_url in the host key, not the port. And from what I know of docker networking InfluxDB should be available at influxdb. Have you tired:

output {
  influxdb {
    data_points => {}
    host => "influxdb"
  }
}

Thanks, It looks better now. But I can see warning logs about Non recoverable exception.

Is it ok?

[2017-05-19T23:57:18,281][WARN ][logstash.outputs.influxdb] Non recoverable exception while writing to InfluxDB {:exception=>nil}
[2017-05-19T23:57:18,715][WARN ][logstash.outputs.influxdb] Non recoverable exception while writing to InfluxDB {:exception=>nil}
[2017-05-19T23:57:18,874][WARN ][logstash.outputs.influxdb] Non recoverable exception while writing to InfluxDB {:exception=>nil}

@Steven_1 Is there anything in the InfluxDB logs?

It looks like it shows 404 status code:

[I] 2017-05-19T23:20:29Z retention policy shard deletion check commencing service=retention
[I] 2017-05-19T23:30:29Z new shard group 4 successfully precreated for database _internal, retention policy monitor service=metaclient

[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:18 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e3f93018-3cee-11e7-8001-000000000000 647
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:18 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e43ae714-3cee-11e7-8002-000000000000 124
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:18 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e453ff61-3cee-11e7-8003-000000000000 161
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:18 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e45f2f56-3cee-11e7-8004-000000000000 144
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e470a9e7-3cee-11e7-8005-000000000000 155
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e48a74c2-3cee-11e7-8006-000000000000 136
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e49a9032-3cee-11e7-8007-000000000000 169
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4b323e2-3cee-11e7-8008-000000000000 176
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4c1089b-3cee-11e7-8009-000000000000 151
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4d95d76-3cee-11e7-800a-000000000000 170
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4e71279-3cee-11e7-800b-000000000000 153
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4e9fc48-3cee-11e7-800c-000000000000 95
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4f543b1-3cee-11e7-800d-000000000000 155
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4f9b3c8-3cee-11e7-800e-000000000000 133
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:20 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e50761d3-3cee-11e7-800f-000000000000 128
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:20 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e50fcd09-3cee-11e7-8010-000000000000 144
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:20 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e5954e49-3cee-11e7-8011-000000000000 168
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:22 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e6ac8dfa-3cee-11e7-8012-000000000000 174

@Steven_1 That means you are trying to write malformed points. Can you share some of the example output from logstash?

I am using old InfluxDB logs to send them to Logstash:

Logstash output - to write to InfluxDB

[I] 2017-05-24T16:06:17Z Starting snapshot service service=snapshot
[I] 2017-05-24T16:06:17Z Starting admin service service=admin
[I] 2017-05-24T16:06:17Z DEPRECATED: This plugin is deprecated as of 1.1.0 and will be removed in a future release service=admin
[I] 2017-05-24T16:06:17Z Listening on HTTP: [::]:8083 service=admin
[I] 2017-05-24T16:06:17Z Starting continuous query service service=continuous_querier
[I] 2017-05-24T16:06:17Z Starting HTTP service service=httpd
[I] 2017-05-24T16:06:17Z Authentication enabled:false service=httpd
[I] 2017-05-24T16:06:17Z Listening on HTTP:[::]:8086 service=httpd

InfluxDB logs:

[httpd] xxx.xx.x.xx - - [24/May/2017:16:29:43 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" 31996992-409e-11e7-800f-000000000000 133

@Steven_1 Is there any way you can check and see what the POST requests look like to InfluxDB? Also there is a telegraf plugin that will help you do this too.

I got this in InfluxDB logs when sending data from Telegraf:

[I] 2017-05-24T21:58:33Z CREATE DATABASE telegraf service=query
[httpd] xxx.xx.x.xx - - [24/May/2017:21:58:33 +0000] "POST /query?q=CREATE+DATABASE+telegraf HTTP/1.1" 200 62 "-" "-" 211d85a4-40cc-11e7-8001-000000000000 291174
[I] 2017-05-24T21:58:50Z /var/lib/influxdb/data/telegraf/autogen/5 database index loaded in 21.421µs service=shard
[httpd] xxx.xx.x.xx - - [24/May/2017:21:58:50 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "-" 2b617aba-40cc-11e7-8002-000000000000 133400
[httpd] xxx.xx.x.xx - - [24/May/2017:21:59:00 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "-" 31549d05-40cc-11e7-8003-000000000000 36546
[httpd] xxx.xx.x.xx - - [24/May/2017:21:59:10 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "-" 374aa72c-40cc-11e7-8004-000000000000 23596
[httpd] xxx.xx.x.xx - - [24/May/2017:21:59:20 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "-" 3d405e83-40cc-11e7-8005-000000000000 9185

That looks good! Are you seeing the data in the database?

I think I have to create database first: CREATE DATABASE "db_name" and then forward Telegraf data to that db, correct?

@Steven_1 Telegraf will create the database for you. You can see that query in the second line in the logs above.

@jackzampolin Thank you for explanation regarding Telegraf. But do you have any ideas what’s wrong when Logstash tries to send data to InfluxDB?

[httpd] xxx.xx.x.xx - - [24/May/2017:16:29:43 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" 31996992-409e-11e7-800f-000000000000 133