I am trying to send data from Logstash to InxluxDB but Logstash is not able to connect:
[2017-05-18T16:39:37.652000 #1] ERROR -- InfluxDB: Failed to contact host server_name:8086: #<SocketError: initialize: name or service not known> - retrying in 1s.
[2017-05-18T16:39:38.676000 #1] ERROR -- InfluxDB: Failed to contact host server_name:8086: #<SocketError: initialize: name or service not known> - retrying in 2s.
This is influxdb.conf
[reporting]
reporting-disabled = true
[meta]
dir = "/var/lib/influxdb/meta"
[data]
dir = "/var/lib/influxdb/data"
engine = "tsm1"
wal-dir = "/var/lib/influxdb/wal"
[retention]
enabled = true
check-interval = "60m"
[shard-precreation]
enabled = true
check-interval = "10m"
advance-period = "30m"
[admin]
enabled = true
bind-address = ":8083"
https-enabled = false
[monitor]
store-enabled = false
store-database = "_internal" # The destination database for recorded statistics
store-interval = "10s" # The interval at which to record statistics
[http]
enabled = true
bind-address = ":8086"
auth-enabled = false
log-enabled = true
write-tracing = false
pprof-enabled = false
https-enabled = false
max-row-limit = 10000
realm = "InfluxDB"
And this is logstash.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => ...
ssl_key => ...
}
output {
influxdb {
data_points => {}
host => "server_name:8086"
}
}
Did I miss something in config files?
Also, it’s not clear what is the name for my InfluxDB database?
Hello! The server_name
in the logstash config would need to be the domain or ip for the InfluxDB server. for example if it was available at localhost:8086
the config would be as follows:
output {
influxdb {
data_points => {}
host => "localhost:8086"
}
}
Hi. Thanks.
I tried it with domain name and I also tried it with localhost:8086
but still getting the same errors:
[2017-05-18T18:18:05.083000 #1] ERROR -- InfluxDB: Failed to contact host localhost:8086: #<SocketError: initialize: name or service not known> - retrying in 2s.
@Steven_1 Where are you running? Sometimes if you are in an environment like AWS there can be firewall rules that prevent communication between hosts. Can you give a more complete description of how you have these pieces deployed? Also can you share the influxdb logs as well?
InfluxDB logs look good:
[I] 2017-05-18T16:13:55Z InfluxDB starting, version 1.2.4, branch master, commit 77909d7c7826afe597b12d957996d6e16cd1afaa
[I] 2017-05-18T16:13:55Z Go version go1.7.4, GOMAXPROCS set to 1
[I] 2017-05-18T16:13:55Z Using configuration at: /etc/influxdb/influxdb.conf
[I] 2017-05-18T16:13:55Z Using data dir: /var/lib/influxdb/data service=store
[I] 2017-05-18T16:13:55Z /var/lib/influxdb/data/_internal/monitor/1/000000001-000000001.tsm (#0) opened in 2.064758ms engine=tsm1 service=filestore
[I] 2017-05-18T16:13:55Z reading file /var/lib/influxdb/wal/_internal/monitor/1/_00006.wal, size 0 engine=tsm1 service=cacheloader
[I] 2017-05-18T16:13:55Z /var/lib/influxdb/data/_internal/monitor/1 database index loaded in 2.229183ms service=shard
[I] 2017-05-18T16:13:55Z /var/lib/influxdb/data/_internal/monitor/1 opened in 17.699596ms service=store
[I] 2017-05-18T16:13:55Z opened service service=subscriber
[I] 2017-05-18T16:13:55Z Starting monitor system service=monitor
[I] 2017-05-18T16:13:55Z 'build' registered for diagnostics monitoring service=monitor
[I] 2017-05-18T16:13:55Z 'runtime' registered for diagnostics monitoring service=monitor
[I] 2017-05-18T16:13:55Z 'network' registered for diagnostics monitoring service=monitor
[I] 2017-05-18T16:13:55Z 'system' registered for diagnostics monitoring service=monitor
[I] 2017-05-18T16:13:55Z Starting precreation service with check interval of 10m0s, advance period of 30m0s service=shard-precreation
[I] 2017-05-18T16:13:55Z Starting snapshot service service=snapshot
[I] 2017-05-18T16:13:55Z Starting admin service service=admin
[I] 2017-05-18T16:13:55Z DEPRECATED: This plugin is deprecated as of 1.1.0 and will be removed in a future release service=admin
[I] 2017-05-18T16:13:55Z Listening on HTTP: [::]:8083 service=admin
[I] 2017-05-18T16:13:55Z Starting continuous query service service=continuous_querier
[I] 2017-05-18T16:13:55Z Starting HTTP service service=httpd
[I] 2017-05-18T16:13:55Z Authentication enabled:false service=httpd
[I] 2017-05-18T16:13:55Z Listening on HTTP:[::]:8086 service=httpd
[I] 2017-05-18T16:13:55Z Starting retention policy enforcement service with check interval of 1h0m0s service=retention
[I] 2017-05-18T16:13:55Z Listening for signals
[I] 2017-05-18T16:13:55Z Sending usage statistics to usage.influxdata.com
[I] 2017-05-18T16:19:00Z SHOW DATABASES service=query
[httpd] 10.133.224.235 - - [18/May/2017:16:19:00 +0000] "GET /query?db=&q=SHOW+DATABASES HTTP/1.1" 200 115 "server:8083/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" b3deecce-3be5-11e7-8001-000000000000 8627
[I] 2017-05-18T16:19:08Z SHOW STATS service=query
[httpd] 10.133.224.235 - - [18/May/2017:17:19:41 +0000] "GET /query?db=_internal&q=SHOW+STATS HTTP/1.1" 200 978 "server:8083/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 2de748ec-3bee-11e7-8005-000000000000 2676
[I] 2017-05-18T18:13:55Z retention policy shard deletion check commencing service=retention
I am running both InfluxDB and Logstash with Docker on the same server:
docker run -d --name influxdb -p 8086:8086 -p 8083:8083 \
-v /space/influxdb:/var/lib/influxdb \
-v /space/config/influxdb.conf:/etc/influxdb/influxdb.conf \
influxdb
docker run -d --name logstash--link influxdb:influxdb -p 5044:5044 \
-v /opt/logstash/pipeline/:/usr/share/logstash/pipeline/ \
logstash
Maybe problem with config files?
@Steven_1 Have you tried:
output {
influxdb {
data_points => {}
host => "influxdb:8086"
}
}
My knowledge of docker networking says that should work…
Same result:
[2017-05-18T23:50:32.560000 #1] ERROR -- InfluxDB: Failed to contact host influxdb:8086: #<SocketError: initialize: name or service not known> - retrying in 1s.
[2017-05-18T23:50:33.570000 #1] ERROR -- InfluxDB: Failed to contact host influxdb:8086: #<SocketError: initialize: name or service not known> - retrying in 2s.
[2017-05-18T23:50:35.573000 #1] ERROR -- InfluxDB: Failed to contact host influxdb:8086: #<SocketError: initialize: name or service not known> - retrying in 4s.
[2017-05-18T23:50:39,577][WARN ][logstash.outputs.influxdb] Connection Error while writing to InfluxDB {:exception=>#<InfluxDB::ConnectionError: Tried 3 times to reconnect but failed.>}
@Steven_1 Acording to the documentation here you just need the server_url
in the host
key, not the port. And from what I know of docker networking InfluxDB should be available at influxdb
. Have you tired:
output {
influxdb {
data_points => {}
host => "influxdb"
}
}
Thanks, It looks better now. But I can see warning logs about Non recoverable exception
.
Is it ok?
[2017-05-19T23:57:18,281][WARN ][logstash.outputs.influxdb] Non recoverable exception while writing to InfluxDB {:exception=>nil}
[2017-05-19T23:57:18,715][WARN ][logstash.outputs.influxdb] Non recoverable exception while writing to InfluxDB {:exception=>nil}
[2017-05-19T23:57:18,874][WARN ][logstash.outputs.influxdb] Non recoverable exception while writing to InfluxDB {:exception=>nil}
@Steven_1 Is there anything in the InfluxDB logs?
It looks like it shows 404 status code:
[I] 2017-05-19T23:20:29Z retention policy shard deletion check commencing service=retention
[I] 2017-05-19T23:30:29Z new shard group 4 successfully precreated for database _internal, retention policy monitor service=metaclient
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:18 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e3f93018-3cee-11e7-8001-000000000000 647
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:18 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e43ae714-3cee-11e7-8002-000000000000 124
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:18 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e453ff61-3cee-11e7-8003-000000000000 161
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:18 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e45f2f56-3cee-11e7-8004-000000000000 144
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e470a9e7-3cee-11e7-8005-000000000000 155
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e48a74c2-3cee-11e7-8006-000000000000 136
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e49a9032-3cee-11e7-8007-000000000000 169
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4b323e2-3cee-11e7-8008-000000000000 176
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4c1089b-3cee-11e7-8009-000000000000 151
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4d95d76-3cee-11e7-800a-000000000000 170
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4e71279-3cee-11e7-800b-000000000000 153
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4e9fc48-3cee-11e7-800c-000000000000 95
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4f543b1-3cee-11e7-800d-000000000000 155
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:19 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e4f9b3c8-3cee-11e7-800e-000000000000 133
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:20 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e50761d3-3cee-11e7-800f-000000000000 128
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:20 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e50fcd09-3cee-11e7-8010-000000000000 144
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:20 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e5954e49-3cee-11e7-8011-000000000000 168
[httpd] xxx.xx.x.xx - - [19/May/2017:23:57:22 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" e6ac8dfa-3cee-11e7-8012-000000000000 174
@Steven_1 That means you are trying to write malformed points. Can you share some of the example output from logstash?
I am using old InfluxDB logs to send them to Logstash:
Logstash output - to write to InfluxDB
[I] 2017-05-24T16:06:17Z Starting snapshot service service=snapshot
[I] 2017-05-24T16:06:17Z Starting admin service service=admin
[I] 2017-05-24T16:06:17Z DEPRECATED: This plugin is deprecated as of 1.1.0 and will be removed in a future release service=admin
[I] 2017-05-24T16:06:17Z Listening on HTTP: [::]:8083 service=admin
[I] 2017-05-24T16:06:17Z Starting continuous query service service=continuous_querier
[I] 2017-05-24T16:06:17Z Starting HTTP service service=httpd
[I] 2017-05-24T16:06:17Z Authentication enabled:false service=httpd
[I] 2017-05-24T16:06:17Z Listening on HTTP:[::]:8086 service=httpd
InfluxDB logs:
[httpd] xxx.xx.x.xx - - [24/May/2017:16:29:43 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" 31996992-409e-11e7-800f-000000000000 133
@Steven_1 Is there any way you can check and see what the POST requests look like to InfluxDB? Also there is a telegraf
plugin that will help you do this too.
I got this in InfluxDB logs when sending data from Telegraf:
[I] 2017-05-24T21:58:33Z CREATE DATABASE telegraf service=query
[httpd] xxx.xx.x.xx - - [24/May/2017:21:58:33 +0000] "POST /query?q=CREATE+DATABASE+telegraf HTTP/1.1" 200 62 "-" "-" 211d85a4-40cc-11e7-8001-000000000000 291174
[I] 2017-05-24T21:58:50Z /var/lib/influxdb/data/telegraf/autogen/5 database index loaded in 21.421µs service=shard
[httpd] xxx.xx.x.xx - - [24/May/2017:21:58:50 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "-" 2b617aba-40cc-11e7-8002-000000000000 133400
[httpd] xxx.xx.x.xx - - [24/May/2017:21:59:00 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "-" 31549d05-40cc-11e7-8003-000000000000 36546
[httpd] xxx.xx.x.xx - - [24/May/2017:21:59:10 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "-" 374aa72c-40cc-11e7-8004-000000000000 23596
[httpd] xxx.xx.x.xx - - [24/May/2017:21:59:20 +0000] "POST /write?db=telegraf HTTP/1.1" 204 0 "-" "-" 3d405e83-40cc-11e7-8005-000000000000 9185
That looks good! Are you seeing the data in the database?
I think I have to create database first: CREATE DATABASE "db_name"
and then forward Telegraf data to that db, correct?
@Steven_1 Telegraf will create the database for you. You can see that query in the second line in the logs above.
@jackzampolin Thank you for explanation regarding Telegraf. But do you have any ideas what’s wrong when Logstash tries to send data to InfluxDB?
[httpd] xxx.xx.x.xx - - [24/May/2017:16:29:43 +0000] "POST /write?db=statistics&precision=ms&rp=autogen HTTP/1.1" 404 47 "-" "Ruby" 31996992-409e-11e7-800f-000000000000 133