Logstash-InfluxDB plugin doesnot send complete data to InfluxDB

,
1

Hello,

I am trying to send data from Logstash(6.2.3) to influxDB(1.7.5) using Logstash-influxDB plugin.

I tried to send 10 lines of logs where 7 lines went through and 3 lines didn’t get into InfluxDB. I tried to figure out whats the odd part but couldn’t figure out the issue.

LOGS sent:

20190524193000|18|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/iconv-lite|HTTP/1.1|304|0
20190524193000|26|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/grunt-legacy-util|HTTP/1.1|200|0
20190524193000|3|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/colors/-/colors-0.6.2.tgz|HTTP/1.1|200|6289
20190524193000|30|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/js-yaml|HTTP/1.1|304|0
20190524193000|7|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/iconv-lite/-/iconv-lite-0.2.11.tgz|HTTP/1.1|200|235160
20190524193000|0|REQUEST|10.25.79.236|non_authenticated_user|GET|/api/docker/docker-login/v2/|HTTP/1.1|401|0
20190524193000|5|REQUEST|10.25.79.236|anonymous|GET|/api/docker/docker-login/v2/token|HTTP/1.1|200|0
20190524193000|5|REQUEST|173.39.45.237|content-security-builds-deployer|GET|/api/storage/content-security-builds-snapshot/ESA/phoebe-12-1-br/|HTTP/1.1|200|0
20190524193000|4|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/which/-/which-1.0.9.tgz|HTTP/1.1|200|2285

Logs didnot go through(3 logs didnot go through)

20190524193000|4|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/nopt/-/nopt-1.0.10.tgz|HTTP/1.1|200|10240
20190524193000|26|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/grunt-legacy-util|HTTP/1.1|200|0
20190524193000|7|REQUEST|10.194.20.31|cctg-npm-deployer|GET|/api/npm/cctg-npm-npm/iconv-lite/-/iconv-lite-0.2.11.tgz|HTTP/1.1|200|235160

Logstash filter

grok {
        # Enable multiple matchers
        break_on_match => false

        match => { "message" => "%{DATA:timestamp_local}\|%{NUMBER:duration}\|%{WORD:requesttype}\|%{IP:ip}\|%{DATA:username}\|%{WORD:method}\|%{DATA:resource}\|%{DATA:protocol}\|%{NUMBER:statuscode}\|%{NUMBER:bytes}" }

        # Extract repo and path
        match => { "resource" => "/%{DATA:repo}/%{GREEDYDATA:resource_path}"}

        # Extract resource name
        match => { "resource_path" => "(?<resource_name>[^/]+$)" }

        # Extract file extension
        match => { "resource_path" => "(?<resource_type>[^.]+$)" }
      }

Logstash outputs

influxdb {
      host => "***.*.*.*"
      db => "testing"
      port => 8086
      codec => "json"
      measurement => "requests"
      send_as_tags => ["ip","repo","site"]
      use_event_fields_for_data_points => true
      exclude_fields => ["@timestamp","timestamp","timestamp_object","timestamp_local","protocol","message","@version"]
      coerce_values => {
      'bytes' => 'integer'
      'duration' => 'integer'}
      }

when i tried to query influxdb

select bytes,ip,resource_path,username from requests
name: requests
time                bytes   ip            resource_path                                                username
----                ----- --------      -------------                                                --------
1558726546126000000 0     10.194.20.31 npm/cctg-npm-npm/iconv-lite                                  cctg-npm-deployer
1558726546153000000 0     10.194.20.31 npm/cctg-npm-npm/js-yaml                                     cctg-npm-deployer
1558726546160000000 0     10.25.79.236 docker/docker-login/v2/                                      non_authenticated_user
1558726546160000000 6289  10.194.20.31 npm/cctg-npm-npm/colors/-/colors-0.6.2.tgz                   cctg-npm-deployer
1558726546161000000 0     10.25.79.236 docker/docker-login/v2/token                                 anonymous
1558726546161000000 0     173.39.45.237 storage/content-security-builds-snapshot/ESA/phoebe-12-1-br/ content-security-builds-deployer
1558726551069000000 2285  10.194.20.31 npm/cctg-npm-npm/which/-/which-1.0.9.tgz                     cctg-npm-deployer

I tried enabling debug logs in logstash but that didnot help and neither does influxdb throw any errors.

Can anyone see whats missing ? Thanks