[needHelp] Cant collect the output of lsof -i -P -n

ohlolol · March 4, 2021, 12:11pm

Hi Folks,

im using telegraf with grafana to visulize data so far it worked well for me. But i cant figure out how to get the output of the command lsof -i -P -n (list of open and connected Ports) to get saved in db.

So far i figured out how to send a single line via netcat on udp but how can i wrap the outpot of lsof to insert it via netcat?

im on debian 10, evrything up to date.

Kind regards

Ohlolol

Franky1 · March 4, 2021, 1:18pm

What do a few lines of your lsof output look like as an example?

I don’t know your setup, but here are some quick ideas.

Maybe you already reach the goal with the inputs.netstat plugin which uses lsof under the hood.
If not:
- You could read the output lines of lsof in the Value input data format as a data_type = "string" and then split the individual lines of the output into the desired parameters with a processors plugin in a second step.
- Or you could read the output of lsof in the Grok input data format to extract the parameters from the lsof output lines with the help of regular expressions.
- Or you could possibly tweak the output of lsof so that it is directly compatible with an input format (e.g. csv, logfmt, json) from Telegraf? lsof has some output lsof -F0[information needed here] options, but I don’t know enough about them. Or use something like awk in between, so that the lsof output is directly compatible with an input format of telegraf?

https://manpages.ubuntu.com/manpages/bionic/man8/lsof.8.html#output%20for%20other%20programs

ohlolol · March 4, 2021, 1:39pm

I already use netstat plugin but it dosent provide data about connected ips
this is a part of the output i like to store in influxdb:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
grafana-s 16331 grafana 8u IPv6 3693741709 0t0 TCP *:3000 (LISTEN)
grafana-s 16331 grafana 10u IPv6 543515911 0t0 TCP 185.248.140.252:3000->178.27.241.35:50443 (ESTABLISHED)
grafana-s 16331 grafana 11u IPv4 991113598 0t0 TCP 185.248.140.252:58666->35.241.23.245:443 (ESTABLISHED)
grafana-s 16331 grafana 13u IPv6 543515913 0t0 TCP 185.248.140.252:3000->178.27.241.35:50444 (ESTABLISHED)
grafana-s 16331 grafana 14u IPv6 543515914 0t0 TCP 185.248.140.252:3000->178.27.241.35:50440 (ESTABLISHED)
grafana-s 16331 grafana 15u IPv6 543515915 0t0 TCP 185.248.140.252:3000->178.27.241.35:50441 (ESTABLISHED)
grafana-s 16331 grafana 16u IPv4 987273025 0t0 TCP 185.248.140.252:39364->185.199.110.133:443 (ESTABLISHED)
grafana-s 16331 grafana 18u IPv6 543515925 0t0 TCP 185.248.140.252:3000->178.27.241.35:50442 (ESTABLISHED)
/opt/lamp 22684 daemon 4u IPv6 3460234011 0t0 TCP *:80 (LISTEN)
/opt/lamp 22684 daemon 6u IPv6 3460234019 0t0 TCP *:443 (LISTEN)

Franky1 · March 4, 2021, 1:46pm

If your lsof output always(!) has exactly the same number of columns and they are always(!) separated with a space, you could try the csv input format and use the space as separator.

ohlolol · March 4, 2021, 1:51pm

yeah i will try it with csv today.

Thank you alot! Im a professional programmer but new in linux system and influxdb

Franky1 · March 4, 2021, 10:16pm

I tried the csv input format with your data.

Input logfile:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
grafana-s 16331 grafana 8u IPv6 3693741709 0t0 TCP *:3000 (LISTEN)
grafana-s 16331 grafana 10u IPv6 543515911 0t0 TCP 185.248.140.252:3000->178.27.241.35:50443 (ESTABLISHED)
grafana-s 16331 grafana 11u IPv4 991113598 0t0 TCP 185.248.140.252:58666->35.241.23.245:443 (ESTABLISHED)
grafana-s 16331 grafana 13u IPv6 543515913 0t0 TCP 185.248.140.252:3000->178.27.241.35:50444 (ESTABLISHED)
grafana-s 16331 grafana 14u IPv6 543515914 0t0 TCP 185.248.140.252:3000->178.27.241.35:50440 (ESTABLISHED)
grafana-s 16331 grafana 15u IPv6 543515915 0t0 TCP 185.248.140.252:3000->178.27.241.35:50441 (ESTABLISHED)
grafana-s 16331 grafana 16u IPv4 987273025 0t0 TCP 185.248.140.252:39364->185.199.110.133:443 (ESTABLISHED)
grafana-s 16331 grafana 18u IPv6 543515925 0t0 TCP 185.248.140.252:3000->178.27.241.35:50442 (ESTABLISHED)
/opt/lamp 22684 daemon 4u IPv6 3460234011 0t0 TCP *:80 (LISTEN)
/opt/lamp 22684 daemon 6u IPv6 3460234019 0t0 TCP *:443 (LISTEN)

The configuration snippet of Telegraf:

[[inputs.file]]
  files = ["lsof.log"]
  data_format = "csv"
  csv_header_row_count = 1  # first line is header
  csv_delimiter = " "  # space as delimiter
  csv_trim_space = true
  name_override = "lsof"

[[outputs.file]] # only for debugging
  files = ["lsof.out"]
  influx_sort_fields = true

The snippet with the output of Telegraf in influx line protocol format:

lsof,host=localhost COMMAND="grafana-s",DEVICE=3693741709i,FD="8u",NAME="*:3000",NODE="TCP",PID=16331i,SIZE/OFF="0t0",TYPE="IPv6",USER="grafana" 1614895510000000000
lsof,host=localhost COMMAND="grafana-s",DEVICE=543515911i,FD="10u",NAME="185.248.140.252:3000->178.27.241.35:50443",NODE="TCP",PID=16331i,SIZE/OFF="0t0",TYPE="IPv6",USER="grafana" 1614895510000000000
lsof,host=localhost COMMAND="grafana-s",DEVICE=991113598i,FD="11u",NAME="185.248.140.252:58666->35.241.23.245:443",NODE="TCP",PID=16331i,SIZE/OFF="0t0",TYPE="IPv4",USER="grafana" 1614895510000000000
lsof,host=localhost COMMAND="grafana-s",DEVICE=543515913i,FD="13u",NAME="185.248.140.252:3000->178.27.241.35:50444",NODE="TCP",PID=16331i,SIZE/OFF="0t0",TYPE="IPv6",USER="grafana" 1614895510000000000
lsof,host=localhost COMMAND="grafana-s",DEVICE=543515914i,FD="14u",NAME="185.248.140.252:3000->178.27.241.35:50440",NODE="TCP",PID=16331i,SIZE/OFF="0t0",TYPE="IPv6",USER="grafana" 1614895510000000000
lsof,host=localhost COMMAND="grafana-s",DEVICE=543515915i,FD="15u",NAME="185.248.140.252:3000->178.27.241.35:50441",NODE="TCP",PID=16331i,SIZE/OFF="0t0",TYPE="IPv6",USER="grafana" 1614895510000000000
lsof,host=localhost COMMAND="grafana-s",DEVICE=987273025i,FD="16u",NAME="185.248.140.252:39364->185.199.110.133:443",NODE="TCP",PID=16331i,SIZE/OFF="0t0",TYPE="IPv4",USER="grafana" 1614895510000000000
lsof,host=localhost COMMAND="grafana-s",DEVICE=543515925i,FD="18u",NAME="185.248.140.252:3000->178.27.241.35:50442",NODE="TCP",PID=16331i,SIZE/OFF="0t0",TYPE="IPv6",USER="grafana" 1614895510000000000
lsof,host=localhost COMMAND="/opt/lamp",DEVICE=3460234011i,FD="4u",NAME="*:80",NODE="TCP",PID=22684i,SIZE/OFF="0t0",TYPE="IPv6",USER="daemon" 1614895510000000000
lsof,host=localhost COMMAND="/opt/lamp",DEVICE=3460234019i,FD="6u",NAME="*:443",NODE="TCP",PID=22684i,SIZE/OFF="0t0",TYPE="IPv6",USER="daemon" 1614895510000000000

Seems to work in principle, but might need to be adapted for practical implementation.

Topic		Replies	Views
Monitoring dump1090 Telegraf telegraf	7	1081	March 18, 2021
Telegraf input plugin that reads from localhost/bucket Telegraf telegraf , flux	2	725	March 15, 2022
Telegraf on IoT collecting metrics, outputting to Telegraf on Server? influxdb , telegraf	6	1154	May 23, 2018
Using openweathermap with influxdb,telegraf,andgrafana Telegraf	3	2524	October 4, 2021
[[inputs.exec]] Test shows data, yet none makes it to influxdb Telegraf influxdb , telegraf , grafana	4	930	September 8, 2020

[needHelp] Cant collect the output of lsof -i -P -n

Related topics