2020-04-25T16:41:16Z E! [telegraf] Error running agent: listen unix /var/run/suricata-stats.sock: bind: address already in use

Telegraf
1

Following the steps to get Suricata to feed data via a unix socket (unix_stream) for Telegraf to read, I added eve-log config lines in suricata.yaml to send data to /var/run/suricata-stats(dot)sock. Did a test by cat /var/run/suricata-stats(dot)sock. There are lots of data coming out.

Then putting the input(dot)suricata lines in telegraf(dot)conf. After restarting telegraf service, I see this line:

2020-04-25T16:41:16Z E! [telegraf] Error running agent: listen unix /var/run/suricata-stats(dot)sock: bind: address already in use

For some reason telegraf could not read from the socket. Could you help?

Really couldn’t wait to be able to visualize data from suricata.
Thanks!

Steps followed:
[Network Security Monitoring with Suricata and Telegraf | InfluxData](http://Influxdata article on Suricata and Telegraf)

Correction for the instructions on page:

  1. sock location in /var/run/suricata-stats.sock. Notice part of the notes says /tmp/suricata-stats.sock
  2. input statement for telegraf.conf:
    Windows Terminal
    [[inputs.suricata]]

Notice original article says [[input.suricata]]

2

===============Software versions=========
suricata-4.1.6
Name : suricata
Version : 4.1.6
Installed on : Wed Jan 29 16:07:34 2020 EST
Origin : security/suricata
Architecture : FreeBSD:11:amd64
Prefix : /usr/local
Categories : security
Licenses : GPLv2
Maintainer : franco@opnsense.org
Comment : High Performance Network IDS, IPS and Security Monitoring engine
Options :slight_smile:

==============================================
telegraf-1.14.1
Name : telegraf
Version : 1.14.1
Installed on : Fri Apr 24 11:46:09 2020 EDT
Origin : net-mgmt/telegraf
Architecture : FreeBSD:11:amd64
Prefix : /usr/local
Categories : net-mgmt
Licenses : MIT
Maintainer : girgen@FreeBSD.org
Comment : Time-series data collection
Options :
PIE : on
RELRO : on
Annotations :
FreeBSD_version: 1102000
repo_type : binary
repository : OPNsense
Flat size : 63.4MiB

====================================
FreeBSD 11.2-RELEASE-p16-HBSD FreeBSD 11.2-RELEASE-p16-HBSD fc65add89c3(stable/20.1) amd64

5

Hello @TIG,
Welcome!
Sorry if this is obvious, but it seems like there is a port conflict. Can you please look a list of all process is listening on ports and verify that there isn’t a conflict?

6

Short version: yes the socket is opened by suricata to write. Telegraf, however, should be able to read. But it could not. Help Forum!

Long version:
Great question! I was looking for lsof but found equivalents in FreeBSD:

root@OPNsense:~ # fstat /var/run/suricata-stats.sock
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W NAME
root suricata 80019 5 / 1364983 -rw-r–r-- 695303 w /var/run/suricata-stats.sock
root@OPNsense:~ # sockstat | grep suri
root suricata 80019 3 dgram -> /var/run/logpriv
root@OPNsense:~ #

Looks like Suricata has the socket opened (to write eve-log!). Other process still could access, for example #cat /var/run/suricata-stats.sock

Are you suggesting Telegraf was complaining not being able to read from the .sock because a process (suricata) had it open to write? I thought UNIX sockets are there to enable reading while it’s opened by a process that writes to it. See diagram below:

image

Thanks!!

7

FreeBSD tuning

Under FreeBSD it is necessary to increase the localhost buffer space to at least 16384, default is 8192 otherwise messages from Suricata are truncated as they exceed the default available buffer space, consequently no statistics are processed by the plugin.

sysctl -w net.local.stream.recvspace=16384
sysctl -w net.local.stream.sendspace=16384

Increase both net.local.stream.recvspace and net.local.stream.sendspace through tunables in OPNsense. Apply, reboot and the verify the through syctl -a | grep net.local.stream.

Anyways, a culprit might be User/group separation (security.bsd): System hardening vs performance — OPNsense documentation

How are you able to cat a .sock? Should you not have to use netcat?
nc -U /var/run/suricata-stats.sock to view data.

And what about permissions;
Do a id telegraf and a ls -l /var/run/suricata.sock.

If you could view the data in /tmp/suricata-stats.sock (as root) - is the user telegraf able to view the data? Try: sudo -u telegraf nc -U /var/run/suricata-stats.sock

Try the following; stop the services:
service suricata stop
service telegraf stop

Edit /usr/local/etc/suricata/suricata.yaml and add an additional eve-log output:

outputs:
  - eve-log:
      enabled: yes
      filetype: unix_stream
      filename: /tmp/suricata-stats.sock
      types:
        - stats:
           threads: yes

Should go like this:

service suricata start
Starting suricata.
29/7/2021 -- 14:03:58 - <Info> - Including configuration file installed_rules.yaml.
29/7/2021 -- 14:03:58 - <Info> - Configuration node 'rule-files' redefined.
29/7/2021 -- 14:03:58 - <Info> - Including configuration file custom.yaml.

Nothing is created in the /tmp directory at my machine. I have a theory nothing is created before a rule is triggered by Suricata. I’ll wait a bit before I start the service telegraf and then I’ll see if something happens.

Edit: ls -alh /tmp/s* shows that no suricata-stas.sock file has been created. I’ve listed the /tmp directory before and after I saw the file (a new entry was created) /var/log/suricata/eve.json tas updated.

I am on

  • OPNsense 21.1.8_1-amd64
  • Suricata version 5.0.6 RELEASE running in SYSTEM mode

Looks like my installation does not even have a suricata user, the Intrusion Detection service is running as root.

root@opnsense:/var/run # id suricata
id: suricata: no such user
root@opnsense:/var/run # ps aux | grep suricata
root      8222   2.9 19.4 4452872 1585200  -  Ss   14:03       3:02.31 /usr/local/bin/suricata -D --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
8

Some further progress - it should read (??) type: unix_stream, not filetype: unix_stream. Now my /tmp/suricata-stats.sock was created.

Edit /usr/local/etc/suricata/suricata.yaml and add an additional eve-log output:

outputs:
  - eve-log:
      enabled: yes
      type: unix_stream
      filename: /tmp/suricata-stats.sock
      types:
        - stats:
           threads: yes

I was able to view the output with more, so forget about what I wrote about netcat.

ls -alh /tmp/suricata-stats.sock 
-rw-r--r--  1 root  wheel   286K Jul 29 14:33 /tmp/suricata-stats.sock

You’ll notice that the user telegraf will not have the possibility to view this file.

 id telegraf
uid=884(telegraf) gid=884(telegraf) groups=884(telegraf),62(proxy)

Add the user telegraf to the wheel group with pw group mod wheel -m telegraf

9

Telegraf creates a listen socket?

If you do service suricata start before you do service telegraf start, you’ll notice an error in the /var/log/telegraf/telegraf.log:

2021-07-29T12:43:36Z I! Starting Telegraf 1.19.0
2021-07-29T12:43:36Z E! [telegraf] Error running agent: starting input inputs.suricata: listen unix /tmp/suricata-stats.sock: bind: address already in use

If you service suricata stop, and then rm /tmp/suricata-stats.sock - followed by a service telegraf start- you’ll see

root@opnsense:/var/run # ls -alh /tmp/suricata-stats.sock
srwxr-xr-x  1 telegraf  wheel     0B Jul 29 14:44 /tmp/suricata-stats.sock

And then, if you try to start suricata:

Jul 29 14:48:24 opnsense suricata[11956]: [100428] <Notice> -- This is Suricata version 5.0.7 RELEASE running in SYSTEM mode 
Jul 29 14:48:24 opnsense suricata[51194]: [100601] <Error> -- [ERRCODE: SC_ERR_FOPEN(44)] - Error opening file: "/tmp/suricata-stats.sock": Operation not supported

Try to chmod 775 /tmp/suricata-stats.sock- you will get the same “Operation not supported” error.

Maybe it should read filetype: unix_stream , and not what I altered it to be; type: unix_stream. Now suricata starts without any errors. I’ll wait and see if data is beiing sent…

With type: unix_stream in suricata.yaml, suricata creates a file, not a socket. What I wrote above was wrong, it should actually be filetype: unix_stream:

file /tmp/suricata-stats.sock
/tmp/suricata-stats.sock: ASCII text, with very long lines

Further investigating this topic, I found this statement over at Suricata Fails to Create Socket - Help - Suricata :

  1. suricata doesn’t create the unix socket, but instead if connects to an existing one.

Opening a new terminal window as root, running sudo -u telegraf nc -U -l /tmp/suricata-stats.sock creates
srwxr-xr-x 1 telegraf wheel 0B Jul 29 15:21 /tmp/suricata-stats.sock

This is indeed a socket:

file /tmp/suricata-stats.sock
/tmp/suricata-stats.sock: socket

And watching the netcat terminal, I see that I get data from suricata! Progress. If you do ls /tmp/s*, you will notice even though as data is flowing, the socket always stays at 0Bytes.

Using sockstat, you can see that a stream is connected to suricata-sockstat.sock:

root@opnsense:/var/run # sockstat | grep suri
root     suricata   93358 3  dgram  -> /var/run/logpriv
root     suricata   93358 4  stream -> /tmp/suricata-stats.sock

Now. Why does the metric never show up in InfluxDBv2? All other metrics are showing.