Telegraf is not sending data from suricata-stats.sock to InfluxDBv2

Telegraf
,
1
  • OPNsense 21.7-amd64
  • os-telegraf 1.11.0 (Telegraf 1.19.0)
  • InfluxDB 2.0.7
  • Suricata version 5.0.7 RELEASE running in SYSTEM mode

Telegraf is not sending metrics to InfluxDBv2 related to [[inputs.suricata]]. I do not know why nothing is happening. All other metrics are OK (default System metrics; swap/cpu/disk/pf/etc). I was hoping I could skip Grafana and just use InfluxDBv2 dashboard functionality.

Steps to reproduce:

  1. SSH into OPNsense box. As root, service suricata stop & service telegraf stop.

  2. Edit /usr/local/etc/telegraf.conf and add:

[[inputs.suricata]]
  ## Data sink for Suricata stats log.
  # This is expected to be a filename of a
  # unix socket to be created for listening.
  source = "/tmp/suricata-stats.sock"

  # Delimiter for flattening field keys, e.g. subitem "alert" of "detect"
  # becomes "detect_alert" when delimiter is "_".
  delimiter = "_"

Using service telegraf start, a UNIX-socket has been created in the /tmp directory:

file /tmp/suricata-stats.sock
/tmp/suricata-stats.sock: socket

Suricata Fails to Create Socket - #2 by vjulien - Help - Suricata say that;

  1. suricata doesn’t create the unix socket, but instead if connects to an existing one.

Suricata is started as root, so there are no permission issues for the connection from suricata to this socket.

  1. Add a new eve-log output in /usr/local/etc/suricata/suricata.yaml:
outputs:
  - eve-log:
      enabled: yes
      filetype: unix_stream
      filename: /tmp/suricata-stats.sock
      types:
        - stats:
           threads: yes
  1. Issuing service suricata start yields no error-messages:
service suricata start
Starting suricata.
26/7/2021 -- 23:28:33 - <Info> - Including configuration file installed_rules.yaml.
26/7/2021 -- 23:28:33 - <Info> - Configuration node 'rule-files' redefined.
26/7/2021 -- 23:28:33 - <Info> - Including configuration file custom.yaml.

However, there are no metrics populated in InfluxDBv2, and I do not know why. I do not know how to “sniff” in on a UNIX-socket and the size of the .sock always stays at 0B.

However, stopping telegraf and trying to create a UNIX-socket with the user telegraf sudo -u telegraf nc -U -l /tmp/suricata-stats.sock gives me a lot of data ("event_type":"stats"), so the stream is flowing.

Also verified by sockstat when using the telegraf service;

sockstat | grep suri
root     suricata   93358 3  dgram  -> /var/run/logpriv
root     suricata   93358 4  stream -> /tmp/suricata-stats.sock

By the looks of it, it seems that noone has ever got this to work, only the authors over at influxdata/telegraf/tree/master/plugins/inputs/suricata

  • I am not allowed to post more than 2 links as I am a new user
  • I am not allowed to post more than 2 links as I am a new user
  • I am not allowed to post more than 2 links as I am a new user
  • I am not allowed to post more than 2 links as I am a new user

PS: Yes, as this is OPNsense based on BSD, I have increased the localhost buffer space stated in README.md;

Under FreeBSD it is necessary to increase the localhost buffer space to at least 16384, default is 8192 otherwise messages from Suricata are truncated as they exceed the default available buffer space, consequently no statistics are processed by the plugin.

sysctl -a | grep local.stream
net.local.stream.recvspace: 16384
net.local.stream.sendspace: 16384
2

Hm. The last bit got me thinking. I have captured the output from sudo -u telegraf nc -U -l /tmp/suricata-stats.sock and searching for "timestamp" throughout the document, it looks as though messages from Suricata are truncated.

The start of the output:
{"timestamp":"2021-07-29T16:26:14.841116+0200","event_type":"stat (…)

and throughout the output I have
,"invalid{"timestamp":"2021-0 (…)
0,"pseud{"timestam(…)
mcap_drop":0,"ps{"timestamp (…)
emcap_drop"{"timestamp":"2021-07-2

If {"timestamp" is indeed a new eve-log entry, I would like to think I have to further increase the localhost buffer space, because it does not look like suricata has enough space to send the whole event for telegraf to process. Let us try with 24576 (3 times the default 8192) and maybe this post could be marked as [SOLVED].

There are no errors in telegraf (/var/log/telegraf.log) depicting this.

3

Yes, the buffer space was not large enough. How large it should be, depends on your firewall (number of interfaces).

A method to figure out the correct size of the bufferspace, is to get suricata to write to a file by replacing filetype : unix_stream with type : unix_stream in suricata.yaml.

You will have to stop telegraf or use another filename : in suricata.yaml.

After a while, you will see that suricata writes to this file. Copy one whole event, from {"timestamp:" to the next {"timestamp": (one event should end with }}}}), to a file.

If the file is for example 124230 bytes long, you should have the buffer space in BSD set to bit larger size than this.

Then you avoid getting truncated, and the [[inputs.suricata]] plugin will parse the event correctly.