Syslog plugin from remote server

Telegraf
1

I have plans to pull in logs from our remote servers but struggle with integrating the syslog plugin. My first remote machine is debian Jessie and have installed telegraf 1.7 to send data to my homebrew test influxdb.
I see my homebrew system and at least my remote system metrics so that implies that on my remote system /etc/telegraf.conf has its [[outputs.influxdb]] set correctly.

I have followed the readme in telegraf/plugins/inputs/syslog at master · influxdata/telegraf · GitHub
I would like to use UDP to send logs directly from rsyslog but I just want to get it to work. Lots of config files in play here.

I have opened the 8086 and 6514 ports open between my remote machine to my homebrew influxdb.

Using the Readme instructions I placed this into /etc/rsyslog.d/50-telegraf.conf

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode

$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down

# forward over tcp with octet framing according to RFC 5425
*.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

The last line has alternated from localhost to my homebrew influx server. No difference.

I’ve seen this post where everything is on localhost but abandoned it since I’m wanting to collect logs from a remote machine.

Added /etc/telegraf/telegraf.d/inputs-syslog-plugin.conf
In [[inputs.syslog]] I have alternated between
server = "tcp://:6514"
and
server = "tcp://myhomebrewserver:6514"

Nothing.
So on the remote box I ran
telegraf --config /etc/telegraf/telegraf.d/inputs-syslog-plugin.conf

E! Error: no outputs found, did you provide a valid config file?

OK, In this file I added:
[[outputs.syslog]]
url = “http://myhomebrewserver:8086”
database = “telegraf”
username = “myusername”
password = “mypwd”

Which failed as expected.
Reading this forum it seems that telegraf isn’t set up to push data directly remotely, at least in the master branch.

How is this done? What am I missing?, sam

2

Can you paste your full config?
That [[outputs.syslog]] will fail because there is no syslog output plugin. For your input, you’ll want server = tcp://:6514, then in your syslog config, you’ll use the accessible ip address for that telegraf host. You’ll then need to configure the address of your [[outpus.influxdb]] to the accessible address of that host.

3

Thank you for replying. Makes sense if theres not syslog output plugin. I’m puzzled by your phrase “then in your syslog config you’ll use the accessible ip address for that telegraf host.” I’m sure you are right, but which context is this? Telegraf running on the remote server? Not Telegraf running on myhomebrewserver?

I had said the ports 6514 and 8086 are open. My telnet tests suggest not.
from remote to myhomebrewserver
6514 fails, but 8086 works
from myhomebrewserver to remote
6514 appears to connect briefly, 8086 fails

On my remote box I have added these rules to iptables
-A OUTPUT -d myhomebrewserverIP -p tcp -m tcp --dport 8086 -j ACCEPT

17682 -A OUTPUT -d myhomebrewserverIP -p udp -m udp --dport 8086 -j ACCEPT

17683 -A OUTPUT -d myhomebrewserverIP -p tcp -m tcp --dport 6514

17684 -A OUTPUT -d myhomebrewserverIP -p udp -m udp --dport 6514

Which config would be helpful to see past what I’ve shown so far? Which context? remote sever or myhomebrewserver?
I’m guessing my /etc/telegraf/telegraf.conf on my remote server. Here is that config stripped of most of the commented areas.

Configuration for telegraf agent

[agent]
interval = “10s”
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = “0s”
flush_jitter = “0s”
logfile = “”
omit_hostname = false

###############################################################################

OUTPUT PLUGINS

###############################################################################

Configuration for sending metrics to InfluxDB

[[outputs.influxdb]]
urls = [“http://myhomebrewserver:8086”]
database = “telegraf”
skip_database_creation = false
username = “myusername”
password = “myuserpwd”

###############################################################################

PROCESSOR PLUGINS

###############################################################################

###############################################################################

AGGREGATOR PLUGINS

###############################################################################

###############################################################################

INPUT PLUGINS

###############################################################################

Read metrics about cpu usage

[[inputs.cpu]]
report_active = false
[[inputs.disk]]
ignore_fs = [“tmpfs”, “devtmpfs”, “devfs”]

[inputs.diskio]]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]

###############################################################################

SERVICE INPUT PLUGINS

###############################################################################

Of course, the remote server also has a config folder it checks. In /etc/telegraf/telegraf.d/50-syslog.conf I show:
[[inputs.syslog]]
server=tcp://:6514

That is all that is uncommented.

Thanks for your time.

4

To make some progress on this, I renamed this supplemental config file found in /etc/telegraf/telegraf.d/ to /etc/telegraf/telegraf.d/renamed.conf.unused and even commented out the call to server.
I know that telegraf is reading from /etc/telegraf/telegraf.d/ because it successfully is reading my apache.conf.

I then went back to /etc/telegraf/telegraf.conf and uncommented [[inputs.syslog]] and server=tcp://6514

service telegraf restart
telegraf --config /etc/telegraf/telegraf.conf
returns E! Service for input inputs.syslog failed to start, exiting
listen tcp :6514: bind: address already in use

sudo netstat -tulpn | grep 6514
returns only one entry and telegraf is using 6514.

Commented out the inputs.syslog again in /etc/telegraf/telegraf.conf and restarted telegraf.
Un-commented the inputs.syslog and server=tcp:6514
but did NOT restart telegraf service.
telegraf --config /etc/telegraf/telegraf.conf
output shows:
I! Loaded inputs: inputs.cpu inputs.disk inputs.mem inputs.swap inputs.syslog i Inputs.diskio inputs.kernel inputs.processes inputs.system
2018-09-24T12:36:58Z I! Loaded aggregators:
2018-09-24T12:36:58Z I! Loaded processors:
2018-09-24T12:36:58Z I! Loaded outputs: influxdb
2018-09-24T12:36:58Z I! Tags enabled: host=myremotehostname
2018-09-24T12:36:58Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"myremotehostname", Flush Interval:10s

Of course, this is just the config, not actual telegraf. Restarting telegraf returns the port already open error.

/etc/rsyslog.d/50-telegraf.conf has only the modules enabled by the syslog input README.
#forward over tcp with octet framing according to RFC 5425
#. @@(o)myinfluxbox:6514;RSYSLOG_SyslogProtocol23Format

I have searched telegraf.conf, rsyslog.conf, /etc/rsyslog.d/50-telegraf.conf and find only one mention of port 6514 and that’s in the main telegraf.conf file mentioned above. I’ve spent many days trying things but it has never sent ANY log files to my Chronograph.
What to try next?

5

If I’m understanding you properly, you have some remote machines running telegraf that you’d like to report metrics, including logs, to a centralized server running influxdb.

Each remote server should be able to talk to your “homebrew server” via port 8086, and influxdb should be listening and accessible on this port to those remote servers.

The output section of your telegraf config looks correct:

[[outputs.influxdb]]
  urls = ["http://myhomebrewserver:8086"]
  database = "telegraf"
  skip_database_creation = false
  username = "myusername"
  password = "myuserpwd"

You can continue to collect other metrics (diskio, kernel, mem, …etc), but the question is about syslog. Since telegraf is running on all your remote machines, your syslog input is also correct (notice you can listen on localhost since it’s only gathering it’s own syslogs):

[[inputs.syslog]]
  server="tcp://localhost:6514"

Each server’s rsyslog.conf file should include configuration to submit logs to the local telegraf:

*.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

Using TCP/UDP simply requires adjusting the config in rsyslog.conf and the server key in the telegraf config.

This should get each remote machine’s syslog to report to it’s local telegraf, which then reports to the centralized influxdb.

6

Thank you for replying and giving some feedback on the config.

These are the changes I’ve made.
/etc/rsyslog.conf
I’ve commented out the calls to the UDP module and UDPServerRun and uncommented its equivalents in TCP.
$ModLoad imtcp
$InputTCPServerRun 514

Updated /etc/rsyslog/rsyslog.d/50-telegraf.con
*.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format
Thank you.

Restarted rsyslog

Rebuilt this line in /etc/telegraf/telegraf.conf
server = "tcp://localhost:6514"
Thank you.

This is the output of
telegraf --config /etc/telegraf/telegraf.conf

E! Unable to append to /etc/telegraf/telegraf.log (open /etc/telegraf/telegraf.log: permission denied), using stderr
2018-09-26T17:56:26Z D! Attempting connection to output: influxdb
2018-09-26T17:56:26Z D! Successfully connected to output: influxdb
2018-09-26T17:56:26Z I! Starting Telegraf v1.7.4
2018-09-26T17:56:26Z I! Loaded inputs: inputs.cpu inputs.swap inputs.system inputs.syslog inputs.disk inputs.diskio inputs.kernel inputs.mem inputs.processes
2018-09-26T17:56:26Z I! Loaded aggregators:
2018-09-26T17:56:26Z I! Loaded processors:
2018-09-26T17:56:26Z I! Loaded outputs: influxdb
2018-09-26T17:56:26Z I! Tags enabled: host=myremotebox
2018-09-26T17:56:26Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"myremotebox", Flush Interval:10s
2018-09-26T17:56:26Z E! Service for input inputs.syslog failed to start, exiting
listen tcp 127.0.0.1:6514: bind: address already in use

My effort to chown the telegraf.log to telegraf:telegraf to root:root fails. I do not know which user:group will work.

Based on the issue pages of the syslog plugin in Github, it was suggested to use UDP which is why I’ve attempted to do so. Both UDP and TCP return the same error in its equivalent form.

As before, Debian is reporting via netstat -tulpn | grep 6514
tcp 0 0 127.0.0.1:6514 0.0.0.0:* LISTEN 4685/telegraf
Only one entry and my previous post applies. Restarting telegraf runs everything except the syslog. Clearly telegraf cannot start syslog plugin because telegraf has beaten itself to it. The only other place where I find 6514 in a config file is this commented out line above my server declaration.
## Specify an ip or hostname with port - eg., tcp://localhost:6514, tcp ://10.0.0.1:6514

7

Is that process 4685 another telegraf service that you can stop/kill? It looks like once you end it, your new command should work.

8

Yes, except it seems to be the legitimate one from telegraf.
kill -9 4685
then netstat -tupln | grep 6514 returns a higher pid.
telegraf --config /etc/telegraf/telegraf.conf
returns the same response prior and the syslog plugin fails to run.
It seems that telegraf on this remote box is set to run on this port but when it gets to the inputs.syslog and its call to 6514 it fails. For the life of me, I’m unaware of anyother config file that’s in play. My /etc/telegraf/telegraf.d/50-apache-plugin.conf does not have any port listed and it’s running fine.
From your description, my homebrewinflux machine only needs port 8086 open to receive telegraf data. Perhaps 6514 is local only to this remote Debian machine?
It seems based on your response that ryslog is pushing logs to localhost:6514 and telegraf is listening on localhost:6514. But it’s telegraf reporting that it’s already running 6514 so rsyslog and telegraf must use the same port to communicate.
As I’ve said, searching the live telegraf config files for 6514 report only the single instance I’ve reported.

9

You must have another telegraf service already running. search for and disable the other one while you test out your new config. Sometimes it’s as simple as service telegraf stop

10

I actually accomplished this in a very simple manor: I enabled the local input.syslog collector in my telegraph.conf file, and then simply set the output to be a remote instance of InfluxDB. Now all my syslog logs are going to the remote instance. It made debugging the local syslog config easy since I could check the local InfluxDB instance to make sure it was getting syslog logs, then once that was working, I pointed it at the remote InfluxDB instance and was done.

HTH,
dg

11

Hello Sam452,
Did you start telegraf before with systemctl start telegraf? If yes and you try to kill its PID the systemd will automatically start it again. So please try stop running telegraf with systemctl stop telegraf and systemctl disable telegraf and after this check 6514 port with netstat. It is not in use you will able to start telegraf “by hand” with:
telegraf --config /etc/telegraf/telegraf.conf

12

I’ve used service instead of systemctl directly. Disabling telegraf from systemctl does seem to prevent this error from showing up when I run
telegraf --config /etc/telegraf/telegraf.conf
Thank you.

Question: I suspect that when this is test is over that I should re-enable telegraf in systemctl and note that if it’s running as a service that when I run the telegraf --config I should expect the port error again? If I follow you, that’s because it would seem that I’m running a new instance of telegraf by this --config test?

One other mea culpa here that others like me may find helpful: I’ve been looking at my Chronograph Host List to see syslog added to my list. I see system, then apache when I got them running. I’ve been making these attempts expecting to see “syslog” in this host list for myremotebox host. Apparently, I won’t see it there. Thankfully, during this test I do see myremotebox logs in the “Log Viewer”.

Again, thanx. sam

13

Hello Sam,
Service command just a wrapper in your case. It calls systemctl command in the background. Many distributions kept service command because backward compatibility. So I suggest to use systemctl.
After you enable and start telegraf again with systemctl and you want to test your config use telegraf command like this:
telegraf --debug --config /etc/telegraf/telegraf.conf --test
The --test flag is the key for you and --debug flag could be very usefull to test new config.
Don’t worry inputs.syslog plugin isn’t supported in test mode:
WARNING: skipping plugin [[inputs.syslog]]: service inputs not supported in --test mode
If you have any question about this please let me know.