What is the purpose of Syslog?
Syslog is used as a standard to produce, forward, and collect logs produced on a
Linux instance. Syslog defines severity levels as well as facility levels helping users
having a greater understanding of logs produced on their computers. Logs can,
later on, be analyzed and visualized on servers referred to as Syslog servers.
What is Syslog architecture?
When designing a logging architecture, as a centralized logging server, it is very
likely that multiple instances will work together.
Some will generate log messages, and they will be called “devices” or “syslog
clients“.
Some will simply forward the messages received, they will be called “relays“.
Finally, there are some instances where you are going to receive and store log
data, those are called “collectors” or “syslog servers”.
Knowing those concepts, we can already state that a standalone Linux machine
acts as a “syslog client-server” on its own: it produces log data, it
is collected by rsyslog and stored right into the filesystem.
Here’s a set of architecture examples around this principle.
In the first design, you have one device and one collector. This is the most simple
form of logging architecture out there.
Add a few more clients to your infrastructure, and you have the basis of
a centralized logging architecture.
Multiple clients are producing data and are sending it to a centralized syslog
server, responsible for aggregating and storing client data.
If we were to complexify our architecture, we can add a “relay“.
Examples of relays could be Logstash instances for example, but they also could
be rsyslog rules on the client-side.
The rsyslog service sorts and writes syslog messages to the log files that do persist across
reboots in /var/log. The rsyslog service sorts the log messages to specific log files based on
the type of program that sent each message, or facility, and the priority of each syslog
message.
In addition to syslog message files, the /var/log directory contains log files from other
services on the system. The following table lists some useful files in the /var/log directory.
Selected System Log Files
1. Display syslogs with the ls command
Listing the contents of /var/log for an Ubuntu 20.04 machine using
the ls command:
$ sudo ls /var/log
Listing /var/log
2. View system logs in Linux using the tail command
Using the tail command you can view the last few logs. Adding the -f
option lets you watch them in real time.
For RedHat based systems:
$ sudo tail -f /var/log/messages
For Ubuntu/Debian based systems:
$ sudo tail -f /var/log/syslog
Similarly, the tail command can be used to view kernel logs (kern.log),
boot logs (boot.log), etc .
The rules for which logs go where are defined in the Syslog daemon’s
configuration file. For rsyslog, it is /etc/rsyslog.conf
3. View and Edit syslogs in Linux with a text editor
Let’s look at rsyslog‘s configuration file using the nano editor:
$ sudo nano /etc/rsyslog.conf
As can be seen in the screenshot, it uses imjournal module to read the
messages from the journal. Scrolling through the file, the rules for the
location of logs can be seen:
Server Configuration for Remote System Logging
We will be configuring a CentOS 8 machine as the remote server that
receives Syslog messages from hosts through TCP. You’ll
need superuser privileges for every step. So, either change to the root
user or prefix sudo before every command.
1. Check if rsyslog is installed
rsyslog is the Syslog daemon that will listen for logs from host. To check
if it’s installed, type:
$ rsyslogd -v
It will print some information if it’s installed
If it is not already installed, you can install it using the dnf command:
$ sudo dnf install rsyslog
2. Edit rsyslog’s configuration file
The file we need to modify is /etc/rsyslog.conf. You can use the editor
of your choice. I’ll be using the nano editor.
$ sudo nano /etc/rsyslog.conf
You can also group the logs by creating separate directories for separate
client systems using what rsyslog calls ‘templates’. These templates are
directives for rsyslog.
To enable grouping of logs by systems add lines 7 and 8. To enable
TCP, uncomment lines 4 and 5 by deleting the ‘#’ character at the start
of the line.
1
...
2 # Provides TCP syslog reception
3 # for parameters see http://www.rsyslog.com/doc/imtcp.html
4 module(load="imtcp") # needs to be done just once
5 input(type="imtcp" port="514")
7 $template FILENAME,"/var/log/%HOSTNAME%/syslog.log"
*.* ?FILENAME
8
...
9
3. Configure the firewall to open the port used by rsyslog
By default rsyslog listens on port 514. We need to open this port using
the firewall-cmd command:
$ sudo firewall-cmd --add-port=514/tcp --zone=public --permanent
$ sudo firewall-cmd --reload
4. Restart rsyslog
Now that we’ve made changes to the configuration file and opened the
port, we need to restart rsyslog so that it can pick up the new
configuration. We can restart rsyslog using the systemctl command:
$ sudo systemctl restart rsyslog
If you want rsylog to automatically start every time you boot up, type:
$ sudo systemctl enable rsyslog
5. Check if rsyslog is listening on the port opened
We can use the netstat command to list all the open ports:
$ sudo netstat -pnlt
As is highlighted in the screenshot above, rsyslog is listening on port
514.
