How to Set Up Remote Logging on Linux Using

rsyslog
Logging is a critical aspect of Linux server management. Log messages are useful for root cause
analysis and avoiding potential error occurrences in the future. Analyzing and debugging server errors
is a core skill to have for both IT engineers and system administrators.
This guide will show you how to set up a remote logging server, also known as a log host, on Linux. A
log host allows you to aggregate local Linux logs to a remote centralized server for ease of access and
analysis.

Why Have a Dedicated Log Server?

The Linux operating system logs most activities on your server for auditing and debugging using the
syslog (system logging protocol) daemon. So you might be wondering, why do I need a dedicated
server for my logs? Here are some advantages to having a dedicated logging server:
• Better security because the remote logging server only has a few ports open to the outside.
• Improved server performance because the remote logging host does not run many services,
except the ones used for logging.
• Eases archiving and management of log messages.
Log messages are important for auditing your servers and base-lining and are a core part of preventive
maintenance procedures on your server infrastructure.

Step 1: Installing rsyslog on Linux

This guide focuses on Ubuntu 20.04, but the process should be pretty much the same if you are using
other mainstream Linux distros.
rsyslog is a remote logging service for Linux and comes preinstalled by default on most modern Linux
distros, for example, Ubuntu and other Debian-based systems.
The rsyslog service is a modern and improved daemon to syslog, which only allows you to manage
logs locally. With the rsyslog daemon, you can send your local logs to some configured remote Linux
server.
If you do not have rsyslog installed on your PC, you can easily do so using the following command, on
Debian-based distros:
sudo apt install rsyslog

On Red Hat Linux, you can install it by typing:

 yum install rsyslog

On Fedora and its derivatives, run:

dnf install rsyslog

To install rsyslog on Arch Linux:

yay -S rsyslog

To check the status of rsyslog, run the following command:

systemctl status rsyslog

Output:

Step 2: Configuring the Log Host Server

The log host is the server configured to receive log messages from other servers or PCs. The rsyslog
configuration resides in the /etc/rsyslog.conf file.
You can open the /etc/rsyslog.conf file using any text editor of your choice. In this guide, we'll use
Vim.
You'll need elevated privileges to make changes to the config file.
Before you start editing the config file, you should take a backup or copy of the file. To do so, run the
command:
sudo cp /etc/rsyslog.conf /etc/rsyslog_original.config

Next, open the /etc/rsyslog.conf file using a text editor.

sudo vim /etc/rsyslog.conf

There are two protocols you can use for sending/receiving log files with rsyslog: TCP and UDP. This
guide shows you how to configure both.
You do not need to configure both UDP and TCP for remote logging to work. Only choose one of the
two.
If you prefer to use UDP, look for and uncomment the following lines by removing the leading Pound
(#) symbol preceding the lines. You can find these lines under the modules section of the config file.
module(load="imudp")
input(type="imudp" port="514")

If you prefer to use TCP, then uncomment the following lines by removing the leading Pound (#)
symbol located at the beginning of the lines:
module(load="imtcp")
input(type="imtcp" port="514")
The following figure shows the rsyslog configuration file configured to use UDP communication:
Next, configure the location where rsyslog will store your logs. For better organization, you should
categorize incoming logs by their origin. Define a template in your rsyslog config file by adding the
following lines:
$template remote-incoming-logs, "/var/log/remote/%HOSTNAME%".log
*.* ?remote-incoming-logs

The aforementioned lines command rsyslog to store the logs in the folder /var/log/remote/hostname,
where hostname is the name of the remote client that is sending log messages to the log host.
Now, save the changes you've made. If you are using Vim, here is how to save and quit a file.
Finally, restart the rsyslog services for the changes you've made to take effect.
sudo systemctl restart rsyslog

Step 3: Configuring Your Firewall

If your firewall is enabled, make sure that the port you have configured above is able to communicate
with the outside world. You'll need to edit your firewall rules to allow incoming logs.
For Debian-based distros, simply use the UFW tool, to enable either the UDP or TCP transfer protocol.
Related: How to Configure the Firewall in Ubuntu Using UFW
If you are using UDP, run the following command, where 514 is the configured port number:
sudo ufw 514/udp

If you are using TCP on port 514, simply run:

sudo ufw 514/tcp

On Fedora, you can use firewall-cmd to achieve similar results.

firewall-cmd --zone=zone --add-port=514/udp

For Red Hat Linux, open the iptables file located at /etc/sysconfig/iptables using your text editor of
choice, and add the following rule:
-A INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT

Restart the iptables service for the changes to take effect.

service iptables restart

Step 4: Configuring the Logging Client

The client is the machine that sends its logs to a remote or centralized log host server. Open the rsyslog
config file located at /etc/rsyslog.conf:
 sudo vim /etc/rsyslog.conf

Add the following line if you are using UDP, where 192.168.12.123 is the IP address of the remote
server, you will be writing your logs to:
*.* @192.168.12.123:514

If you are using TCP, add the following line instead. Note that the line has two @ symbols.
*.* @@192.168.12.123:514

Save your changes and restart the rsyslog service on the client with the command:
sudo systemctl restart rsyslog

Step 5: Viewing the Log Messages on the Server

You can use SSH to log in to your remote server and view the logs sent from the client servers. In this
case, rsyslog is configured so that it stores the client logs in the /var/log/remote directory of the remote
server.
cd /var/logs/remote

Then list the contents of the directory using the ls command:

ls -l

As you can see in the output, the directory contains log messages for the remote servers named andiwa
and rukuru. Their log files are named andiwa.log and rukuru.log respectively.
You can then look at the log files using a text editor or with Linux file viewing tools such as cat or less.

Remote Logging Gives You More Control

This guide has looked at how to set up a remote logging server (log host) on Linux.
A log host offers you better organization and control when it comes to logging. Even in scenarios
where a system is damaged or inaccessible, you can still view its logs from the log host and figure out
what went wrong.

Getting Started With System Logging in Linux

System logging is the most reliable way of knowing which activities were carried out on your Linux
system.
System logs in Linux provide you with great insight into core activities on your PC or server
infrastructure. They're critical for keeping your system stable and secure. System logs also provide you
with an opportunity to audit various activities that have taken place in the past.
This guide introduces you to the logging system in Linux. All the major activities carried out by core
system applications and services are recorded in the form of logs and at the heart of all this is a system
known as Syslog.

Why are System Logs Important?

Imagine that your Linux PC has recently been experiencing startup errors or you suspect that someone
has been trying to log onto your system. These events can be easily traced as your system keeps track
of such activities in the form of logs.
In Linux, system logs are human-readable records of the core system activities performed by services,
daemons, and system applications. Some of the important activities logged on a Linux machine include
user logins and login failures, operating system booting, system failures, etc.
Linux has a dedicated service known as Syslog that is specifically responsible for creating logs via the
System Logger. Syslog comprises of several components such as the Syslog Message Format, Syslog
Protocol, and the Syslog Daemon: popularly known as syslogd or rsyslogd in newer versions of Linux.
The /var/log directory stores most of the logs on a Linux system. The /var directory mostly contains
variable files and directories i.e data that is bound to change often. There is no standard format for logs
but at the minimum, logs should contain a timestamp and the details of the activity being logged.

Listing Files Managed by syslog

All general logs on your system are stored in the /var/log/syslog file on Debian-based Linux distros.
Other distributions use the /var/log/messages file for storing logs.
Note: Different Linux distros may use different files for logging specific messages. For example, on
Debian-based Linux distros, the /var/log/auth.log file contains authentication logs, while RedHat
systems use the /var/log/secure file to store such logs.
To find out more about all the files that are responsible for storing logs, you can take a look at the
/etc/rsyslog.d directory, which contains important Syslog configuration files. For example, to list
standard log files, you can take a look at the /etc/rsyslog.d/50-default.conf file.
cat /etc/rsyslog.d/50-default.conf

The file shows you the names of the system applications and the corresponding log files associated
with them.

How to Inspect Log Files

Most log files are pretty long. As such, one of the most important commands for inspecting log files on
Linux is the less command, which outputs file content in easily navigable sections.
For example, to view the contents of the /var/log/syslog file, use the less command as follows.
less /var/log/syslog
Use the F keyboard key to scroll forward and the B key to scroll backward.
The syslog file contains logs of some of the most critical activities such as system errors and service
activities on your system.
If you only want to inspect the most recent logs you can use the tail command, which only lists the last
10 log messages by default.
tail /var/log/syslog

You can also specify the number of log messages that you want to view with the tail utility. The
command takes the following format tail -n file-to-inspect, where n is the number of lines you want to
view. For example, to view the last 7 log messages in the syslog file you can use the following
command.
tail -7 /var/log/syslog

To view the most recent logs in real-time, you can use the tail command with the -f option as follows.
tail -f /var/log/syslog

Another important command for inspecting log messages is the head command. Unlike the tail
command which displays the last log messages in a file, the head command shows you the first lines in
a file. By default, the command will output the first 10 lines only.
head /var/log/syslog

Authentication Logs
If you want to find information about user logins on your system, you can take a look at the
/var/log/auth.log file. Information related to user logins, login failures, and the authentication method
used can be found here.

Kernel Logs
When your Linux system boots, important data about the kernel ring buffer is recorded in the
/var/log/dmesg file. Other information about hardware drivers, kernel, and boot status is all recorded in
this file.
Instead of inspecting the boot log messages with the less or cat command, you can use dmesg to view
these log files.
dmesg

Note: Log messages in the /var/log/dmesg file are reset whenever the system boots.
Another important log file related to kernel issues is the /var/log/kern.log.
Logging Messages With the logger Command
Apart from just viewing log messages logged by system applications or services, the logging system in
Linux also allows you to log messages manually using the logger command. A user can log messages
to the /var/log/syslog file by default. For example, to log a simple message you can run the following
command.
logger hello world!

You can now use the tail command to view the recently logged message.
tail -3 /var/log/syslog

You can even log the output of other commands with the logger command by enclosing the command
within the back-tick (`) character.
logger `whoami`

You can also use the logger command within your scripts to log important events. Use the man pages to
learn more about the logger command and its options.
man logger

Managing Log Files

As you might have noticed, there is a lot of data that gets logged on a Linux machine. Therefore, you
need to have a proper system in place to manage disk space used by the log files. In addition to that,
having a logging system ensures that you find the log messages that you are looking for easily. Linux's
solution to this problem is the logrotate utility.
Use the logrotate utility to configure what log file to keep, how long you want to keep them for,
manage the mailing of logs, and how to compress old log files, etc.
You can configure the logrotate utility with any text editor of your choice. The config file for logrotate
can be found at /etc/logrotate.conf.

Keeping Your System Robust With Logs

System logs in Linux are a great way to get insight into the major activities happening on your system
that can comprise security and overall stability of the system. Knowing how to view and analyze log
messages on a server or PC will go a long way in helping you keep your system robust.
Sometimes, users find it hard to use certain applications on their system because of the low availability
of system resources. In such situations, killing unresponsive programs can free up space on your
system's main memory.