Linux Troubleshooting,
Performance, and Security
From Linux+ Guide to Linux Certification, 3e
’19H2
송인식
Objectives
• Describe and outline good troubleshooting practices
• Effectively troubleshoot common hardware- and
software-related problems
• Monitor system performance using command-line and
graphical utilities
• Identify and fix common performance problems
• Describe the different facets of Linux security
• Increase the security of a Linux computer
• Outline measures and utilities that can be used to
detect a Linux security breach
Linux Troubleshooting, Performance,
2
and Security
Troubleshooting Methodology
Figure 14-1: The maintenance cycle
Linux Troubleshooting, Performance,
3
and Security
Troubleshooting Methodology
(continued)
• Monitoring: observing log files and running
performance utilities system to identify problems and
their causes
• Proactive maintenance: minimizing chance of future
problems
– e.g., perform regular system backups
• Reactive maintenance: correcting problems when they
arise
– Documenting solutions
– Developing better proactive maintenance methods
Linux Troubleshooting, Performance,
4
and Security
Troubleshooting Methodology (continued)
• Documentation: system information stored in a log
book for future references
– All maintenance actions should be documented
• Troubleshooting procedures: tasks performed when
solving system problems
Linux Troubleshooting, Performance,
5
and Security
Troubleshooting Methodology (continued)
Figure 14-2: Common troubleshooting procedures
Linux Troubleshooting, Performance,
6
and Security
Troubleshooting Methodology (continued)
• Two troubleshooting golden rules:
– Prioritize problems according to severity
• Spend reasonable amount of time on each problem given its
priority
• Ask for help if you can’t solve the problem
– Try to solve the root of the problem
• Avoid missing underlying cause
• Justify why a certain solution is successful
Linux Troubleshooting, Performance,
7
and Security
Resolving Common System Problems
• Three categories of problems:
– Hardware-related
– Software-related
– User interface-related
Linux Troubleshooting, Performance,
8
and Security
Hardware-Related Problems
• Often involve improper hardware or software
configuration
– SCSI termination
– Video card and monitor configuration
– All hardware is on Hardware Compatibility List
– POST test alerts
• Loose hardware connections
– Problems specific to the type of hardware
• View output of dmesg command
• View content of /var/log/boot.log, /var/log/messages
Linux Troubleshooting, Performance,
9
and Security
Hardware-Related Problems (continued)
• Absence of device drivers prevent OS from using
associated devices
– dmesg command: displays the hardware that is detected by
the Linux kernel
– lsusb command: displays a list of USB devices detected by
the Linux kernel
– lspci command: displays a list of PCI devices detected by
the Linux kernel
– Compare outputs of commands to output of lsmod to
determine if driver module is missing from kernel
Linux Troubleshooting, Performance,
10
and Security
Hardware-Related Problems (continued)
• Hardware failure can render a device unusable
– HDDs most common hardware components to fail
– If HDD containing partitions mounted on noncritical
directories fails:
• Power down computer and replace failed HDD
• Boot Linux system
• Use fdisk to create partitions on replaced HDD
• Use mkfs to create filesystems
• Restore original data
• Ensure /etc/fstab has appropriate entries to mount filesystems
Linux Troubleshooting, Performance,
11
and Security
Hardware-Related Problems (continued)
• If HDD containing / filesystem fails:
– Power down computer and replace failed HDD
– Reinstall Linux on new HDD
– Restore original configuration and data files
Linux Troubleshooting, Performance,
12
and Security
Software-Related Problems: Application-
Related Problems
• Missing program libraries/files, process restrictions, or
conflicting applications
• Dependencies: prerequisite shared libraries or
packages required for program execution
– Programs usually check at installation
– Package files may be removed accidentally
Linux Troubleshooting, Performance,
13
and Security
Software-Related Problems: Application-
Related Problems (continued)
• rpm –V command: identify missing files in a package
or package dependency
• ldd command: display shared libraries used by a
program
• ldconfig command: updates list of shared library
directories (/etc/ld.so.conf) and list of shared libraries
(/etc/ld.so.cache)
Linux Troubleshooting, Performance,
14
and Security
Software-Related Problems: Application-
Related Problems (continued)
• Too many running processes
– –Solve by killing parent process of zombie processes
• Filehandles: connections programs make to files
• ulimit command: modify process limit parameters in
current shell
– Can also modify max number of file handles
Linux Troubleshooting, Performance,
15
and Security
Software-Related Problems: Application-
Related Problems (continued)
• /var/log directory: contains most system log files
– Some are hard linked to /var/log directory
• If applications stop functioning due to difficulty
gaining resources, restart using SIGHUP
– Do determine if another process trying to access the same
resources attempt to start application in Single User Mode
• If resource conflict is the cause of the problem, download newer
version of application or application fix
Linux Troubleshooting, Performance,
16
and Security
Software-Related Problems:
Operating System-Related Problems
• Most software-related problems related to OS
– X windows, boot loader, and filesystem problems
• Problem detecting video card or monitors by the
kernel
– To isolate problem starting X Windows or gdm:
• View /var/log/Xorg.0.log file
• Execute xwininfo or xdpyinfo
Linux Troubleshooting, Performance,
17
and Security
Software-Related Problems:
OS-Related Problems (continued)
• LILO problems: place “linear” in, remove “compact”
from /etc/lilo.conf file
• GRUB problems: typically result of missing files in
/boot directory
• Ensure Linux kernel resides before 1024th cylinder and
lba32 keyword is in configuration file
– Eliminates BIOS problems with large HDDs
Linux Troubleshooting, Performance,
18
and Security
Software-Related Problems:
OS-Related Problems (continued)
• If filesystem on partition mounted to noncritical
directory becomes corrupted:
– Unmount filesystem
– Run fsck command with –f (full) option
– If fsck command cannot repair filesystem, use mkfs
command to re-create the filesystem
– Restore filesystem’s original data
Linux Troubleshooting, Performance,
19
and Security
Software-Related Problems:
OS-Related Problems (continued)
• If / filesystem is corrupted:
– Boot from Fedora installation media and enter System Rescue
– At shell prompt within System Rescue:
• Use mkfs to recreate the filesystem
• Use backup utility to restore original data to the re-created /
filesystem
– Exit System Rescue and reboot system
• Knoppix Linux and BBC Linux: bootable Linux
distributions with many filesystem repair utilities
Linux Troubleshooting, Performance,
20
and Security
Software-Related Problems:
User Interface-Related Problems
• Assistive technologies: tools that users can use to
modify their desktop experience
– Assistive Technologies Preference utility within GNOME
Desktop Environment
• Preferred Applications to configure Web browser, multimedia
player and terminal applications to be opened automatically
• Mouse Accessibility to configure speed and click behavior
• Keyboard Accessibility to configure keyboard related assistive
technologies
Linux Troubleshooting, Performance,
21
and Security
Software-Related Problems:
User Interface-Related Problems (continued)
Figure 14-3: The Assistive Technologies Preferences utility
Linux Troubleshooting, Performance,
22
and Security
Performance Monitoring
• Jabbering: failing hardware components send large
amounts of information to CPU
• Other causes of poor performance:
– Software monopolizes system resources
– Too many processes
– Too many read/write requests to HDD
– Rogue processes
Linux Troubleshooting, Performance,
23
and Security
Performance Monitoring (continued)
• To solve software performance issues:
– Remove software from the system
– Move software to another Linux system
– Add CPU or otherwise alter hardware
• Bus mastering: peripheral components perform tasks
normally executed by CPU
Linux Troubleshooting, Performance,
24
and Security
Performance Monitoring (continued)
• To increase performance:
– Add RAM
– Upgrade to faster HDDs
– Disk Striping RAID
– Keep CD/DVD drives on a separate HDD controller
• Run performance utilities on a regular basis
– Record results in a system log book
– Eases identification of performance problems
• Baseline: measure of normal system activity
Linux Troubleshooting, Performance,
25
and Security
Monitoring Performance with sysstat
Utilities
• System Statistics (sysstat) package: contains wide
range of system monitoring utilities
– Use yum install sysstat command to install
• mpstat (multiple processor statistics) command:
displays CPU statistics
– Used to monitor CPU performance
– Can specify interval and number of measurements rather than
displaying average values
– %sys should be smaller than %usr and %nice combined
Linux Troubleshooting, Performance,
26
and Security
Monitoring Performance with sysstat
Utilities
• iostat (Input/Output Statistics) command: measures
flow of information to and from disk devices
– Displays CPU statistics similar to mpstat
– Displays statistics for each disk device on the system
– Output includes:
• Transfers per second
• Number of blocks read and written per second
• Total number of blocks read and written for the device
Linux Troubleshooting, Performance,
27
and Security
Monitoring Performance with sysstat
Utilities
• sar (System Activity Reporter) command: displays
various system statistics taken in the last day
– Provides more information than mpstat and iostat
– By default scheduled to run every 10 minutes
• Output logged to a file in /var/log/sa directory
– -f option: View statistics from a specific file
– Can be used to take current system measurements
Linux Troubleshooting, Performance,
28
and Security
Monitoring Performance with sysstat
Utilities
• Additional sar options:
– -q option: Displays processor queue statistics
• runq -sz value: Number of processes waiting for execution on
processor run queue
• plist -sz value: Indicates number of processes currently
running
• ldavg values: Represent average CPU load
– -W option: Displays number of pages sent to and taken from
swap partition
• Large number causes slower performance
• Add RAM to resolve
Linux Troubleshooting, Performance,
29
and Security
Monitoring Performance with sysstat
Utilities
Table 14-1: Common options to the sar command
Linux Troubleshooting, Performance,
30
and Security
Other Performance Monitoring Utilities
• top command: displays CPU statistics, swap usage,
memory usage and average CPU load
• free command: displays total amounts of physical
and swap memory and their utilizations
– Can be used to indicate whether more physical memory is
required
• vmstat command: displays memory, CPU, and swap
statistics
– Can be used to indicate whether more physical memory is
required
Linux Troubleshooting, Performance,
31
and Security
Security
• Linux systems typically made available across networks
such as the Internet
– More prone to security loopholes and attacks
• Should improve local and network security
• Understand how to detect intruders who breach the
system
Linux Troubleshooting, Performance,
32
and Security
Securing the Local Computer
• Limit access to physical computer itself
– Prevent malicious users from accessing files by directly
booting the computer with their own device
• Server closet: secured room to store servers
• Remove floppy, CD, and DVD drives from workstations
• Ensure BIOS prevents booting from USB ports
Linux Troubleshooting, Performance,
33
and Security
Securing the Local Computer (continued)
• Ensure BIOS password is set
• Set boot loader password in LILO or GRUB
configuration file
– Prevents intruder from interacting with boot loader
• Limit access to graphical desktops and shells
– Exit command-line shell before leaving computer
• nohup command: prevents background processes from being
killed when parent shell is killed or exited
– Lock screen using GNOME or KDE
Linux Troubleshooting, Performance,
34
and Security
Securing the Local Computer (continued)
• Minimize root user’s time logged in
• su (switch user) command: switch current user account
to another
– Used to switch between root user and regular user
• sudo command: perform commands as another user if
you have the rights to do that listed in /etc/sudoers
file
Linux Troubleshooting, Performance,
35
and Security
Protecting Against Network Attacks
• Always a possibility that hackers can manipulate a
network service by interacting with it in unusual ways
• Buffer overrun: program information for a network
service altered in memory
Linux Troubleshooting, Performance,
36
and Security
Network Security Essentials
• Minimize number of running network services
• nmap (network mapper) command: scans ports on
network computers
– User can determine what network services are running
• Ensure that services that are not needed are not
automatically started when entering the runlevel
Linux Troubleshooting, Performance,
37
and Security
Network Security Essentials (continued)
• Ensure network service daemons for essential services
not run as root user when possible
• Ensure that shell listed in /etc/passwd for daemons is
set to /sbin/nologin
– Hacker will not be able to get BASH shell
• New network service versions usually include fixes for
known network attacks
– Keep network services up-to-date
Linux Troubleshooting, Performance,
38
and Security
Network Security Essentials (continued)
• TCP wrapper: program that can start a network
daemon
– Checks /etc/hosts.allow and /etc/hosts.deny files before
starting a network daemon
• Examine permissions for files and directories
associated with system and network services
Linux Troubleshooting, Performance,
39
and Security
Configuring a Firewall
• netfilter/iptables: used to configure a firewall
– Discard network packets according to chains of rules
• Chains: specify general type of network traffic to apply
rules to
• Rules: match network traffic to be allowed or dropped
• Three chain types:
– INPUT: incoming packets
– FORWARD: packets passing through computer
– OUTPUT chain: outgoing packets
Linux Troubleshooting, Performance,
40
and Security
Configuring a Firewall (continued)
• iptables command: creates rules for a chain
– Can be based on source IP, destination IP, protocol used, or
packet status
• Stateful packet filter: Remembers traffic allowed in an
existing session and adjust rules appropriately
• Easier to use graphical utility to configure firewalls
Linux Troubleshooting, Performance,
41
and Security
Configuring a Firewall (continued)
Table 14-2: Common iptables options
Linux Troubleshooting, Performance,
42
and Security
Configuring a Firewall (continued)
Figure 14-4: The Firewall Configuration utility
Linux Troubleshooting, Performance,
43
and Security
Configuring SELinux
• SELinux: Security Enhanced Linux
– By default, configured and enabled during Fedora installation
– Series of kernel patches and utilities created by NSA
• Enforces role-based security
• To enable, edit /etc/selinux/config file
• Configure SELINUXTYPE option
• Reboot and relabel the system
• sestatus command: view current SELinux status
Linux Troubleshooting, Performance,
44
and Security
Using Encryption to Protect Network Data
• Use encryption algorithms to protect data before it is
transmitted on a network
• Asymmetric encryption: uses a pair of keys uniquely
generated on each system
– Public key: freely distributed
– Private key: used only by the system, never distributed
– Can be used to authenticate messages
• Digital signature: message that has been encrypted
using a private key
Linux Troubleshooting, Performance,
45
and Security
Working with SSH
• By default, SSH uses RSA to encrypt data and DSA to
digitally sign data
• System wide RSA and DSA key pairs are generated the
first time SSH daemon is started
– Tunneling: enclosing network traffic within encrypted SSH
packets
• SSH identity: used to automatically authenticate to
other computers using digital signatures
• Manage keys using Password and Encryption Keys
utility
Linux Troubleshooting, Performance,
46
and Security
Working with SSH (continued)
Figure 14-5: The Passwords and Encryption Keys utility
Linux Troubleshooting, Performance,
47
and Security
Working with GPG
• Open source version of PGP
• Each user has a key pair used for encryption and
authentication
– –Authentication uses trust model
• Typically uses RSA and DSA key pairs for asymmetric
encryption and digital signing
• Can manage GPG keys and encrypt data using:
– gpg command
– Graphical utility such as Passwords and Encryption Keys utility
Linux Troubleshooting, Performance,
48
and Security
Detecting Intrusion
• Log files can contain information or irregularities
indicating an intrusion
– Review log files in /var/log associated with network services
– At minimum, review system log files associated with
authentication
• Pluggable Authentication Module (PAM): handles
authentication requests by network applications
– Log file in /var/log/secure
Linux Troubleshooting, Performance,
49
and Security
Detecting Intrusion (continued)
• Check /var/log/wtmp log file
– Lists users who receive BASH shells
– Use who command to view the file
• lsof (list open files) command: lists files that are
currently being edited
• Periodically search for files that have SUID bit set
• Tripwire: monitors important files and directories
• Intrusion Detection System (IDS): program used to
detect intruders on a Linux system
Linux Troubleshooting, Performance,
50
and Security
Detecting Intrusion (continued)
Table 14-3: Common Linux Intrusion Detection Systems
Linux Troubleshooting, Performance,
51
and Security
Summary
• Administrators monitor the system, perform
proactive/reactive maintenance, and document system
information
• Common troubleshooting procedures involve:
– Isolating and determining the cause of system problems and
implementing and testing solutions that can be documented
for future use
• Invalid hardware settings, absence of device drivers,
and hard disk failure are common hardware-related
problems
Linux Troubleshooting, Performance,
52
and Security
Summary (continued)
• Software-related problems can be application-related
or OS-related
• Users can use assistive technologies to modify their
desktop experience
• System performance is affected by a variety of
hardware and software factors
– Using performance monitoring utilities to create a baseline is
helpful for diagnosing future performance problems
Linux Troubleshooting, Performance,
53
and Security
Summary (continued)
• Securing a Linux computer involves:
– Improving local and network security and monitoring to
detect intruders
• Greatly improve local security by:
– Restricting access to the computer and using root account
only when required via su and sudo commands
Linux Troubleshooting, Performance,
54
and Security
Summary (continued)
• Reduce chance of network attacks by:
– Reducing number of network services, implementing firewalls,
SELinux, service updates, encryption, and TCP wrappers, and
restricting services from running as root user and permissions
on key files
• Analyzing log files and key system files and running
IDS applications can be used to detect intruders
Linux Troubleshooting, Performance,
55
and Security
Questions?
Linux Troubleshooting, Performance,
56
and Security
