1

Kingdom of Saudi Arabia

Ministry of Higher Education
Al-Imam Muhammad Ibn Saud Islamic University
College of Computer and Information Sciences

IS 380 Cybersecurity
Chapter 3: Operating Systems Security
Fall 2022

1
Operating System Functions

2
Chapter 3: Operating System Security
OS Layered Design

3
Chapter 3: Operating System Security
Operating System Security

• Possible for a system to be compromised during the

installation process before it can install the latest patches
• Building and deploying a system should be a planned
process designed to counter this threat
• Process must:
• Assess risks and plan the system deployment
• Secure the underlying operating system and then the key
applications
• Ensure any critical content is secured
• Ensure appropriate network protection mechanisms are used
• Ensure appropriate processes are used to maintain security

4
Chapter 3: Operating System Security
Operating System Security
The first step in
deploying a new system
is planning

Plan needs to identify

Planning should
appropriate
include a wide
personnel and
security assessment
training to install and
of the organization
manage the system

Planning process
needs to determine
Aim is to maximize
security requirements
security while
for the system,
minimizing costs
applications, data,
and users 5
Chapter 3: Operating System Security
Operating System Security Process
• The purpose of the system, the type of information stored, the
applications and services provided, and their security requirements
• The categories of users of the system, the privileges they have, and
the types of information they can access
• How the users are authenticated
• How access to the information stored on the system is managed
• What access the system has to information stored on other hosts,
such as file or database servers, and how this is managed
• Who will administer the system, and how they will manage the
system (via local or remote access)
• Any additional security measures required on the system, including
the use of host firewalls, anti-virus or other malware protection
mechanisms, and logging

6
Chapter 3: Operating System Security
Operating System Hardening
• First critical step in securing a system is to secure the base
operating system
• Basic steps
• Install and patch the operating system
• Harden and configure the operating system to adequately address the
indentified security needs of the system by:
• Removing unnecessary services, applications, and
protocols
• Configuring users, groups, and permissions
• Configuring resource controls
• Install and configure additional security controls, such as anti-virus, host-
based firewalls, and intrusion detection system (IDS)
• Test the security of the basic operating system to ensure that the steps
taken adequately address its security needs 7
Chapter 3: Operating System Security
Security Maintenance

• Process of maintaining security is continuous

• Security maintenance includes:
• Monitoring and analyzing logging information
• Performing regular backups
• Recovering from security compromises
• Regularly testing system security
• Using appropriate software maintenance processes to patch and
update all critical software, and to monitor and revise configuration
as needed

8
Chapter 3: Operating System Security
Data Backup and Archive

Performing regular backups of data is a critical control that assists

with maintaining the integrity of the system and user data

May be legal or operational requirements for the retention of data

9
Chapter 3: Operating System Security
Virtualization
• With virtualization, the OS presents each user with just
the resources that user should see
• The user has access to a virtual machine (VM), which
contains those resources
• The user cannot access resources that are available to
the OS but exist outside the VM
• A hypervisor, or VM Monitor, is the software that
implements a VM
• Translates access requests between the VM and the OS
• Can support multiple OSs in VMs simultaneously
• Honeypot: A VM meant to lure an attacker into an
environment that can be both controlled and monitored
10
Chapter 3: Operating System Security
Kernelized Design
• A kernel is the part of the OS that performs the lowest-
level functions
• Synchronization
• Interprocess communication
• Message passing
• Interrupt handling
• A security kernel is responsible for enforcing the security
mechanisms of the entire OS
• Typically contained within the kernel

11
Chapter 3: Operating System Security
Kernelized Design

12
Chapter 3: Operating System Security
 i
u n
s t
a
i Security
i
Virtualization System
n
n
g th
e
i
v r
i s
r ec
t u
u ri
a ty
l
• E
i n
z su
ar Chapter 3: Operating System Security
13
Virtualization Infrastructure Security
Access to VM
image and
snapshots
must be
carefully
controlled

Access
must be
limited to
just the
appropriate
guest
Systems
manage
access to
hardware
resources

14
Chapter 3: Operating System Security
Rootkits
• A rootkit is a malicious software package that
attains and takes advantage of root status or
effectively becomes part of the OS
• Rootkits often go to great length to avoid being
discovered or, if discovered and partially
removed, to reestablish themselves
• This can include intercepting or modifying basic OS
functions

15
Chapter 3: Operating System Security
Rootkit Evading Detection

16
Chapter 3: Operating System Security
Linux/Unix Security

Self-Learning

17
17
 Self-Learning
Linux/Unix Security

• Patch management
• Keeping security patches up to date is a widely recognized and
critical control for maintaining security
• Application and service configuration
• Most commonly implemented using separate text files for each
application and service
• Generally located either in the /etc directory or in the installation
tree for a specific application
• Individual user configurations that can override the system defaults
are located in hidden “dot” files in each user’s home directory
• Most important changes needed to improve system security are to
disable services and applications that are not required
18
Chapter 3: Operating System Security
 Self-Learning
Linux/Unix Security

• Users, groups, and permissions

• Access is specified as granting read, write, and execute
permissions to each of owner, group, and others for each resource
• Guides recommend changing the access permissions for critical
directories and files
• Local exploit
• Software vulnerability that can be exploited by an
attacker to gain elevated privileges
• Remote exploit
• Software vulnerability in a network server that could
be triggered by a remote attacker
19
Chapter 3: Operating System Security
 Self-Learning
Linux/Unix Security

Remote access Logging and

controls
• Several host log rotation
firewall programs • Should not
may be used
• Most systems assume
provide an that the
administrative
utility to select
default
which services setting is
will be permitted
to access the
necessarily
system appropriate

20
Chapter 3: Operating System Security
 Users
administratio
n and access
Windows Security controls
• Systems
Patch implement
management discretionary
access controls
• “Windows resources
Update” and
“Windows • Vista and later
Server Update systems include
Service” assist mandatory
with regular integrity controls
maintenance • Objects are
and should be labeled as being
used of low, medium,
• Third party high, or system
applications integrity level
also provide • System ensures
automatic the subject’s
update support integrity is equal
or higher than
the object’s
level
• Implements a
form of the Biba
Integrity model

21
Chapter 3: Operating System Security
Windows Security:
User Administration and Access Control
Windows systems also Combination of share and
define privileges NTFS permissions may be
used to provide additional
•System wide and
security and granularity when
granted to user accessing files on a shared
accounts resource

User Account Control (UAC) Low Privilege Service

•Provided in Vista and later systems Accounts
•Assists with ensuring users with
administrative rights only use them
•Used for long-lived service
when required, otherwise accesses processes such as file,
the system as a normal user print, and DNS services
22
 Windows Security
Application and service
configuration

• Much of the configuration information is

centralized in the Registry
• Forms a database of keys and values that may be queried and
interpreted by applications
• Registry keys can be directly modified using the
“Registry Editor”
• More useful for making bulk changes
23
Chapter 3: Operating System Security
 Windows Security
Other security controls

• Essential that anti-virus, anti-spyware, personal firewall, and other malware and attack detection and handling software packages
are installed and configured
• Current generation Windows systems include basic firewall and malware countermeasure capabilities
• Important to ensure the set of products in use are compatible

Windows systems also support a range of

cryptographic functions:
• Encrypting files and directories using the Encrypting File System (EFS)
• Full-disk encryption with AES using BitLocker

“Microsoft Baseline Security Analyzer”

• Free, easy to use tool that checks for compliance with Microsoft’s security
recommendations 24
Chapter 3: Operating System Security
Summary
• OSs have evolved from supporting single users and single
programs to many users and programs at once
• Resources that require OS protection: memory, I/O
devices, programs, and networks
• OSs use layered and modular designs for simplification
and to separate critical functions from noncritical ones
• Resource access control can be enforced in a number of
ways, including virtualization, segmentation, hardware
memory protection, and reference monitors
• Rootkits are malicious software packages that attain root
status or effectively become part of the OS

25
Chapter 3: Operating System Security