PARALLEL AND
DISTRIBUTED
COMPUTING
WHAT IS COMPUTING?
Computing is the process of
using computer technology to
complete a given goal-oriented task.
It includes the study and
experimentation
of algorithmic processes and
development of
both hardware and software.
Major computing disciplines
include computer
engineering, computer
science, cybersecurity, data
SERIAL COMPUTATION:
Traditionally software has been written
for serial computations:
To be run on a single computer having a
single Central Processing Unit (CPU)
A problem is broken into a discrete set of
instructions
Instructions are executed one after another
Only one instruction can be executed at any
moment in time
SERIAL COMPUTATION:
PARALLEL COMPUTING:
In the simplest sense, parallel
computing is the simultaneous use of
multiple compute resources to solve a
computational problem:
To be run using multiple CPUs
A problem is broken into discrete parts that
can be solved concurrently
Each part is further broken down to a series
of instructions
Instructions from each part execute
simultaneously on different CPUs
PARALLEL COMPUTERS:
Virtually all stand-alone computers
today are parallel from a hardware
perspective:
Multiple functional units (floating point,
integer, GPU, etc.)
Multiple execution units / cores
Multiple hardware threads
PARALLEL COMPUTERS:
Networks connect multiple stand-alone
computers (nodes) to create larger
parallel computer clusters
Each compute node is a multi-processor
parallel computer in itself
Special purpose nodes, also multi-
processor, are used for other purposes
DISTRIBUTED COMPUTING
A distributed computer system consists of
multiple software components that are on
multiple computers, but run as a single system.
The computers that are in a distributed system
can be physically close together and connected
by a local network, or they can be
geographically distant and connected by a
wide area network.
A distributed system can consist of any number
of possible configurations, such as mainframes,
personal computers, workstations,
minicomputers, and so on.
The goal of distributed computing is to make
such a network work as a single computer.
parallel computing focuses on using
multiple processors or cores within a
single computer to solve a problem,
while distributed computing focuses on
using multiple computers connected by
a network to solve a problem.
The choice between these two
approaches will depend on the specific
requirements of the problem being
solved, including the amount of data
being processed, the level of
computational power required, and the
nature of the computation itself.
Both parallel and distributed computing
have both benefits and drawbacks, so
selecting one to choose will depend on
the system’s unique needs and
limitations. Parallel computing can
achieve high performance with low
latency and high bandwidth, while
distributed computing can achieve fault
tolerance and scalability with
geographic dispersion.
DIFFERENCE BETWEEN PARALLEL
COMPUTING AND DISTRIBUTED
COMPUTING:
S.N
Parallel Computing Distributed Computing
O
Many operations are performed System components are located at
1.
simultaneously different locations
2. Single computer is required Uses multiple computers
Multiple processors perform multiple Multiple computers perform
3.
operations multiple operations
It may have shared or distributed
4. It have only distributed memory
memory
Processors communicate with each other Computer communicate with each
5.
through bus other through message passing.
Improves system scalability, fault
6. Improves the system performance tolerance and resource sharing
capabilities
WHY USE PARALLEL COMPUTING?
Parallel computing is complex on any
aspect!
The primary reasons for using parallel
computing:
Save time - wall clock time
Solve larger problems
Provide concurrency (do multiple things at
the same time)
WHY USE PARALLEL COMPUTING?
Other reasons might include:
Taking advantage of non-local resources -
using available compute resources on a
wide area network, or even the Internet
when local compute resources are scarce.
Cost savings - using multiple "cheap"
computing resources instead of paying for
time on a supercomputer.
Overcoming memory constraints - single
computers have very finite memory
resources. For large problems, using the
memories of multiple computers may
overcome this obstacle.
LIMITATIONS OF SERIAL COMPUTING
Limits to serial computing
both physical and practical reasons pose significant constraints
to simply building ever faster serial computers.
Transmission speeds - the speed of a serial computer is
directly dependent upon how fast data can move through
hardware. Absolute limits are the speed of light (30
cm/nanosecond) and the transmission limit of copper wire
(9 cm/nanosecond). Increasing speeds necessitate
increasing proximity of processing elements.
Limits to miniaturization - processor technology is allowing
an increasing number of transistors to be placed on a chip.
However, even with molecular or atomic-level components,
a limit will be reached on how small components can be.
Economic limitations - it is increasingly expensive to make
a single processor faster. Using a larger number of
moderately fast commodity processors to achieve the
same (or better) performance is less expensive.
