Parallel Computing with Matlab
UVACSE Short Course
E Hall1
1 University of Virginia Advanced Computing Services and Engagement
uvacse@virginia.edu
October 6, 2014
(UVACSE) October 6, 2014 1/1
Outline
(UVACSE) October 6, 2014 2/1
NX Client for Remote Linux Desktop
Starting and Configuring NX Client
Once logged into fir.itc.virginia.edu through NX
Open a terminal from Applications/Accessories/Terminal menu
Select and right-click on Terminal to add to launcher
Create a Matlab directory with mkdir command
Start web browser from icon at top of desktop
(UVACSE) October 6, 2014 3/1
NX Client for Remote Linux Desktop
Download Short Course Examples
Download the short-course materials from
http://www.uvacse.virginia.edu/software/Matlab-at-uva/
Follow the links,
→High Performance Computing
→Parallel Computing Toolbox
→Parallel Computing Short Course
and download 3 files to Matlab directory you create with mkdir
command
ClassExamples_Fa14.zip
matlab_parallel_Fa14.pdf
(UVACSE) October 6, 2014 4/1
Matlab Parallel Computing Toolbox
Solving Big Technical Problems
Computationally intensive, long-running codes
Run tasks in parallel on many processors
Task parallel
Large Data sets
Load data across multiple machines that work in parallel
Data parallel
(UVACSE) October 6, 2014 5/1
Matlab Parallel Computing Toolbox
Parallel Computing Toolbox Features
Support for data-parallel and task-parallel application development
Ability to annotate code segments
parfor (parallel for-loops) for task-parallel algorithms
spmd (single program multiple data) for data-parallel algorithms
These high-level programming constructs convert serial Matlab
code to run in parallel on several workers
The parallel programming constructs function even in the absence of
workers so that a single version of code can run for both serial and
parallel execution
(UVACSE) October 6, 2014 6/1
Matlab Parallel Computing Toolbox
Programming Parallel Applications in Matlab
Simplifies parallel code development by abstracting away complexity of
managing coordination and distribution of computations and data
between a Matlab client and workers.
Run 12 workers locally on a multicore desktop
Integration with Matlab Distributed Computing Server for
cluster-based applications that can use up to 256 workers
(UVACSE) October 6, 2014 7/1
Matlab Parallel Computing Toolbox
Matlab Parallel Workflow
The toolbox enables application prototyping on the desktop with up to
eight local workers (left), and with Matlab Distributed Computing
Server (right), applications can be scaled to multiple computers on a
cluster.
(UVACSE) October 6, 2014 8/1
Matlab Parallel Computing Toolbox
UseParallel for Optimization Algorithms
The Optimization Toolbox solvers fmincon, fgoalattain, and
fminimax can automatically distribute the numerical estimation of
gradients of objective functions and nonlinear constraint functions to
multiple processors.
Parallel computing is enabled with matlabpool, a Parallel
Computing Toolbox function
The option UseParallel is set to ’Always’. The default value
of this option is ’Never’.
(UVACSE) October 6, 2014 9/1
Matlab Parallel Computing Toolbox
UseParallel for Optimization Algorithms
Optimization Toolbox
fmincon
fminimax
fgoalattain
Genetic Algorithm and Direct Search Toolbox
ga
gamultiobj
patternsearch
(UVACSE) October 6, 2014 10 / 1
Matlab Parallel Computing Toolbox
UseParallel for Optimization Algorithms
Caveats:
The built-in parallel support in Optimization Toolbox is beneficial
for problems that have objective/constraint functions with
execution times greater than network overhead associated
with distributing computations across multiple workers
However, parallelizing the objective/constraint function itself
can be a better approach if it is the most expensive step in the
optimization problem and can be accelerated by parallelizing the
objective function
Documentation:
Improving Optimization Performance with Parallel Computing
Minimizing an Expensive Optimization Problem Using Parallel
Computing Toolbox
(UVACSE) October 6, 2014 11 / 1
Matlab Parallel Computing Toolbox
Example Code: Parallel Optimization with fmincon
(UVACSE) October 6, 2014 12 / 1
Matlab Parallel Computing Toolbox
Task-parallelism using parfor
Parallel for-Loops - parfor
Code Annotation
parfor i = 1 : n
% do something with i
end
Mix task-parallel and serial code in the same function
Run loops on a pool of Matlab resources
Iterations must be order-independent
(UVACSE) October 6, 2014 13 / 1
Matlab Parallel Computing Toolbox
Task-parallelism using parfor
(UVACSE) October 6, 2014 14 / 1
Matlab Parallel Computing Toolbox
Task-parallelism using parfor
(UVACSE) October 6, 2014 15 / 1
Matlab Parallel Computing Toolbox
Task-parallelism using parfor
Parallel for-loops (parfor) automatically distribute a set of
independent tasks over a set of workers.
Work distribution across worker is dynamic for load balancing.
The matlabpool command sets up the interactive execution
environment for parallel constructs such as parfor or spmd
The running code automatically detects the presence of workers and
reverts to serial behavior if none are present.
Using the batch command allows parallel code to run in batch mode
across the compute nodes of a cluster.
Applications: Monte Carlo simulations, Parameter sweeps
(UVACSE) October 6, 2014 16 / 1
Matlab Parallel Computing Toolbox
Setting Up the Matlab Workers with matlabpool
The matlabpool command allocates a set of dedicated
computational resources by reserving a number of Matlab workers for
executing parallel Matlab code.
Parallel Computing Toolbox provides the ability to use up to 12 local
workers on a multicore or multiprocessor computer using a single
toolbox license.
Within this environment, parfor and spmd constructs can set up data
and Matlab code exchange between a Matlab client.
(UVACSE) October 6, 2014 17 / 1
Matlab Parallel Computing Toolbox
spmd for Data-Parallel Processing
For Matlab algorithms that require large data set processing, the
Parallel Computing Toolbox provides,
distributed arrays, parallel functions
the spmd keyword to annotate sections of your code for parallel
execution on several workers.
These parallel constructs handle the inter-worker communication and
coordinate parallel computations behind the scenes.
(UVACSE) October 6, 2014 18 / 1
Matlab Parallel Computing Toolbox
spmd for Data-Parallel Processing
Using distributed arrays, you can allocate matrices of any data type
across all workers participating in the parallel computation.
Parallel functions let you perform mathematical operations such as
indexing, matrix multiplication, decomposition, and transforms directly
on distributed arrays.
The toolbox also provides more than150 parallel functions for
distributed arrays, including linear algebra routines based on
ScaLAPACK.
(UVACSE) October 6, 2014 19 / 1
Matlab Parallel Computing Toolbox
spmd for Data-Parallel Processing
Single Program Multiple Data - spmd
Code Annotation
spmd
% data parallel operation
end
Single program
Runs simultaneously across all workers
Enables easy writing and debugging
Multiple Data
Data spread across workers
Runs serially if no workers are available
(UVACSE) October 6, 2014 20 / 1
Matlab Parallel Computing Toolbox
spmd for Data-Parallel Processing
Distributed Arrays
Distributed arrays are special arrays that store segments of data
on Matlab workers that are participating in a parallel computation.
can handle larger data sets than you can on a single Matlab
session.
You can construct distributed arrays in several ways:
Using constructor functions such as rand, ones, and zeros
Concatenating arrays with same name but different data on
different labs
Distributing a large matrix
(UVACSE) October 6, 2014 21 / 1
Matlab Parallel Computing Toolbox
spmd for Data-Parallel Processing: Example Code
Solving a Large Linear System
(UVACSE) October 6, 2014 22 / 1
Matlab Parallel Computing Toolbox
spmd for Data-Parallel Processing: Example
Numerical Estimation of Pi Using Message Passing
Use the fact that
Z 1
4
dx = 4(atan(1) − atan(0)) = π
0 1 + x2
to approximate pi by approximating the integral on the left
use the spmd keyword to mark the parallel blocks of code and the
Matlab worker pool performs the calculations in parallel
(UVACSE) October 6, 2014 23 / 1
Matlab Parallel Computing Toolbox
spmd for Data-Parallel Processing: Example
Divide the work between four labs by having each lab calculate the
integral of the function over a subinterval of [0, 1]
Define the variables a and b on each lab using the labindex so that
the intervals [a, b] correspond to the subintervals above.
(UVACSE) October 6, 2014 24 / 1
Matlab Parallel Computing Toolbox
spmd for Data-Parallel Processing: Example Code
Parallel Esimation of Pi:
(UVACSE) October 6, 2014 25 / 1
Matlab Parallel Computing Toolbox
Matlab MPI for Message Passing
Use when a high degree of control over parallel algorithm is required
High-level abstractions of MPI message-passing routines based
on the MPI standard (MPICH2)
labSendReceive, labBroadcast, and others
Send, receive, and broadcast any data type in Matlab including
structures and cell arrays, without any special setup.
Use any MPI implementation that is binary-compatible with
MPICH-2
(UVACSE) October 6, 2014 26 / 1
Matlab Parallel Computing Toolbox
Message Passing for Data Parallel Algorithms
Message passing functions can be used with spmd statements
(UVACSE) October 6, 2014 27 / 1
Matlab Parallel Computing Toolbox
pmode for Interactive Parallel Computing
Parallel Command Window
The Parallel Command Window provides an extension to the
Matlab command window for executing data-parallel Matlab code
directly on workers participating in the interactive parallel session
Commands issued at the pmode prompt are executed immediately
on all the labs and results are returned immediately to the client
session.
You can use distributed arrays, associated parallel functions, as
well as message-passing functions in this mode.
This tool facilitates the debugging process, as it allows you to
watch the results and the interaction between labs at each step.
(UVACSE) October 6, 2014 28 / 1
Matlab Parallel Computing Toolbox
pmode for Interactive Parallel Computing
Interactive Prototyping / Development Using Parallel Command
Window
Parallel Command Window similar to regular command window
Execute commands and observe behavior on each worker of the
cluster
Can run up to 8 workers on Matlab desktop
Workers are processes, not tied to cores or processors as such
(UVACSE) October 6, 2014 29 / 1
Matlab Parallel Computing Toolbox
pmode for Interactive Parallel Computing
Example Code
>>pmode start
P>>a=1
P>>labindex
P>>numlabs
P>> A=rand(2000, 2000, codistributor());
P>>whos
P>>size(localPart(A))
(UVACSE) October 6, 2014 30 / 1
Matlab Parallel Computing Toolbox
pmode for Interactive Parallel Computing
(UVACSE) October 6, 2014 31 / 1
Parallel Computing on the Linux Cluster
Scaling Up from the Desktop
Parallel Computing Toolbox provides the ability to use up to 12 local
workers on a multicore or multiprocessor computer using a single
toolbox license.
When used together with MATLAB Distributed Computing Server, you
can scale up your application to use any number of workers running on
any number of computers.
ITS Linux cluster has MDCS licenses for 256 workers.
Alternatively, you can run up to 12 workers on a single multi-core
compute node of the cluster.
(UVACSE) October 6, 2014 32 / 1
Parallel Computing on the Linux Cluster
Using Parallel Configurations with PBS Pro
Parallel Configurations - Where and How the Code is Executed
Maintain named configurations
Predefine cluster information and environment-specific
parameters
No code changes required
Set once, use many times
Useful for both interactive and batch workflows
Toolbox provides GUI to manage configurations
(UVACSE) October 6, 2014 33 / 1
Parallel Computing on the Linux Cluster
Scaling Up from the Desktop
Example: Running a gene regulation model on a cluster using
MATLAB Distributed Computing Server.
(UVACSE) October 6, 2014 34 / 1
Parallel Computing on the Linux Cluster
Parallel Matlab on ITC Linux Clusters
ITC Linux Clusters
Distriubted Computing Server Licenses for 256 workers. Users
encouraged not to use more than 16 at one time.
Matlab Configurations interface to PBS Pro for submitting jobs to
the cluster
Distributed Computing Server Licenses available on any Linux
cluster that mounts /common for ITC servers
(UVACSE) October 6, 2014 35 / 1
Parallel Computing on the Linux Cluster
Interactive Parallel Matlab Jobs on Cluster
The matlabpool command can be used with the cluster configuration
file to launch an interactive job to the cluster from within the Desktop
interface on the cluster front-end.
(UVACSE) October 6, 2014 36 / 1
Parallel Computing on the Linux Cluster
Interactive Parallel Matlab Jobs on Cluster
After execution of the matlabpool command with the cluster
configuration file, the PBS command qstat command shows the
compute nodes serving as workers have been allocated.
After the matlabpool close command, the job is exiting.
(UVACSE) October 6, 2014 37 / 1
Additional Computational Resources
Additional Computational Resources
The Cross Campus Grid Project
http://www.uvacse.virginia.edu/cross-campus-grid-xcg/
XSEDE: Extreme Science and Engineering Discovery Environment
https://www.xsede.org/
MATLAB GPU Computing with NVIDIA CUDA-Enabled GPUs
A Tesla GPU card is available for testing Matlab GPU code in the CS
department.
(UVACSE) October 6, 2014 38 / 1
References
References
1 Mathworks Parallel Computing Toolbox Documentation
http://www.mathworks.com/products/parallel-computing/
2 Mathworks Parallel Computing Toolbox Demos and Webinars
http://www.mathworks.com/products/parallel-
computing/demos.html
3 Parallel Matlab for Multicore and Multinode Computers, by Jeremy
Kepner, SIAM Press.
Need further help? Email uvacse@virginia.edu.
(UVACSE) October 6, 2014 39 / 1
