2/20/2019 OpenFOAM v6 User Guide: 3.

4 Running applications parallel

   

CFD Direct
The Architects of OpenFOAM

Search 

Home OpenFOAM Cloud Training Support Contact

OpenFOAM v6 User Guide: 3.4 Running

applications in parallel

[Table of Contents] [Index] [ Version 6 | Version 5 | Version 4 ] OpenFOAM

[prev] [next] Training

25 Feb Houston,
3.4 Running applications in parallel USA
This section describes how to run OpenFOAM in parallel on distributed 12 Mar Virtual,
processors. The method of parallel computing used by OpenFOAM is known as Europe
domain decomposition, in which the geometry and associated elds are broken
into pieces and allocated to separate processors for solution. The process of 21 Mar Virtual,
parallel computation involves: decomposition of mesh and elds; running the Americas
application in parallel; and, post-processing the decomposed case as described in 13 May Cologne,
the following sections. The parallel running uses the public domain Germany
openMPI implementation of the standard message passing interface (MPI) by
13 May Virtual,
default, although other libraries can be used.
Cologne

10 Jun Denver, USA

3.4.1 Decomposition of mesh and initial eld data
10 Jun Virtual,
The mesh and elds are decomposed using the decomposePar utility. The
Denver
underlying aim is to break up the domain with minimal effort but in such a way to
guarantee an economic solution. The geometry and elds are broken up 09 Jul London, UK
according to a set of parameters speci ed in a dictionary named
09 Jul Virtual,
decomposeParDict that must be located in the system directory of the case of
London
interest. An example decomposeParDict dictionary is available from the

https://cfd.direct/openfoam/user-guide/v6-running-applications-parallel/ 1/9
2/20/2019 OpenFOAM v6 User Guide: 3.4 Running applications parallel

interFoam/damBreak tutorial if the user requires one; the dictionary entries 17 Sep Virtual,
within it are reproduced below: Americas

17 23 Sep London, UK
18 numberOfSubdomains 4;
23 Sep Virtual,
19
London
20 method simple;
21
22 simpleCoeffs
23 { Recent Posts
24 n (2 2 1);
25 delta 0.001; Learn Effective CFD
26 }
27 OpenFOAM v6
28 hierarchicalCoeffs Training 2019
29 {
OpenFOAM
30 n (1 1 1);
31 delta 0.001; Sustainability 2018
32 order xyz;
Productive CFD
33 }
with OpenFOAM
34
35 manualCoeffs OpenFOAM v6
36 {
Training
37 dataFile "";
38 } CFDDFC on AWS
39 C5/M5
40 distributed no;
41 CFD Direct Year 3:
42 roots ( ); 2017-2018
43
44
Cloud CFD Course |
45 // *********************************************************** OpenFOAM

Sustainable
The user has a choice of four methods of decomposition, speci ed by the method OpenFOAM
keyword as described below. Development

OpenFOAM v5
simple
Training
Simple geometric decomposition in which the domain is split into pieces by
direction, e.g. 2 pieces in the direction, 1 in etc.
Follow

hierarchical
Follow
Hierarchical geometric decomposition which is the same as simple except Follow @cfddirect 2,8
the user speci es the order in which the directional split is done, e.g. rst
YouTube 1K
in the -direction, then the -direction etc.

scotch

https://cfd.direct/openfoam/user-guide/v6-running-applications-parallel/ 2/9
2/20/2019 OpenFOAM v6 User Guide: 3.4 Running applications parallel

Scotch decomposition which requires no geometric input from the user and Monthly
attempts to minimise the number of processor boundaries. The user can
specify a weighting for the decomposition between processors, through an Newsletter
optional processorWeights keyword which can be useful on machines with
differing performance between processors. There is also an optional First Name
keyword entry strategy that controls the decomposition strategy through a
complex string supplied to Scotch. For more information, see the source Last Name
code le:
$FOAM_SRC/parallel/decompose/scotchDecomp/scotchDecomp.C
Email Address

manual
Type of C
Manual decomposition, where the user directly speci es the allocation of
each cell to a particular processor.
Country

For each method there are a set of coef cients speci ed in a sub-dictionary of * I consent to my
decompositionDict, named <method>Coeffs as shown in the dictionary listing. data being
The full set of keyword entries in the decomposeParDict dictionary are processed in
explained in Table 3.3. accordance with the
Privacy Policy
Compulsory entries
numberOfSubdomains Total number of subdomains Subscribe!

method Method of decomposition simple/

hierarchical/
scotch/
manual/ Latest News
simpleCoeffs entries
on Twitter
n Number of subdomains in , , ( )
Don't miss out! Last
seats remaining for
delta Cell skew factor Typically,
our #OpenFOAM
Training #Houston,
hierarchicalCoeffs entries
25-28 Feb 2019.
n Number of subdomains in , , ( )
Book your place
now:
delta Cell skew factor Typically,
https://t.co/miyMlW
order Order of decomposition xyz/xzy/yxz… ImRJ5 hours ago

Added radiation to
scotchCoeffs entries the reverseBurner
processorWeights List of weighting factors for (<wt1>…<wtN>) conjugate heat
(optional) allocation of cells to processors; transfer tutorial in
<wt1> is the weighting factor for
#OpenFOAM-dev
processor 1, etc. ; weights are
https://t.co/UGKJRi
normalised so can take any
range of values. fr0G8 hours ago
https://cfd.direct/openfoam/user-guide/v6-running-applications-parallel/ 3/9
2/20/2019 OpenFOAM v6 User Guide: 3.4 Running applications parallel

strategy Decomposition strategy: We design our

optional and complex
OpenFOAM
Training so that
manualCoeffs entries
users can be
dataFile Name of le containing data of "<fileName>" productive with
allocation of cells to processors
OpenFOAM:
https://t.co/KixhnYo
Distributed data entries (optional) — see section 3.4.4
94E12 hours ago
distributed Is the data distributed across yes/no
several disks?

roots Root paths to case directories; (<rt1>…<rtN>)

<rt1> is the root path for node
1, etc.

Table 3.3: Keywords in decompositionDict dictionary.

The decomposePar utility is executed in the normal manner by typing

decomposePar

3.4.2 File input/output in parallel

Using standard le input/output completion, a set of subdirectories will have
been created, one for each processor, in the case directory. The directories are
named processor where represents a processor number and
contains a time directory, containing the decomposed eld descriptions, and a
constant/polyMesh directory containing the decomposed mesh description.
While this le structure is well-organised, for large parallel cases, it generates a
large number of les. In very large simulations, users can experience problems
including hitting limits on the number of open les imposed by the operating
system.

As an alternative, the collated le format was introduced in OpenFOAM in which

the data for each decomposed eld (and mesh) is collated into a single le that is
written (and read) on the master processor. The les are stored in a single
directory named processors.

The le writing can be threaded allowing the simulation to continue running

while the data is being written to le — see below for details. NFS (Network File
System) is not needed when using the collated format and, additionally, there is a
masterUncollated option to write data with the original uncollated format
without NFS.

https://cfd.direct/openfoam/user-guide/v6-running-applications-parallel/ 4/9
2/20/2019 OpenFOAM v6 User Guide: 3.4 Running applications parallel

The controls for the le handling are in the OptimisationSwitches of the

global etc/controlDict le:

OptimisationSwitches
{
...

//- Parallel IO file handler

// uncollated (default), collated or masterUncollated
fileHandler uncollated;

//- collated: thread buffer size for queued file writes.

// If set to 0 or not sufficient for the file size threading
// Default: 2e9
maxThreadFileBufferSize 2e9;

//- masterUncollated: non-blocking buffer size.

// If the file exceeds this buffer size scheduled transfer i
// Default: 2e9
maxMasterFileBufferSize 2e9;
}

3.4.2.1 Selecting the le handler

The fileHandler can be set for a speci c simulation by:

over-riding the global OptimisationSwitches {fileHandler …;} in the

case controlDict le;
using the -fileHandler command line argument to the solver;
setting the $FOAM_FILEHANDLER environment variable.

3.4.2.2 Updating exisiting les

A foamFormatConvert utility allows users to convert les between the collated
and uncollated formats, e.g.

mpirun -np 2 foamFormatConvert -parallel -fileHandler uncollat

An example case demonstrating the le handling methods is provided in:

$FOAM_TUTORIALS/IO/fileHandling

3.4.2.3 Threading support

Collated le handling runs faster with threading, especially on large cases. But it 
requires threading support to be enabled in the underlying MPI. Without it, the
https://cfd.direct/openfoam/user-guide/v6-running-applications-parallel/ 5/9
2/20/2019 OpenFOAM v6 User Guide: 3.4 Running applications parallel

simulation will “hang” or crash. For openMPI, threading support is not set by
default prior to version 2, but is generally switched on from version 2 onwards.
The user can check whether openMPI is compiled with threading support by the
following command:

ompi_info -c | grep -oE "MPI_THREAD_MULTIPLE[^,]*"

When using the collated le handling, memory is allocated for the data in the
thread. maxThreadFileBufferSize sets the maximum size of memory that is
allocated in bytes. If the data exceeds this size, the write does not use threading.
Note: if threading is not enabled in the MPI, it must be disabled for collated le
Note
handling by setting in the global etc/controlDict le:

maxThreadFileBufferSize 0;

When using the masterUncollated le handling, non-blocking MPI

communication requires a suf ciently large memory buffer on the master node.
maxMasterFileBufferSize sets the maximum size of the buffer. If the data
exceeds this size, the system uses scheduled communication.

3.4.3 Running a decomposed case

A decomposed OpenFOAM case is run in parallel using the openMPI
implementation of MPI. openMPI can be run on a local multiprocessor machine
very simply but when running on machines across a network, a le must be
created that contains the host names of the machines. The le can be given any
name and located at any path. In the following description we shall refer to such
a le by the generic name, including full path, <machines>.

The <machines> le contains the names of the machines listed one machine per
line. The names must correspond to a fully resolved hostname in the
/etc/hosts le of the machine on which the openMPI is run. The list must
contain the name of the machine running the openMPI. Where a machine node
contains more than one processor, the node name may be followed by the entry cpu=
where is the number of processors openMPI should run on that node.

For example, let us imagine a user wishes to run openMPI from machine aaa on
the following machines: aaa; bbb, which has 2 processors; and ccc. The
<machines> would contain:

aaa
bbb cpu=2
ccc 
https://cfd.direct/openfoam/user-guide/v6-running-applications-parallel/ 6/9
2/20/2019 OpenFOAM v6 User Guide: 3.4 Running applications parallel

An application is run in parallel using mpirun.

mpirun --hostfile <machines> -np <nProcs>

<foamExec> <otherArgs> -parallel > log &

where: <nProcs> is the number of processors; <foamExec> is the executable,

e.g. icoFoam; and, the output is redirected to a le named log. For example, if
icoFoam is run on 4 nodes, speci ed in a le named machines, on the cavity
tutorial in the $FOAM_RUN/tutorials/incompressible/icoFoam directory,
then the following command should be executed:

mpirun --hostfile machines -np 4 icoFoam -parallel > log &

3.4.4 Distributing data across several disks

Data les may need to be distributed if, for example, if only local disks are used in
order to improve performance. In this case, the user may nd that the root path
to the case directory may differ between machines. The paths must then be
speci ed in the decomposeParDict dictionary using distributed and roots
keywords. The distributed entry should read

distributed yes;

and the roots entry is a list of root paths, <root0>, <root1>, …, for each node

roots
<nRoots>
(
"<root0>"
"<root1>"
…
);

where <nRoots> is the number of roots.

Each of the processor directories should be placed in the case directory at
each of the root paths speci ed in the decomposeParDict dictionary. The
system directory and files within the constant directory must also be
present in each case directory. Note: the les in the constant directory are
needed, but the polyMesh directory is not.

3.4.5 Post-processing parallel processed cases


https://cfd.direct/openfoam/user-guide/v6-running-applications-parallel/ 7/9
2/20/2019 OpenFOAM v6 User Guide: 3.4 Running applications parallel

When post-processing cases that have been run in parallel the user has two
options:

reconstruction of the mesh and eld data to recreate the complete domain
and elds, which can be post-processed as normal;
post-processing each segment of decomposed domain individually.

3.4.5.1 Reconstructing mesh and data

After a case has been run in parallel, it can be reconstructed for post-processing.
The case is reconstructed by merging the sets of time directories from each processor
directory into a single set of time directories. The reconstructPar utility
performs such a reconstruction by executing the command:

reconstructPar

When the data is distributed across several disks, it must be rst copied to the
local case directory for reconstruction.

3.4.5.2 Post-processing decomposed cases

The user may post-process decomposed cases using the paraFoam post-
processor, described in section 6.1. The whole simulation can be post-processed
by reconstructing the case or alternatively it is possible to post-process a
segment of the decomposed domain individually by simply treating the
individual processor directory as a case in its own right.

[prev] [next]

© 2011-2019 OpenFOAM Foundation

 Chris Greenshields  10th July 2018  User Guide

← OpenFOAM v6 User Guide: 3.3 Running applications

OpenFOAM v6 User Guide: 3.5 Standard solvers →


Copyright © 2015-2019 CFD Direct Ltd Contact Us Twitter LinkedIn YouTube Website Terms of Use Privacy
https://cfd.direct/openfoam/user-guide/v6-running-applications-parallel/ 8/9
2/20/2019 OpenFOAM v6 User Guide: 3.4 Running applications parallel


https://cfd.direct/openfoam/user-guide/v6-running-applications-parallel/ 9/9