Choosing a decomposition method: scotch, hierarchical or simple, measured
Published
decomposePar splits the mesh into one piece per MPI process. How it cuts decides two things that
set the speed of a parallel run: how evenly the cells are spread (load balance), and how many faces
lie on the cuts (communication). Different methods trade these off differently, and forum answers
on which is “best” disagree because the answer depends on the mesh. This note shows how to measure
it on your own case in less than an hour of compute time, instead of trusting someone else’s
numbers.
The three methods
simplecuts the bounding box of the mesh intonx × ny × nzequal slabs along the coordinate axes. Each piece covers the same volume, not the same number of cells. On a uniform block mesh, that is fine. On a mesh refined around a body, the pieces near the body get many more cells than those in the far field.hierarchicalalso cuts along the axes, but places each cut so that the pieces hold equal numbers of cells, one direction after the other (theorderentry,xyzby default). Load balance is good. The cut surfaces are still flat planes, though, and can run through regions of fine mesh, which adds processor faces.scotchtreats the mesh as a graph and partitions it to balance cells while keeping the number of cut faces low. It needs no geometric input, onlynumberOfSubdomains. The cut surfaces follow the mesh rather than the axes.
numberOfSubdomains 64;
method scotch;
// Only read by the simple and hierarchical methods:
coeffs
{
n (4 4 4);
order xyz; // hierarchical only
}
Older cases use method-specific blocks such as simpleCoeffs and hierarchicalCoeffs. Both
spellings are still read by current versions.
What decomposePar tells you
At the end of its output, decomposePar prints statistics about the decomposition. These lines
are the ones to compare (the numbers are an example for a mesh of about 3.3 million cells on 64
processes):
Number of processor faces = 81236
Max number of cells = 52813 (1.6% above average 51985.4)
Max number of processor patches = 11 (37.5% above average 8)
Max number of faces between processors = 3471 (21.2% above average 2863.6)
- Max number of cells above average is the load imbalance. Every time step, all processes wait for the slowest one. A process with 20% more cells than average makes the whole run up to 20% slower than it could be.
- Number of processor faces is the total size of the cuts. Each processor face means data exchanged every time the solver updates boundary values, many times per time step.
- Max number of processor patches is how many neighbours the busiest process talks to. Each neighbour is a separate message, and latency adds up per message, not per byte.
You can collect these numbers without running the solver at all. Decomposing a mesh of a few million cells takes minutes on one core.
Step 1: compare the decompositions
Run this in a copy of the case, on a compute node (not the login node). It decomposes once per method and keeps each log:
#!/bin/bash
NP=64
for method in scotch hierarchical simple; do
foamDictionary -entry numberOfSubdomains -set "${NP}" system/decomposeParDict
foamDictionary -entry method -set "${method}" system/decomposeParDict
decomposePar -force > "log.decomposePar.${method}" 2>&1
echo "== ${method}"
grep -E 'Number of processor faces|Max number of (cells|processor patches|faces between)' \
"log.decomposePar.${method}"
done
For simple and hierarchical, set n so that it multiplies to NP and roughly follows the
shape of the domain: for a long channel, cut mostly along its length, for example n (16 2 2).
A badly chosen n makes these methods look worse than they need to.
Step 2: time a short run with each
The decomposition statistics predict the speed, but only a run measures it. Set up a short run that represents the real one:
- Start from a converged or developed state if you have one, so the solver work per step is typical. The first few iterations from initial conditions are not.
- Run a fixed number of steps, for example 100 iterations for a steady-state solver or 50 time steps
for a transient one, with
writeIntervallarger than that so no time is spent writing. - Use the same nodes, or at least the same node type, for every method. Run each twice.
Then compare the ExecutionTime at the end of each log, minus the ExecutionTime after the first
step, to leave out start-up:
for method in scotch hierarchical simple; do
# ... decompose with ${method} as above, then:
mpirun -np "${NP}" simpleFoam -parallel > "log.simpleFoam.${method}" 2>&1
first=$(grep -m1 '^ExecutionTime' "log.simpleFoam.${method}" | awk '{print $3}')
last=$(grep '^ExecutionTime' "log.simpleFoam.${method}" | tail -n 1 | awk '{print $3}')
echo "${method}: $(awk -v a="${first}" -v b="${last}" 'BEGIN { print b - a }') s"
done
What the numbers usually show
Your measurements are what count, but some patterns come up again and again:
- On meshes refined around a geometry (external aerodynamics, anything from
snappyHexMesh),simpleproduces a large imbalance, often tens of percent, because equal volumes hold very different cell counts.hierarchicalfixes the balance.scotchusually has the fewest processor faces and is fastest or close to it. - On structured block meshes of simple shape (channels, boxes, pipes made with
blockMesh),hierarchicalwith a well-chosennis as good asscotch, sometimes slightly better, because flat cuts on a regular mesh are already minimal. - As the process count grows, communication matters more and
scotch’s advantage in processor faces grows with it. A method that is fine on 32 processes can fall behind on 512. - Imbalance below a few percent rarely matters. Differences in processor faces of 10-20% typically show up as single-digit percentage differences in run time. Large differences only come from large imbalances.
scotch is therefore the sensible default for scripts and for users who don’t want to think about
it. It also adapts to any process count by itself, which removes a whole class of errors (see the
note on decomposition mismatches). Use hierarchical where
you have measured that it helps, or where you need cuts along a known direction, for example to keep
a periodic pair or a thin layer on one process.
Run the same benchmark at several process counts and you also get a speed-up curve for the case, which is the basis for sizing an OpenFOAM job.