Choosing a decomposition method: scotch, hierarchical or simple, measured

Published

decomposePar splits the mesh into one piece per MPI process. How it cuts decides two things that set the speed of a parallel run: how evenly the cells are spread (load balance), and how many faces lie on the cuts (communication). Different methods trade these off differently, and forum answers on which is “best” disagree because the answer depends on the mesh. This note shows how to measure it on your own case in less than an hour of compute time, instead of trusting someone else’s numbers.

The three methods

numberOfSubdomains 64;

method          scotch;

// Only read by the simple and hierarchical methods:
coeffs
{
    n           (4 4 4);
    order       xyz;     // hierarchical only
}

Older cases use method-specific blocks such as simpleCoeffs and hierarchicalCoeffs. Both spellings are still read by current versions.

What decomposePar tells you

At the end of its output, decomposePar prints statistics about the decomposition. These lines are the ones to compare (the numbers are an example for a mesh of about 3.3 million cells on 64 processes):

Number of processor faces = 81236
Max number of cells = 52813 (1.6% above average 51985.4)
Max number of processor patches = 11 (37.5% above average 8)
Max number of faces between processors = 3471 (21.2% above average 2863.6)

You can collect these numbers without running the solver at all. Decomposing a mesh of a few million cells takes minutes on one core.

Step 1: compare the decompositions

Run this in a copy of the case, on a compute node (not the login node). It decomposes once per method and keeps each log:

#!/bin/bash
NP=64
for method in scotch hierarchical simple; do
    foamDictionary -entry numberOfSubdomains -set "${NP}" system/decomposeParDict
    foamDictionary -entry method -set "${method}" system/decomposeParDict
    decomposePar -force > "log.decomposePar.${method}" 2>&1
    echo "== ${method}"
    grep -E 'Number of processor faces|Max number of (cells|processor patches|faces between)' \
        "log.decomposePar.${method}"
done

For simple and hierarchical, set n so that it multiplies to NP and roughly follows the shape of the domain: for a long channel, cut mostly along its length, for example n (16 2 2). A badly chosen n makes these methods look worse than they need to.

Step 2: time a short run with each

The decomposition statistics predict the speed, but only a run measures it. Set up a short run that represents the real one:

Then compare the ExecutionTime at the end of each log, minus the ExecutionTime after the first step, to leave out start-up:

for method in scotch hierarchical simple; do
    # ... decompose with ${method} as above, then:
    mpirun -np "${NP}" simpleFoam -parallel > "log.simpleFoam.${method}" 2>&1
    first=$(grep -m1 '^ExecutionTime' "log.simpleFoam.${method}" | awk '{print $3}')
    last=$(grep '^ExecutionTime' "log.simpleFoam.${method}" | tail -n 1 | awk '{print $3}')
    echo "${method}: $(awk -v a="${first}" -v b="${last}" 'BEGIN { print b - a }') s"
done

What the numbers usually show

Your measurements are what count, but some patterns come up again and again:

scotch is therefore the sensible default for scripts and for users who don’t want to think about it. It also adapts to any process count by itself, which removes a whole class of errors (see the note on decomposition mismatches). Use hierarchical where you have measured that it helps, or where you need cuts along a known direction, for example to keep a periodic pair or a thin layer on one process.

Run the same benchmark at several process counts and you also get a speed-up curve for the case, which is the basis for sizing an OpenFOAM job.