Why your OpenFOAM run fails after decomposePar: matching numberOfSubdomains to your allocation
Published
A parallel OpenFOAM run involves three numbers that have to agree:
numberOfSubdomainsinsystem/decomposeParDict, whichdecomposeParreads when it splits the case.- The number of
processor*directories thatdecomposeParactually wrote (or, in the collated format, the number in the name of theprocessorsNdirectory). - The number of MPI processes the solver is started with:
mpirun -np N, or the task count of the Slurm allocation when you start withsrun.
The first two drift apart when decomposeParDict is edited after decomposing. The second and
third drift apart when the job script asks for a different number of cores than last time. Most
of the “it ran yesterday” failures in parallel OpenFOAM jobs come from one of these two.
The messages and what they mean
The exact wording varies a little between versions and between the openfoam.org and openfoam.com lines, but the messages below are the ones to look for in the solver log or in the Slurm output.
“number of processor directories = 4 is not equal to the number of processors = 8”
The solver was started on 8 MPI processes, but the case was decomposed into 4. OpenFOAM checks this
right at start-up, before reading any fields. Either the job script asked for more cores than last
time, or decomposePar was not rerun after numberOfSubdomains changed.
Fix: decompose again for the new count, or start the solver with the count the case was decomposed for. Do not “fix” it by creating empty processor directories. The solver then fails later, with a less helpful message about missing files.
If the message says number of processor directories = 0, nothing was decomposed at all. Usually
decomposePar failed earlier in the script and the script carried on anyway. Check
log.decomposePar, and add set -e to the job script, as in the
Slurm job script for OpenFOAM.
“Wrong number of processor divisions in geomDecomp”
decomposePar itself refuses to run. This comes from the simple and hierarchical methods. Their
coefficients n (nx ny nz) must multiply to numberOfSubdomains. The message lists both: the number
of domains, and the decomposition you asked for. With numberOfSubdomains 8; and n (2 2 1);,
decomposePar expects 4 domains and stops.
Fix: change n so that nx × ny × nz matches, or switch to scotch, which only needs
numberOfSubdomains. Which method suits which case is covered in the
note on decomposition methods.
The banner appears once per process, and nProcs : 1
There is no error message, but the solver log contains the OpenFOAM banner several times, and
the header shows nProcs : 1 instead of the expected count. Every MPI process started its own
serial run of the undecomposed case, all writing into the same time directories.
There are two common causes:
-parallelis missing from the solver command line. Without it, the solver does not use MPI at all and ignores the processor directories.srunstarted the processes, but the MPI library has no Slurm support. Each process then sees itself as rank 0 of a one-process job. Usempiruninstead, or thesrun --mpi=option your cluster documents.
Kill such a job straight away. The results are garbage, because several processes overwrite each other’s output.
“Cannot find file “points” in directory “polyMesh”” under processor0
The processor directories exist but are incomplete. Usually an earlier decomposePar was killed
(by the walltime limit or by running out of memory on the login node) halfway through. Run
decomposePar -force again on a compute node.
The collated format: processors8 vs processor0
With the collated file handler, decomposePar writes one directory called processors8 (for 8
subdomains) instead of processor0 to processor7. Some newer versions add a rank range to the
name. A solver started without the collated file handler looks for processor0 and reports zero
processor directories, even though the case is decomposed. Start the solver with the same file
handler that decomposePar used. The page on collated vs uncollated parallel I/O
covers the switch.
A pre-flight check for the job script
The safest fix is to never let the numbers drift, by writing numberOfSubdomains from the
allocation in the job script itself. Where that is not possible (for example when a restart must
reuse an existing decomposition), check before starting the solver. This Bash function compares
all three numbers and stops the job with a readable message if they differ:
check_decomposition() {
local np=${SLURM_NTASKS:?not running under Slurm}
local wanted ndirs
wanted=$(foamDictionary -entry numberOfSubdomains -value system/decomposeParDict)
if compgen -G "processors*" > /dev/null; then
# Collated: the count is part of the directory name, e.g. processors8 or processors8_0-3
ndirs=$(ls -d processors* | head -n 1 | sed -E 's/^processors([0-9]+).*/\1/')
else
ndirs=$(find . -maxdepth 1 -type d -name 'processor[0-9]*' | wc -l)
fi
echo "allocation: ${np} tasks, decomposeParDict: ${wanted}, decomposed into: ${ndirs}"
if [[ "${np}" != "${wanted}" || "${np}" != "${ndirs}" ]]; then
echo "Decomposition does not match the allocation, not starting the solver." >&2
return 1
fi
}
check_decomposition
mpirun -np "${SLURM_NTASKS}" simpleFoam -parallel > log.simpleFoam 2>&1
With set -e in the script, a failed check ends the job within seconds instead of after the
solver has queued for hours and then failed. The job’s exit code is non-zero, so a dependent job
does not start.
Restarting with a different core count
Sometimes you want to continue a run on more (or fewer) cores than it was started with, because the queue for the larger partition is shorter, for example. There are two ways to do it:
- Reconstruct, then decompose again.
reconstructPar -latestTime, thendecomposePar -force -latestTimewith the newnumberOfSubdomains, and restart withstartFrom latestTime;. This is simple and works everywhere, but it needs the whole case in one process’s memory. - Redistribute in parallel. Where
redistributeParis available, it can move an existing decomposition to a new process count without reconstructing. It is faster for large cases, but check its options for your version first, because they differ between the two OpenFOAM lines.
Either way, the check above then confirms that the new decomposition matches the new allocation before the solver starts. For the rest of a restart, see restarting an OpenFOAM job after it hits the walltime limit.