A Slurm job script for OpenFOAM that decomposes, solves and reconstructs in one go
Published
Most parallel OpenFOAM jobs on a cluster run the same three steps: split the case with
decomposePar, run the solver under MPI, and put the pieces back together with reconstructPar.
The step that goes wrong most often is not any of the three. It is the number of cores, which
appears in two places: in system/decomposeParDict as numberOfSubdomains, and on the mpirun
or srun line. When someone changes one and forgets the other, the solver stops straight away
with an error about processor directories. The note on that error
covers how to read it.
The script below avoids the problem by setting the core count once, in the #SBATCH header, and
reading it from Slurm for everything else.
The script
#!/bin/bash
#SBATCH --job-name=pipe-flow
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=64
#SBATCH --time=12:00:00
#SBATCH --output=%x-%j.out
set -euo pipefail
# 1. Environment. Adjust to how OpenFOAM is installed on your cluster.
module purge
module load openfoam-com/v2406
# If there is no module, source the installation's bashrc instead:
# source /opt/openfoam/etc/bashrc
# 2. The one place where the core count comes from.
NP=${SLURM_NTASKS}
SOLVER=$(foamDictionary -entry application -value system/controlDict)
cd "${SLURM_SUBMIT_DIR}"
echo "Job ${SLURM_JOB_ID}: ${SOLVER} on ${NP} tasks, nodes ${SLURM_JOB_NODELIST}"
# 3. Write the core count into decomposeParDict, then decompose.
foamDictionary -entry numberOfSubdomains -set "${NP}" system/decomposeParDict
decomposePar -force > log.decomposePar 2>&1
# 4. Solve. Use srun if your MPI was built with Slurm (PMI/PMIx) support,
# otherwise mpirun, which picks up the allocation from Slurm itself.
srun "${SOLVER}" -parallel > "log.${SOLVER}" 2>&1
# mpirun -np "${NP}" "${SOLVER}" -parallel > "log.${SOLVER}" 2>&1
# 5. Reconstruct only the last written time.
reconstructPar -latestTime > log.reconstructPar 2>&1
Submit it from the case directory with sbatch job.sh. The script assumes it runs there, which
is why it changes to $SLURM_SUBMIT_DIR.
What each part does
The header
--nodes and --ntasks-per-node decide the core count. Slurm exports their product as
SLURM_NTASKS. If you prefer to ask for a total with --ntasks=128 and let Slurm place the tasks,
that works too, and SLURM_NTASKS still holds the right number. Ask for whole nodes when you can.
OpenFOAM is sensitive to memory bandwidth, and sharing a node with another job’s memory traffic can
slow a run down more than the extra cores speed it up.
--output=%x-%j.out names the Slurm log after the job name and job id. Keep the solver’s own
output in a separate log.<solver> file, so the Slurm log stays short and only shows the job’s
progress and errors.
set -euo pipefail
Without it, a failed decomposePar is followed by a solver run on a stale or missing
decomposition, and you lose the allocation finding out. With -e, the script stops at the first
command that fails, and the Slurm log shows which one.
Reading the solver from controlDict
foamDictionary -entry application -value system/controlDict prints the application entry,
such as simpleFoam or pimpleFoam. The same script then works for any case without editing.
On OpenFOAM Foundation releases from version 11 onwards, cases usually name foamRun as the
application and the solver module in a separate solver entry, and the script still picks up the
right command. The note on keeping versions apart
explains why the two OpenFOAM lines differ here.
Writing numberOfSubdomains
foamDictionary -entry numberOfSubdomains -set "${NP}" rewrites that one entry in place and keeps
the rest of the file. This is the line that ties the decomposition to the allocation. Two caveats:
- Only automatic methods adapt on their own. With
method scotch;(orptscotch, orkahipwhere available), the decomposition followsnumberOfSubdomainsand nothing else needs to change. Withsimpleorhierarchical, the coefficientsn (nx ny nz)must multiply to the same number, anddecomposeParrefuses to run if they don’t. Either switch to scotch for scripted runs or computenin the script as well. The note on decomposition methods shows how to compare them on your own case. - The case is changed on disk. That is usually what you want, because the next restart then uses the same count. If several jobs share one case directory, they must not run at the same time anyway.
decomposePar -force
-force removes existing processor* directories before decomposing. Without it, decomposePar
stops if they exist, which protects results from an earlier run. Only use -force in a script
for a fresh start. A restarting job must keep the processor directories and skip this step
entirely. See restarting after the walltime limit.
decomposePar runs on one core. For meshes beyond a few tens of millions of cells, it can take
long enough to matter and needs memory for the whole mesh on one node. Larger cases are better
off meshing in parallel (for example with snappyHexMesh -parallel on a coarse decomposed
background mesh) or, in versions that support it, decomposing with redistributePar -decompose -parallel, so the full mesh never sits on one core.
srun or mpirun
With an MPI library built against Slurm’s process management interface, srun starts one solver
process per task across all allocated nodes and needs no -np. With an MPI library that lacks it,
srun starts every process as an independent serial run. Each process then writes to the same
case at the same time. You notice it when the log shows the banner once per process and
nProcs : 1. mpirun from Open MPI or Intel MPI reads the Slurm allocation by itself, so
mpirun -np "${NP}" is the safe fallback. Your cluster’s documentation usually says which one to
use. When in doubt, check the line nProcs : 128 near the top of the solver log.
-parallel is not optional. Without it, the solver ignores the processor directories and runs the
undecomposed case on every rank.
reconstructPar -latestTime
Reconstructing every written time takes long and doubles the disk use of the case. Most users only
need the final state, or a few selected times, for post-processing. -latestTime reconstructs just
the last one. Use -time 0.5:1.0 for a range, or -newTimes to only add times that are not yet
reconstructed. If the run uses the collated file format, reconstruction works the same way. See
collated vs uncollated parallel I/O for when to use it.
Once the reconstructed result is checked, the processor directories can go. The page on cleaning up cases covers what to delete and in which order.
Variations
- Mesh in the same job. Add
blockMeshandsnappyHexMeshbefore step 3. Run them in parallel if the mesh is large, with their own decomposition step. - Run on node-local disk. On clusters with fast local SSDs, staging the case there can cut I/O time a lot. Running on node-local scratch shows how to copy results back even when the job runs out of time.
- Many similar runs. For parameter studies, do not copy this script a hundred times. Use a Slurm job array.
- Choosing the core count. How many cores a case can use well depends on its cell count. See sizing an OpenFOAM job.
Checklist before you submit
foamDictionary -entry method -value system/decomposeParDictprints an automatic method, or thencoefficients match the core count.controlDictwrites often enough that a timeout does not throw away hours of work.- The case contains no
processor*directories you still need, because-forcedeletes them. - The module (or bashrc) in the script is the same OpenFOAM version the case was set up with.