A Slurm job script for OpenFOAM that decomposes, solves and reconstructs in one go

Published

Most parallel OpenFOAM jobs on a cluster run the same three steps: split the case with decomposePar, run the solver under MPI, and put the pieces back together with reconstructPar. The step that goes wrong most often is not any of the three. It is the number of cores, which appears in two places: in system/decomposeParDict as numberOfSubdomains, and on the mpirun or srun line. When someone changes one and forgets the other, the solver stops straight away with an error about processor directories. The note on that error covers how to read it.

The script below avoids the problem by setting the core count once, in the #SBATCH header, and reading it from Slurm for everything else.

The script

#!/bin/bash
#SBATCH --job-name=pipe-flow
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=64
#SBATCH --time=12:00:00
#SBATCH --output=%x-%j.out

set -euo pipefail

# 1. Environment. Adjust to how OpenFOAM is installed on your cluster.
module purge
module load openfoam-com/v2406
# If there is no module, source the installation's bashrc instead:
# source /opt/openfoam/etc/bashrc

# 2. The one place where the core count comes from.
NP=${SLURM_NTASKS}
SOLVER=$(foamDictionary -entry application -value system/controlDict)

cd "${SLURM_SUBMIT_DIR}"
echo "Job ${SLURM_JOB_ID}: ${SOLVER} on ${NP} tasks, nodes ${SLURM_JOB_NODELIST}"

# 3. Write the core count into decomposeParDict, then decompose.
foamDictionary -entry numberOfSubdomains -set "${NP}" system/decomposeParDict
decomposePar -force > log.decomposePar 2>&1

# 4. Solve. Use srun if your MPI was built with Slurm (PMI/PMIx) support,
#    otherwise mpirun, which picks up the allocation from Slurm itself.
srun "${SOLVER}" -parallel > "log.${SOLVER}" 2>&1
# mpirun -np "${NP}" "${SOLVER}" -parallel > "log.${SOLVER}" 2>&1

# 5. Reconstruct only the last written time.
reconstructPar -latestTime > log.reconstructPar 2>&1

Submit it from the case directory with sbatch job.sh. The script assumes it runs there, which is why it changes to $SLURM_SUBMIT_DIR.

What each part does

The header

--nodes and --ntasks-per-node decide the core count. Slurm exports their product as SLURM_NTASKS. If you prefer to ask for a total with --ntasks=128 and let Slurm place the tasks, that works too, and SLURM_NTASKS still holds the right number. Ask for whole nodes when you can. OpenFOAM is sensitive to memory bandwidth, and sharing a node with another job’s memory traffic can slow a run down more than the extra cores speed it up.

--output=%x-%j.out names the Slurm log after the job name and job id. Keep the solver’s own output in a separate log.<solver> file, so the Slurm log stays short and only shows the job’s progress and errors.

set -euo pipefail

Without it, a failed decomposePar is followed by a solver run on a stale or missing decomposition, and you lose the allocation finding out. With -e, the script stops at the first command that fails, and the Slurm log shows which one.

Reading the solver from controlDict

foamDictionary -entry application -value system/controlDict prints the application entry, such as simpleFoam or pimpleFoam. The same script then works for any case without editing. On OpenFOAM Foundation releases from version 11 onwards, cases usually name foamRun as the application and the solver module in a separate solver entry, and the script still picks up the right command. The note on keeping versions apart explains why the two OpenFOAM lines differ here.

Writing numberOfSubdomains

foamDictionary -entry numberOfSubdomains -set "${NP}" rewrites that one entry in place and keeps the rest of the file. This is the line that ties the decomposition to the allocation. Two caveats:

decomposePar -force

-force removes existing processor* directories before decomposing. Without it, decomposePar stops if they exist, which protects results from an earlier run. Only use -force in a script for a fresh start. A restarting job must keep the processor directories and skip this step entirely. See restarting after the walltime limit.

decomposePar runs on one core. For meshes beyond a few tens of millions of cells, it can take long enough to matter and needs memory for the whole mesh on one node. Larger cases are better off meshing in parallel (for example with snappyHexMesh -parallel on a coarse decomposed background mesh) or, in versions that support it, decomposing with redistributePar -decompose -parallel, so the full mesh never sits on one core.

srun or mpirun

With an MPI library built against Slurm’s process management interface, srun starts one solver process per task across all allocated nodes and needs no -np. With an MPI library that lacks it, srun starts every process as an independent serial run. Each process then writes to the same case at the same time. You notice it when the log shows the banner once per process and nProcs : 1. mpirun from Open MPI or Intel MPI reads the Slurm allocation by itself, so mpirun -np "${NP}" is the safe fallback. Your cluster’s documentation usually says which one to use. When in doubt, check the line nProcs : 128 near the top of the solver log.

-parallel is not optional. Without it, the solver ignores the processor directories and runs the undecomposed case on every rank.

reconstructPar -latestTime

Reconstructing every written time takes long and doubles the disk use of the case. Most users only need the final state, or a few selected times, for post-processing. -latestTime reconstructs just the last one. Use -time 0.5:1.0 for a range, or -newTimes to only add times that are not yet reconstructed. If the run uses the collated file format, reconstruction works the same way. See collated vs uncollated parallel I/O for when to use it.

Once the reconstructed result is checked, the processor directories can go. The page on cleaning up cases covers what to delete and in which order.

Variations

Checklist before you submit

  1. foamDictionary -entry method -value system/decomposeParDict prints an automatic method, or the n coefficients match the core count.
  2. controlDict writes often enough that a timeout does not throw away hours of work.
  3. The case contains no processor* directories you still need, because -force deletes them.
  4. The module (or bashrc) in the script is the same OpenFOAM version the case was set up with.