Running decomposePar and reconstructPar on node-local scratch
Published
Many cluster nodes have a local SSD that jobs can use as scratch space. For OpenFOAM it is
attractive: decomposePar, reconstructPar and an uncollated parallel run create and read many
small files, and a local disk handles that far better than a shared parallel file system under load.
The catch is that local scratch is usually wiped when the job ends. A job that runs out of
walltime, the common case for long runs, loses everything that was not copied back.
This note shows a staging pattern that copies results back on normal exit and on timeout.
Where local scratch is
It differs per cluster. Common names are $TMPDIR, $SLURM_TMPDIR, $LOCAL_SCRATCH or a path
such as /scratch/local/$SLURM_JOB_ID. Some clusters require you to request it, for example with
#SBATCH --tmp=200G or a --gres option. Check your cluster’s documentation, and check the size:
the decomposed case, all written times and the reconstructed result must fit at once.
The single-node pattern
The script below handles a job on one node. Multi-node jobs are covered further down.
#!/bin/bash
#SBATCH --job-name=pipe-flow-local
#SBATCH --nodes=1
#SBATCH --ntasks=64
#SBATCH --time=24:00:00
#SBATCH --signal=B:USR1@600
#SBATCH --output=%x-%j.out
set -uo pipefail
module load openfoam-com/v2406
CASE_HOME=${SLURM_SUBMIT_DIR}
WORK=${TMPDIR:?no node-local scratch}/${SLURM_JOB_ID}
NP=${SLURM_NTASKS}
copy_back() {
echo "$(date): copying results back to ${CASE_HOME}"
cd "${WORK}"
reconstructPar -newTimes > log.reconstructPar 2>&1 || echo "reconstructPar failed" >&2
rsync -a --exclude 'processor*' "${WORK}/" "${CASE_HOME}/"
# Keep the decomposed latest time too, so a restart does not have to decompose again
rsync -a "${WORK}"/processor* "${CASE_HOME}/" 2>/dev/null || true
}
on_timeout() {
echo "$(date): walltime nearly reached, asking the solver to write and stop"
foamDictionary -entry stopAt -set writeNow "${WORK}/system/controlDict"
wait "${SOLVER_PID}"
copy_back
exit 0
}
trap on_timeout USR1
# Stage in: everything except old processor directories
mkdir -p "${WORK}"
rsync -a --exclude 'processor*' "${CASE_HOME}/" "${WORK}/"
cd "${WORK}"
foamDictionary -entry numberOfSubdomains -set "${NP}" system/decomposeParDict
decomposePar -force > log.decomposePar 2>&1 || exit 1
# Run the solver in the background, so the shell can react to the signal
mpirun -np "${NP}" simpleFoam -parallel > log.simpleFoam 2>&1 &
SOLVER_PID=$!
wait "${SOLVER_PID}"
STATUS=$?
copy_back
exit "${STATUS}"
How it works
--signal=B:USR1@600
Slurm sends SIGUSR1 to the batch shell (that is what the B: means) 600 seconds before the
walltime limit. Without B:, the signal goes to the job steps, where nothing catches it. Pick a
lead time long enough for the solver to finish its current time step, write, reconstruct and copy
back. For large cases, ten minutes is a minimum. Slurm checks for due signals periodically and may
send this one up to a minute earlier than requested, which does no harm here.
Why the solver runs in the background
Bash only runs a trap between commands. While a foreground mpirun runs, the trap waits until it
returns, which happens when Slurm kills the job at the limit. That is too late. Starting the solver
with & and then calling wait lets the trap run as soon as the signal arrives, because wait
returns early when a trapped signal comes in.
stopAt writeNow
The trap does not kill the solver. It changes stopAt in controlDict to writeNow. OpenFOAM
re-reads controlDict while it runs (as long as runTimeModifiable is on, the default), so at the
end of the current time step it writes all fields and exits cleanly. The written time is complete
and can be used to restart. Killing the solver instead would leave the newest time directory
half-written, or lose everything since the last regular write.
Some versions also offer optimisation switches (writeNowSignal, stopAtWriteNowSignal) that
make the solver itself react to a signal. They work, but the controlDict route above works in
every version and keeps the logic visible in the job script.
Copying back
reconstructPar -newTimes reconstructs whatever was written since the last reconstruction.
Running it on local disk is the point of the exercise: it reads every processor file, which is
exactly the workload shared file systems handle badly. The first rsync then copies back the
reconstructed case. The second copies the processor directories, so that
a restart can carry on without decomposing again. If you
only care about the final result, drop the second rsync. It is usually the larger one.
set -e is left out on purpose. A failure inside the solver must still lead to copy_back, so
you can see the logs and the last written time. Errors are handled per command instead.
Multi-node jobs
Local scratch is local to each node. A run across several nodes needs each node’s processor
directories on that node’s disk. OpenFOAM supports this with a distributed decomposition:
distributed yes; and a roots list in decomposeParDict that gives the case root on each
node. Staging then means copying the right processor directories to each node (for example with
srun --ntasks-per-node=1 running an rsync per node) and collecting them again at the end.
This is fiddly and easy to get wrong. Before building it, check whether multi-node runs need local scratch at all. The collated file format removes most of the small-file load from the shared file system, and is often enough on its own.
When not to bother
- Small cases that finish in minutes: the copying costs more than it saves.
- Cases with huge outputs that would not fit on the local disk. Write less often, or use function objects for intermediate data.
- Clusters whose parallel file system is fast and lightly loaded. Measure first: run the same
case once on shared storage and once on local scratch, and compare the
ExecutionTimeandClockTimelines at the end of the solver log. A large gap between the two on shared storage points to time lost waiting for I/O.
The same --signal and stopAt writeNow trick is useful without local scratch too. Adding just
the trap to the standard job script makes every timeout end
with a clean, restartable time directory.