Running decomposePar and reconstructPar on node-local scratch

Published

Many cluster nodes have a local SSD that jobs can use as scratch space. For OpenFOAM it is attractive: decomposePar, reconstructPar and an uncollated parallel run create and read many small files, and a local disk handles that far better than a shared parallel file system under load. The catch is that local scratch is usually wiped when the job ends. A job that runs out of walltime, the common case for long runs, loses everything that was not copied back.

This note shows a staging pattern that copies results back on normal exit and on timeout.

Where local scratch is

It differs per cluster. Common names are $TMPDIR, $SLURM_TMPDIR, $LOCAL_SCRATCH or a path such as /scratch/local/$SLURM_JOB_ID. Some clusters require you to request it, for example with #SBATCH --tmp=200G or a --gres option. Check your cluster’s documentation, and check the size: the decomposed case, all written times and the reconstructed result must fit at once.

The single-node pattern

The script below handles a job on one node. Multi-node jobs are covered further down.

#!/bin/bash
#SBATCH --job-name=pipe-flow-local
#SBATCH --nodes=1
#SBATCH --ntasks=64
#SBATCH --time=24:00:00
#SBATCH --signal=B:USR1@600
#SBATCH --output=%x-%j.out

set -uo pipefail
module load openfoam-com/v2406

CASE_HOME=${SLURM_SUBMIT_DIR}
WORK=${TMPDIR:?no node-local scratch}/${SLURM_JOB_ID}
NP=${SLURM_NTASKS}

copy_back() {
    echo "$(date): copying results back to ${CASE_HOME}"
    cd "${WORK}"
    reconstructPar -newTimes > log.reconstructPar 2>&1 || echo "reconstructPar failed" >&2
    rsync -a --exclude 'processor*' "${WORK}/" "${CASE_HOME}/"
    # Keep the decomposed latest time too, so a restart does not have to decompose again
    rsync -a "${WORK}"/processor* "${CASE_HOME}/" 2>/dev/null || true
}

on_timeout() {
    echo "$(date): walltime nearly reached, asking the solver to write and stop"
    foamDictionary -entry stopAt -set writeNow "${WORK}/system/controlDict"
    wait "${SOLVER_PID}"
    copy_back
    exit 0
}
trap on_timeout USR1

# Stage in: everything except old processor directories
mkdir -p "${WORK}"
rsync -a --exclude 'processor*' "${CASE_HOME}/" "${WORK}/"
cd "${WORK}"

foamDictionary -entry numberOfSubdomains -set "${NP}" system/decomposeParDict
decomposePar -force > log.decomposePar 2>&1 || exit 1

# Run the solver in the background, so the shell can react to the signal
mpirun -np "${NP}" simpleFoam -parallel > log.simpleFoam 2>&1 &
SOLVER_PID=$!
wait "${SOLVER_PID}"
STATUS=$?

copy_back
exit "${STATUS}"

How it works

--signal=B:USR1@600

Slurm sends SIGUSR1 to the batch shell (that is what the B: means) 600 seconds before the walltime limit. Without B:, the signal goes to the job steps, where nothing catches it. Pick a lead time long enough for the solver to finish its current time step, write, reconstruct and copy back. For large cases, ten minutes is a minimum. Slurm checks for due signals periodically and may send this one up to a minute earlier than requested, which does no harm here.

Why the solver runs in the background

Bash only runs a trap between commands. While a foreground mpirun runs, the trap waits until it returns, which happens when Slurm kills the job at the limit. That is too late. Starting the solver with & and then calling wait lets the trap run as soon as the signal arrives, because wait returns early when a trapped signal comes in.

stopAt writeNow

The trap does not kill the solver. It changes stopAt in controlDict to writeNow. OpenFOAM re-reads controlDict while it runs (as long as runTimeModifiable is on, the default), so at the end of the current time step it writes all fields and exits cleanly. The written time is complete and can be used to restart. Killing the solver instead would leave the newest time directory half-written, or lose everything since the last regular write.

Some versions also offer optimisation switches (writeNowSignal, stopAtWriteNowSignal) that make the solver itself react to a signal. They work, but the controlDict route above works in every version and keeps the logic visible in the job script.

Copying back

reconstructPar -newTimes reconstructs whatever was written since the last reconstruction. Running it on local disk is the point of the exercise: it reads every processor file, which is exactly the workload shared file systems handle badly. The first rsync then copies back the reconstructed case. The second copies the processor directories, so that a restart can carry on without decomposing again. If you only care about the final result, drop the second rsync. It is usually the larger one.

set -e is left out on purpose. A failure inside the solver must still lead to copy_back, so you can see the logs and the last written time. Errors are handled per command instead.

Multi-node jobs

Local scratch is local to each node. A run across several nodes needs each node’s processor directories on that node’s disk. OpenFOAM supports this with a distributed decomposition: distributed yes; and a roots list in decomposeParDict that gives the case root on each node. Staging then means copying the right processor directories to each node (for example with srun --ntasks-per-node=1 running an rsync per node) and collecting them again at the end.

This is fiddly and easy to get wrong. Before building it, check whether multi-node runs need local scratch at all. The collated file format removes most of the small-file load from the shared file system, and is often enough on its own.

When not to bother

The same --signal and stopAt writeNow trick is useful without local scratch too. Adding just the trap to the standard job script makes every timeout end with a clean, restartable time directory.