Restarting an OpenFOAM job after it hits the walltime limit

Published

Long OpenFOAM runs rarely fit into one job. Partitions have walltime limits of a day or a few days, and a transient simulation can need weeks. The usual answer is a chain of jobs, each continuing where the last one stopped. OpenFOAM makes the restart itself easy. The parts that need care are making sure the last written time is usable, and making the chain stop when the run is done.

Restarting from the latest time

system/controlDict decides where a run starts:

startFrom       latestTime;   // instead of startTime
startTime       0;            // ignored with latestTime
stopAt          endTime;
endTime         2.0;

With startFrom latestTime;, the solver picks the newest time directory it finds and continues from there. In a parallel run, it looks in the processor directories, not in the case directory. So a restart does not need reconstructPar and must not run decomposePar -force, which would delete the processor directories and start again from the initial conditions in 0/.

Setting latestTime from the very first run is harmless. With only 0/ present, the latest time is 0. The same job script can then serve both the first run and every restart.

Is the last time directory complete?

When Slurm kills a job at the limit, the solver gets SIGTERM and, after a grace period, SIGKILL. If that happens while it is writing a time step, the newest time directory is incomplete. Some fields are missing, or a file is truncated. A restart from it fails with an error about reading a field, or, worse, carries on from corrupt data.

The best fix is to avoid the situation: have the job script react to a warning signal before the limit and tell the solver to write and stop cleanly. Running on node-local scratch shows that pattern with --signal and stopAt writeNow. It works just as well without local scratch.

If you cannot prevent it, check before restarting. In an uncollated decomposed case, every processor directory should contain the same files for the latest time, and none of them should be empty:

latest=$(foamListTimes -processor -latestTime)
ref=$(ls processor0/"${latest}" | sort)
for p in processor[0-9]*; do
    if [[ "$(ls "${p}/${latest}" 2>/dev/null | sort)" != "${ref}" ]]; then
        echo "${p}/${latest} has different files than processor0"
    fi
done
find processor[0-9]*/"${latest}" -type f -empty -print

This catches missing and empty files. It cannot catch a file cut off in the middle, so it is a first check, not a guarantee. The time step before the latest one is almost always complete, because it was written long before the kill. If anything looks wrong, remove the latest time from all processor directories and restart from the one before:

foamListTimes -processor -latestTime        # check what will be removed
foamListTimes -processor -latestTime -rm

Remember that purgeWrite 1 leaves no earlier time to fall back on. Use at least 2 for runs that will be restarted. Cleaning up OpenFOAM cases explains how purgeWrite and foamListTimes behave.

Chaining jobs with dependencies

Slurm can start a job only after another one ends:

first=$(sbatch --parsable job.sh)
second=$(sbatch --parsable --dependency=afterany:${first} job.sh)
third=$(sbatch --parsable --dependency=afterany:${second} job.sh)

Use afterany, not afterok. A job killed at the walltime limit ends in the state TIMEOUT, which counts as a failure, so afterok never releases the next job. It stays pending with the reason DependencyNeverSatisfied until someone cancels it.

The downside of afterany is that it also continues after a real failure, such as a diverged solver. The job script should therefore check that a restart makes sense before starting the solver.

Self-resubmitting job

A chain of fixed length either stops too early or wastes allocations at the end. An alternative is a job that submits its own successor, and stops doing so once the run has finished:

#!/bin/bash
#SBATCH --job-name=channel-les
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64
#SBATCH --time=24:00:00
#SBATCH --signal=B:USR1@600
#SBATCH --output=%x-%j.out

set -uo pipefail
module load openfoam-com/v2406
cd "${SLURM_SUBMIT_DIR}"

END=$(foamDictionary -entry endTime -value system/controlDict)
latest() { foamListTimes -processor -latestTime 2>/dev/null || echo 0; }
finished() { awk -v t="$(latest)" -v e="${END}" 'BEGIN { exit !(t + 0 >= e + 0) }'; }

if finished; then
    echo "Run already reached endTime ${END}, nothing to do"
    exit 0
fi

# First job of the chain: decompose. Later jobs: reuse the processor directories.
if ! compgen -G "processor[0-9]*" > /dev/null; then
    decomposePar > log.decomposePar 2>&1 || exit 1
fi

# Queue the successor now. It waits until this job ends, however it ends.
next=$(sbatch --parsable --dependency=afterany:${SLURM_JOB_ID} "$0")
echo "Queued follow-up job ${next}"

trap 'foamDictionary -entry stopAt -set writeNow system/controlDict' USR1
mpirun -np "${SLURM_NTASKS}" pimpleFoam -parallel >> log.pimpleFoam 2>&1 &
solver=$!
wait "${solver}"; status=$?
# wait returns early when USR1 arrives; keep waiting until the solver has written and exited
while kill -0 "${solver}" 2>/dev/null; do wait "${solver}"; status=$?; done

# The solver exits cleanly after writeNow, so reset stopAt for the next job.
foamDictionary -entry stopAt -set endTime system/controlDict

if finished; then
    echo "Reached endTime, cancelling follow-up job ${next}"
    scancel "${next}"
elif (( status != 0 )); then
    echo "Solver failed with status ${status}, cancelling follow-up job ${next}" >&2
    scancel "${next}"
    exit 1
fi

A few details:

Restarting with a different core count

The processor directories fix the process count. If the next job should use a different count, reconstruct the latest time and decompose again first. The note on decomposition mismatches describes both ways to do that, and a check that stops the job early if the counts do not match.

Things that carry over and things that don’t

For the basic single-job script these additions build on, see the Slurm job script for OpenFOAM.