Restarting an OpenFOAM job after it hits the walltime limit
Published
Long OpenFOAM runs rarely fit into one job. Partitions have walltime limits of a day or a few days, and a transient simulation can need weeks. The usual answer is a chain of jobs, each continuing where the last one stopped. OpenFOAM makes the restart itself easy. The parts that need care are making sure the last written time is usable, and making the chain stop when the run is done.
Restarting from the latest time
system/controlDict decides where a run starts:
startFrom latestTime; // instead of startTime
startTime 0; // ignored with latestTime
stopAt endTime;
endTime 2.0;
With startFrom latestTime;, the solver picks the newest time directory it finds and continues
from there. In a parallel run, it looks in the processor directories, not in the case directory.
So a restart does not need reconstructPar and must not run decomposePar -force, which would
delete the processor directories and start again from the initial conditions in 0/.
Setting latestTime from the very first run is harmless. With only 0/ present, the latest time
is 0. The same job script can then serve both the first run and every restart.
Is the last time directory complete?
When Slurm kills a job at the limit, the solver gets SIGTERM and, after a grace period, SIGKILL.
If that happens while it is writing a time step, the newest time directory is incomplete. Some
fields are missing, or a file is truncated. A restart from it fails with an error about reading a
field, or, worse, carries on from corrupt data.
The best fix is to avoid the situation: have the job script react to a warning signal before the
limit and tell the solver to write and stop cleanly. Running on node-local
scratch shows that pattern with --signal and
stopAt writeNow. It works just as well without local scratch.
If you cannot prevent it, check before restarting. In an uncollated decomposed case, every processor directory should contain the same files for the latest time, and none of them should be empty:
latest=$(foamListTimes -processor -latestTime)
ref=$(ls processor0/"${latest}" | sort)
for p in processor[0-9]*; do
if [[ "$(ls "${p}/${latest}" 2>/dev/null | sort)" != "${ref}" ]]; then
echo "${p}/${latest} has different files than processor0"
fi
done
find processor[0-9]*/"${latest}" -type f -empty -print
This catches missing and empty files. It cannot catch a file cut off in the middle, so it is a first check, not a guarantee. The time step before the latest one is almost always complete, because it was written long before the kill. If anything looks wrong, remove the latest time from all processor directories and restart from the one before:
foamListTimes -processor -latestTime # check what will be removed
foamListTimes -processor -latestTime -rm
Remember that purgeWrite 1 leaves no earlier time to fall back on. Use at least 2 for runs that
will be restarted. Cleaning up OpenFOAM cases explains how
purgeWrite and foamListTimes behave.
Chaining jobs with dependencies
Slurm can start a job only after another one ends:
first=$(sbatch --parsable job.sh)
second=$(sbatch --parsable --dependency=afterany:${first} job.sh)
third=$(sbatch --parsable --dependency=afterany:${second} job.sh)
Use afterany, not afterok. A job killed at the walltime limit ends in the state TIMEOUT,
which counts as a failure, so afterok never releases the next job. It stays pending with the
reason DependencyNeverSatisfied until someone cancels it.
The downside of afterany is that it also continues after a real failure, such as a diverged
solver. The job script should therefore check that a restart makes sense before starting the
solver.
Self-resubmitting job
A chain of fixed length either stops too early or wastes allocations at the end. An alternative is a job that submits its own successor, and stops doing so once the run has finished:
#!/bin/bash
#SBATCH --job-name=channel-les
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64
#SBATCH --time=24:00:00
#SBATCH --signal=B:USR1@600
#SBATCH --output=%x-%j.out
set -uo pipefail
module load openfoam-com/v2406
cd "${SLURM_SUBMIT_DIR}"
END=$(foamDictionary -entry endTime -value system/controlDict)
latest() { foamListTimes -processor -latestTime 2>/dev/null || echo 0; }
finished() { awk -v t="$(latest)" -v e="${END}" 'BEGIN { exit !(t + 0 >= e + 0) }'; }
if finished; then
echo "Run already reached endTime ${END}, nothing to do"
exit 0
fi
# First job of the chain: decompose. Later jobs: reuse the processor directories.
if ! compgen -G "processor[0-9]*" > /dev/null; then
decomposePar > log.decomposePar 2>&1 || exit 1
fi
# Queue the successor now. It waits until this job ends, however it ends.
next=$(sbatch --parsable --dependency=afterany:${SLURM_JOB_ID} "$0")
echo "Queued follow-up job ${next}"
trap 'foamDictionary -entry stopAt -set writeNow system/controlDict' USR1
mpirun -np "${SLURM_NTASKS}" pimpleFoam -parallel >> log.pimpleFoam 2>&1 &
solver=$!
wait "${solver}"; status=$?
# wait returns early when USR1 arrives; keep waiting until the solver has written and exited
while kill -0 "${solver}" 2>/dev/null; do wait "${solver}"; status=$?; done
# The solver exits cleanly after writeNow, so reset stopAt for the next job.
foamDictionary -entry stopAt -set endTime system/controlDict
if finished; then
echo "Reached endTime, cancelling follow-up job ${next}"
scancel "${next}"
elif (( status != 0 )); then
echo "Solver failed with status ${status}, cancelling follow-up job ${next}" >&2
scancel "${next}"
exit 1
fi
A few details:
- The successor is submitted early, before the solver starts. A job killed hard at the limit never reaches the end of the script. If submission happened only there, the chain would break exactly when it is needed.
$0resubmits the same script. Inside a Slurm job,$0points to Slurm’s copy of the batch script, whichsbatchaccepts. If your site cleans up those copies, use an absolute path to the script in the case directory.>> log.pimpleFoamappends. Each job adds to the same log, which keeps residual plots continuous. Rotate it by job id if you prefer, withlog.pimpleFoam.${SLURM_JOB_ID}.- The finished check compares numbers with
awk. OpenFOAM time names are decimal numbers, which Bash cannot compare. - The chain stops on a crash. A solver that stops because of
writeNowexits with status 0. Any other status means it failed, and the successor is cancelled. You do not want a diverged run to burn through the queue.
Restarting with a different core count
The processor directories fix the process count. If the next job should use a different count, reconstruct the latest time and decompose again first. The note on decomposition mismatches describes both ways to do that, and a check that stops the job early if the counts do not match.
Things that carry over and things that don’t
- Fields and the time step carry over. With
adjustTimeStep yes;, the currentdeltaTis stored inuniform/timeinside the time directory and is picked up on restart. - Function objects start again. Most write to a new subdirectory of
postProcessing/named after the start time, so plots need stitching together.fieldAveragecan continue averaging from the stored state; check itsrestartOnRestartsetting. - Mesh changes carry over too, as long as the mesh is written into the time directories. That is the case for moving meshes and adaptive refinement.
- Changes to
controlDictmade between jobs take effect on restart. That is the moment to adjustwriteIntervalorendTime, not mid-run.
For the basic single-job script these additions build on, see the Slurm job script for OpenFOAM.