Cleaning up OpenFOAM cases on shared storage: foamListTimes, purgeWrite and what they really delete
Published
On a shared cluster, OpenFOAM cases fill project storage in two ways. They use space, because every written time step holds a full copy of every field. And they use inodes, because a decomposed case writes one file per field, per time step, per processor. Quota warnings about inodes usually arrive long before warnings about space. This page goes through the tools that remove time directories, what each one really deletes, and an order of work that does not throw away anything you still need.
Find out what is using the space first
du -sh */ | sort -h # space per case
find . -xdev -type f | wc -l # files under the current directory
Most clusters also have a quota command (quota -s, lfs quota -h -u $USER /project on Lustre,
or a site-specific myquota). Look at both numbers, space and files.
Within one case, the usual offenders are:
processor*directories left over after reconstruction. They hold the same data as the reconstructed times, split into hundreds of files.- Old time directories from transient runs that wrote every few time steps.
postProcessing/, from function objects that write every time step for weeks.- Mesh copies:
constant/polyMeshplus apolyMeshinside time directories, written by mesh motion or bysnappyHexMeshstages without-overwrite.
foamListTimes: list first, delete second
foamListTimes prints the time directories of a case. With -rm, it removes the same directories
it would print. That makes the safe pattern easy: run it once without -rm, read the list, then run
it again with the same options plus -rm.
foamListTimes # every time directory, except 0
foamListTimes -time '0.1:0.5' # only times from 0.1 to 0.5
foamListTimes -time ':0.9' # every time up to 0.9
foamListTimes -latestTime # only the last one
What surprises people:
- Without a time selection,
-rmremoves every time except0.foamListTimes -rmon its own is a “reset to initial conditions” command, not a “tidy up” command. If you meant “remove all but the last time”, you have to say so with a range that ends below the last time. 0is excluded by default. It is only listed (and removed) with-withZero. That protects your initial conditions, but it also means-withZero -rmdeletes them.constantis never touched. Mesh copies inside time directories go with their time directory, though.- Without
-processor, only the reconstructed case is affected. The processor directories keep all their times.
Processor directories
-processor makes foamListTimes look at the times in processor0 instead of the case
directory. With -rm, it removes the selected times from every processor directory:
foamListTimes -processor # times in the decomposed case
foamListTimes -processor -time ':0.9' # check the selection
foamListTimes -processor -time ':0.9' -rm # remove those times from all processors
Use this when a run is still going (or will be restarted) and you only want to drop old times from the decomposed case. Keep the latest time in the processor directories, because that is what a restart reads. See restarting after the walltime limit.
The collated format
In the collated format, there is a single processorsN directory instead of processor0,
processor1 and so on, and each time directory inside it holds a few large files for all
processors. Time-selection tools need to understand that layout. Older releases of
foamListTimes had bugs with it (the openfoam.com tracker lists one as issue 1588). Whatever
version you run, list the processors directory before and after a cleanup and check that the
right times went away. Collated vs uncollated parallel I/O explains
the layout.
purgeWrite: cleanup during the run
purgeWrite in system/controlDict keeps only the last N written time directories while the
solver runs. When it writes a new time, it deletes the oldest one:
writeControl timeStep;
writeInterval 500;
purgeWrite 2; // keep the last two written times; 0 switches purging off
This is the right setting for steady-state runs where only the converged result matters, and for long transient runs where you only need the most recent state for a restart. It is the wrong setting for a transient run whose intermediate times you want to post-process, because nothing lets you get those times back.
In parallel, it works on the processor directories, and purged times are never reconstructed.
If you planned to reconstruct every hundredth time step at the end of the run, purgeWrite will
already have deleted them. Collect what you need during the run with function objects instead
(probes, surface sampling, forces, field averages). They write small files to postProcessing/
and are not purged.
Keep purgeWrite at 2 or more for runs that may be killed. If the job dies while writing the
newest time, the time before it is still complete.
Removing processor directories after reconstruction
Once the times you need are reconstructed, the processor directories are pure duplicates, but only once reconstruction has really finished. Check before deleting:
foamListTimes # reconstructed times
foamListTimes -processor # decomposed times
reconstructPar -newTimes # reconstructs anything still missing
-newTimes only reconstructs times that do not yet exist in the case directory, so it is cheap to
run again as a check. When both lists match the times you want to keep, remove the processor
directories:
rm -rf processor[0-9]* processors[0-9]*
Do this only for runs that are finished. Restarting from a reconstructed case means decomposing again, which is fine but costs time for large meshes.
postProcessing/ and logs
Function objects append to their files on every restart, and some write a new subdirectory per
start time. They rarely use much space, but they can use many files. Compress them with
tar czf postProcessing.tar.gz postProcessing && rm -r postProcessing once the run is finished.
Solver logs compress well too.
A cleanup order that does not lose data
- Stop writing more than needed. Set
writeIntervalandpurgeWritesensibly before the next run, and move intermediate output to function objects. - Reconstruct the times you want to keep (
reconstructPar -time …or-latestTime). - Check with
foamListTimesandfoamListTimes -processorthat they are there. - Delete processor directories of finished runs.
- Remove old reconstructed times with
foamListTimes -time … -rm, after a dry run without-rm. - Archive
postProcessing/and logs. - Tar finished cases you need to keep but won’t touch again. One archive is one inode.
For parameter studies with hundreds of similar cases, the same steps fit in a loop. Run them as a job, not on the login node. See running parameter sweeps with Slurm job arrays for how to keep such studies small from the start.