Cleaning up OpenFOAM cases on shared storage: foamListTimes, purgeWrite and what they really delete

Published

On a shared cluster, OpenFOAM cases fill project storage in two ways. They use space, because every written time step holds a full copy of every field. And they use inodes, because a decomposed case writes one file per field, per time step, per processor. Quota warnings about inodes usually arrive long before warnings about space. This page goes through the tools that remove time directories, what each one really deletes, and an order of work that does not throw away anything you still need.

Find out what is using the space first

du -sh */ | sort -h            # space per case
find . -xdev -type f | wc -l   # files under the current directory

Most clusters also have a quota command (quota -s, lfs quota -h -u $USER /project on Lustre, or a site-specific myquota). Look at both numbers, space and files.

Within one case, the usual offenders are:

foamListTimes: list first, delete second

foamListTimes prints the time directories of a case. With -rm, it removes the same directories it would print. That makes the safe pattern easy: run it once without -rm, read the list, then run it again with the same options plus -rm.

foamListTimes                       # every time directory, except 0
foamListTimes -time '0.1:0.5'       # only times from 0.1 to 0.5
foamListTimes -time ':0.9'          # every time up to 0.9
foamListTimes -latestTime           # only the last one

What surprises people:

Processor directories

-processor makes foamListTimes look at the times in processor0 instead of the case directory. With -rm, it removes the selected times from every processor directory:

foamListTimes -processor                     # times in the decomposed case
foamListTimes -processor -time ':0.9'        # check the selection
foamListTimes -processor -time ':0.9' -rm    # remove those times from all processors

Use this when a run is still going (or will be restarted) and you only want to drop old times from the decomposed case. Keep the latest time in the processor directories, because that is what a restart reads. See restarting after the walltime limit.

The collated format

In the collated format, there is a single processorsN directory instead of processor0, processor1 and so on, and each time directory inside it holds a few large files for all processors. Time-selection tools need to understand that layout. Older releases of foamListTimes had bugs with it (the openfoam.com tracker lists one as issue 1588). Whatever version you run, list the processors directory before and after a cleanup and check that the right times went away. Collated vs uncollated parallel I/O explains the layout.

purgeWrite: cleanup during the run

purgeWrite in system/controlDict keeps only the last N written time directories while the solver runs. When it writes a new time, it deletes the oldest one:

writeControl    timeStep;
writeInterval   500;
purgeWrite      2;      // keep the last two written times; 0 switches purging off

This is the right setting for steady-state runs where only the converged result matters, and for long transient runs where you only need the most recent state for a restart. It is the wrong setting for a transient run whose intermediate times you want to post-process, because nothing lets you get those times back.

In parallel, it works on the processor directories, and purged times are never reconstructed. If you planned to reconstruct every hundredth time step at the end of the run, purgeWrite will already have deleted them. Collect what you need during the run with function objects instead (probes, surface sampling, forces, field averages). They write small files to postProcessing/ and are not purged.

Keep purgeWrite at 2 or more for runs that may be killed. If the job dies while writing the newest time, the time before it is still complete.

Removing processor directories after reconstruction

Once the times you need are reconstructed, the processor directories are pure duplicates, but only once reconstruction has really finished. Check before deleting:

foamListTimes                                   # reconstructed times
foamListTimes -processor                        # decomposed times
reconstructPar -newTimes                        # reconstructs anything still missing

-newTimes only reconstructs times that do not yet exist in the case directory, so it is cheap to run again as a check. When both lists match the times you want to keep, remove the processor directories:

rm -rf processor[0-9]* processors[0-9]*

Do this only for runs that are finished. Restarting from a reconstructed case means decomposing again, which is fine but costs time for large meshes.

postProcessing/ and logs

Function objects append to their files on every restart, and some write a new subdirectory per start time. They rarely use much space, but they can use many files. Compress them with tar czf postProcessing.tar.gz postProcessing && rm -r postProcessing once the run is finished. Solver logs compress well too.

A cleanup order that does not lose data

  1. Stop writing more than needed. Set writeInterval and purgeWrite sensibly before the next run, and move intermediate output to function objects.
  2. Reconstruct the times you want to keep (reconstructPar -time … or -latestTime).
  3. Check with foamListTimes and foamListTimes -processor that they are there.
  4. Delete processor directories of finished runs.
  5. Remove old reconstructed times with foamListTimes -time … -rm, after a dry run without -rm.
  6. Archive postProcessing/ and logs.
  7. Tar finished cases you need to keep but won’t touch again. One archive is one inode.

For parameter studies with hundreds of similar cases, the same steps fit in a loop. Run them as a job, not on the login node. See running parameter sweeps with Slurm job arrays for how to keep such studies small from the start.