Collated vs uncollated parallel I/O in OpenFOAM: inode quotas and when to switch
Published
By default, a decomposed OpenFOAM case writes one directory per MPI process, and in each of those one file per field and time step. This is the uncollated format. It is simple and every tool understands it, but the number of files grows with the core count. On parallel file systems such as Lustre or GPFS, where the metadata server is the bottleneck and project quotas usually limit the number of files as well as the space, that becomes a problem long before the disk is full.
The collated format writes one file per field and time step for all processes together. This page explains the difference, how to switch, and what to watch for afterwards.
How many files a run writes
For an uncollated run, the count is roughly:
files ≈ processes × (mesh files + written times × fields per time)
A modest example: 256 processes, 100 written times, 8 files per time (velocity, pressure, two
turbulence fields, phi, and the files under uniform/), plus around 10 mesh and addressing files
per processor directory:
256 × (10 + 100 × 8) = 207,360 files
That is one case. A parameter study of fifty such cases is ten million files, which is beyond most
project inode quotas. The metadata load also hurts everyone on the file system: each write of a
time step creates 2,048 files at once, and ls, du, backups and rm -r on such a case are slow
for everybody.
The same run in the collated format writes about 10 + 100 × 8 = 810 files, independent of the
number of processes.
How the collated format works
With the collated file handler, decomposePar creates a single directory named after the process
count, such as processors256, instead of processor0 to processor255. Each field file inside a
time directory holds the data of all processes, one block after the other. The master process
gathers the data and writes the file. To keep the solver from waiting on that write, the gathering
and writing can run in a separate thread. That needs an MPI library with thread support, and the
buffer size is set by the maxThreadFileBufferSize optimisation switch. If the thread cannot be
used, the master writes the data itself and the solver waits, which is still fine for most cases.
For very large process counts, a single writer becomes a bottleneck. ioRanks splits the processes
into groups, each with its own writer and its own directory (the directory names then carry a rank
range). One writer per node is a common choice.
Switching a case to collated
There are three ways to choose the file handler. All of them work for decomposePar, the solver,
reconstructPar and the utilities:
# 1. On the command line, per command
decomposePar -fileHandler collated
mpirun -np 256 simpleFoam -parallel -fileHandler collated
# 2. As an environment variable for the whole job
export FOAM_FILEHANDLER=collated
// 3. In the case's system/controlDict (or the site-wide etc/controlDict)
OptimisationSwitches
{
fileHandler collated;
}
For a cluster, the environment variable in the module file, or the site-wide etc/controlDict,
makes collated the default for everyone. The command-line option is useful for testing on single
cases first.
To convert an existing decomposed case without decomposing it again, use foamFormatConvert in
parallel with the target file handler:
mpirun -np 256 foamFormatConvert -parallel -fileHandler collated
Check the result (ls processors256/) and that the solver restarts from it before deleting the old
processor* directories. The same command with -fileHandler uncollated converts back.
What changes after switching
Every step must use the same file handler
A solver started with the uncollated handler on a collated case looks for processor0, finds none,
and stops with a message about zero processor directories. The note on decomposition
mismatches shows what that looks like. Set the handler once,
in the environment or in controlDict, rather than per command.
Post-processing
reconstructPar handles collated cases. If users post-process the reconstructed case, nothing
changes for them. Readers that open the decomposed case directly are another matter. Not every
reader understands the collated layout. ParaView’s built-in OpenFOAM reader, for example, has
long expected processorN directories. Test with your ParaView version before telling users to
skip reconstruction. Otherwise reconstruct the times they need, or convert back to uncollated
for post-processing.
Scripts that look for processor0
Job scripts and cleanup scripts often contain processor0, processor* or
ls -d processor* | wc -l. They need to handle processors* as well. The pre-flight check in the
decomposition note above does both.
Cleanup
Deleting a collated case is fast, because there are few files to remove. Removing selected times from a decomposed collated case relies on tools that understand the layout. See the caveat in cleaning up OpenFOAM cases.
Restarts
Restarts work the same as before, as long as the same handler and the same process count are used. If a job dies while the master is writing, the newest time can be incomplete for all processes at once. It is not split across hundreds of files, so it is easier to spot and remove.
When to switch
Switch when any of these apply:
- cases run on more than a few dozen processes,
- the file system is a parallel file system with an inode quota,
- users run many cases at once (parameter studies, optimisation loops),
- metadata operations on the file system (listing, backup, deletion) are already slow.
Stay with uncollated for small runs on node-local disk, where the file count does not matter, and
for workflows whose tools only read processorN directories.
For a cluster-wide default, collated with one I/O rank per node is a reasonable start. Combined with sensible write intervals (see sizing an OpenFOAM job), it keeps even large studies within quota.