Collated vs uncollated parallel I/O in OpenFOAM: inode quotas and when to switch

Published

By default, a decomposed OpenFOAM case writes one directory per MPI process, and in each of those one file per field and time step. This is the uncollated format. It is simple and every tool understands it, but the number of files grows with the core count. On parallel file systems such as Lustre or GPFS, where the metadata server is the bottleneck and project quotas usually limit the number of files as well as the space, that becomes a problem long before the disk is full.

The collated format writes one file per field and time step for all processes together. This page explains the difference, how to switch, and what to watch for afterwards.

How many files a run writes

For an uncollated run, the count is roughly:

files ≈ processes × (mesh files + written times × fields per time)

A modest example: 256 processes, 100 written times, 8 files per time (velocity, pressure, two turbulence fields, phi, and the files under uniform/), plus around 10 mesh and addressing files per processor directory:

256 × (10 + 100 × 8) = 207,360 files

That is one case. A parameter study of fifty such cases is ten million files, which is beyond most project inode quotas. The metadata load also hurts everyone on the file system: each write of a time step creates 2,048 files at once, and ls, du, backups and rm -r on such a case are slow for everybody.

The same run in the collated format writes about 10 + 100 × 8 = 810 files, independent of the number of processes.

How the collated format works

With the collated file handler, decomposePar creates a single directory named after the process count, such as processors256, instead of processor0 to processor255. Each field file inside a time directory holds the data of all processes, one block after the other. The master process gathers the data and writes the file. To keep the solver from waiting on that write, the gathering and writing can run in a separate thread. That needs an MPI library with thread support, and the buffer size is set by the maxThreadFileBufferSize optimisation switch. If the thread cannot be used, the master writes the data itself and the solver waits, which is still fine for most cases.

For very large process counts, a single writer becomes a bottleneck. ioRanks splits the processes into groups, each with its own writer and its own directory (the directory names then carry a rank range). One writer per node is a common choice.

Switching a case to collated

There are three ways to choose the file handler. All of them work for decomposePar, the solver, reconstructPar and the utilities:

# 1. On the command line, per command
decomposePar -fileHandler collated
mpirun -np 256 simpleFoam -parallel -fileHandler collated

# 2. As an environment variable for the whole job
export FOAM_FILEHANDLER=collated
// 3. In the case's system/controlDict (or the site-wide etc/controlDict)
OptimisationSwitches
{
    fileHandler collated;
}

For a cluster, the environment variable in the module file, or the site-wide etc/controlDict, makes collated the default for everyone. The command-line option is useful for testing on single cases first.

To convert an existing decomposed case without decomposing it again, use foamFormatConvert in parallel with the target file handler:

mpirun -np 256 foamFormatConvert -parallel -fileHandler collated

Check the result (ls processors256/) and that the solver restarts from it before deleting the old processor* directories. The same command with -fileHandler uncollated converts back.

What changes after switching

Every step must use the same file handler

A solver started with the uncollated handler on a collated case looks for processor0, finds none, and stops with a message about zero processor directories. The note on decomposition mismatches shows what that looks like. Set the handler once, in the environment or in controlDict, rather than per command.

Post-processing

reconstructPar handles collated cases. If users post-process the reconstructed case, nothing changes for them. Readers that open the decomposed case directly are another matter. Not every reader understands the collated layout. ParaView’s built-in OpenFOAM reader, for example, has long expected processorN directories. Test with your ParaView version before telling users to skip reconstruction. Otherwise reconstruct the times they need, or convert back to uncollated for post-processing.

Scripts that look for processor0

Job scripts and cleanup scripts often contain processor0, processor* or ls -d processor* | wc -l. They need to handle processors* as well. The pre-flight check in the decomposition note above does both.

Cleanup

Deleting a collated case is fast, because there are few files to remove. Removing selected times from a decomposed collated case relies on tools that understand the layout. See the caveat in cleaning up OpenFOAM cases.

Restarts

Restarts work the same as before, as long as the same handler and the same process count are used. If a job dies while the master is writing, the newest time can be incomplete for all processes at once. It is not split across hundreds of files, so it is easier to spot and remove.

When to switch

Switch when any of these apply:

Stay with uncollated for small runs on node-local disk, where the file count does not matter, and for workflows whose tools only read processorN directories.

For a cluster-wide default, collated with one I/O rank per node is a reasonable start. Combined with sensible write intervals (see sizing an OpenFOAM job), it keeps even large studies within quota.