Keeping OpenFOAM versions apart on a shared cluster: modules, containers and the two OpenFOAM lines
Published
A user asks for “OpenFOAM 12”, another for “OpenFOAM v2406”, and a third has a case from a paper that says “OpenFOAM 2.3”. All three are called OpenFOAM, but they are not versions of one program in the way 2.3, 12 and v2406 suggest. On a shared cluster, mixing them up causes errors that look like broken cases. This note covers the naming, a module layout that keeps the lines apart, and when to use containers instead.
Two lines with similar names
Since the mid-2010s, OpenFOAM has been developed in two separate lines, both open source under the GPL:
- The openfoam.org line, released by the OpenFOAM Foundation and developed mainly by CFD
Direct. Versions are plain numbers that go up by one each year: 8, 9, 10, 11, 12 and so on. A
rolling development version is called
OpenFOAM-dev. - The openfoam.com line, released by OpenCFD Ltd. Versions are named after the
release date as
vYYMM, with releases in June and December: v2306, v2312, v2406, v2412 and so on.
Both share the code up to about 2016 (version 4.x on the .org side, v3.0+ on the .com side) and have diverged since. Each line has added features the other lacks, and both have renamed and restructured things. Case files written for one often do not run unchanged on the other. Some examples of the differences users run into:
- Solvers. From version 11, the .org line replaced most solver executables (
simpleFoam,pimpleFoamand others) with modular solvers run throughfoamRun, selected by asolverentry incontrolDict. The .com line still hassimpleFoamand its relatives. - Dictionary names. The .org line renamed
turbulencePropertiestomomentumTransport(from version 8) andtransportPropertiestophysicalProperties(from version 10). The .com line kept the old names. - Utilities and options. Tools like
foamDictionary,foamListTimesorredistributeParexist in both lines but differ in options and behaviour.
So “which OpenFOAM?” always needs two answers: which line, and which version. The solver log
answers both after the fact. Its header shows the version (Version: 12 or Version: v2406) and
the build. Ask users to keep logs with their results. That makes a result traceable to the exact
software that produced it.
Naming modules so nobody confuses them
Put the line into the module name, not only the version:
openfoam-org/11
openfoam-org/12
openfoam-com/v2312
openfoam-com/v2406
With a single openfoam module and versions 12 and v2406 side by side, module load openfoam
picks whichever sorts as the default, and users do not notice they switched lines. Separate names
make the choice explicit. Set a default per line, and let module avail openfoam show both lines
next to each other.
Mark the two lines as conflicting (a family in Lmod, a conflict in Environment Modules), so
loading one unloads (or refuses) the other. Both set the same
environment variables (WM_PROJECT_DIR, FOAM_APPBIN and many more), and a shell with parts of
both is the source of the strangest errors.
Writing the module file
OpenFOAM’s environment comes from sourcing etc/bashrc in the installation. It sets dozens of
variables and depends on the compiler and MPI it was built with. Rewriting all of that by hand in a
module file is error-prone, and it breaks on the next version. Both common module systems can
instead run the shell script and capture the changes it makes.
Lmod (version 8.6 or later), in openfoam-com/v2406.lua:
family("openfoam") -- only one OpenFOAM module of either line at a time
depends_on("openmpi/4.1") -- the MPI OpenFOAM was built against
source_sh("bash", "/opt/openfoam/v2406/etc/bashrc")
Environment Modules (version 4.6 or later), in openfoam-org/12:
#%Module
conflict openfoam-org openfoam-com
module load openmpi/4.1
source-sh bash /opt/openfoam/12/etc/bashrc
Both evaluate the script when the module is loaded, record what changed, and undo exactly those
changes on unload. Load the MPI module first, the one OpenFOAM was compiled with. OpenFOAM’s
bashrc picks up the MPI from the environment when WM_MPLIB is set to the system MPI, and a
different MPI at run time than at build time leads to crashes at start-up.
Test the module the way users will use it: in a fresh login shell, load it, run a tutorial in
parallel through Slurm, unload it, and check that env | grep -i foam is empty afterwards.
When containers are the better choice
Building every version of both lines from source, for every compiler and MPI combination, is a lot of work for admins. Old versions are worse still, because they no longer build with current compilers. Containers solve this. On HPC systems, that means Apptainer (formerly Singularity), which runs images without root privileges.
Both lines publish container images or recipes. OpenCFD provides images for the .com line on Docker Hub, and CFD Direct documents container use for the .org line. An image can be pulled and converted in one step. For example:
apptainer pull openfoam-v2406.sif docker://opencfd/openfoam-default:2406
Check each project’s documentation for the current image names and tags.
apptainer exec does not start a login shell, so the OpenFOAM environment inside the image must
be sourced explicitly. Find where the image installs OpenFOAM (for example with
apptainer exec openfoam-v2406.sif ls /usr/lib/openfoam), then:
apptainer exec openfoam-v2406.sif \
bash -c 'source /usr/lib/openfoam/openfoam2406/etc/bashrc && blockMesh'
MPI across nodes
For parallel runs across nodes, the usual method is the hybrid model: the host’s mpirun (or
srun) starts one container process per rank, and the MPI library inside the container talks to
the host’s process manager and network:
mpirun -np "${SLURM_NTASKS}" apptainer exec openfoam-v2406.sif \
bash -c 'source /usr/lib/openfoam/openfoam2406/etc/bashrc && simpleFoam -parallel'
This only works if the MPI inside the image is compatible with the one on the host, typically the
same implementation and a close version, and if the image’s MPI can use the cluster’s network. Run
a small two-node job and check nProcs in the log and the time per step before declaring a
container ready. If the interconnect is not used, the run still works but communicates over plain
TCP and scales badly. The alternative is to bind-mount the host’s MPI into the container, which
gives full network performance at the cost of a more fragile setup.
A module file can hide all of this. A module openfoam-com/v2406-container can define a shell
function or a wrapper script that runs commands through the image, so users write simpleFoam
as usual.
A short policy that works
- Provide the current and previous release of each line as modules, named
openfoam-org/…andopenfoam-com/…, and keep them conflicting. - Serve older versions as Apptainer images only, with a wrapper module.
- Write the line and version into job templates (see the Slurm job script for OpenFOAM), so a restart months later loads the same software. Restarting a run on a different version is a common source of results that cannot be explained later. The restart guide assumes the same version throughout.
- Ask users to state “OpenFOAM v2406 (openfoam.com)” or “OpenFOAM 12 (openfoam.org)” in papers and reports. “OpenFOAM” alone is not enough to reproduce a result.