Keeping OpenFOAM versions apart on a shared cluster: modules, containers and the two OpenFOAM lines

Published

A user asks for “OpenFOAM 12”, another for “OpenFOAM v2406”, and a third has a case from a paper that says “OpenFOAM 2.3”. All three are called OpenFOAM, but they are not versions of one program in the way 2.3, 12 and v2406 suggest. On a shared cluster, mixing them up causes errors that look like broken cases. This note covers the naming, a module layout that keeps the lines apart, and when to use containers instead.

Two lines with similar names

Since the mid-2010s, OpenFOAM has been developed in two separate lines, both open source under the GPL:

Both share the code up to about 2016 (version 4.x on the .org side, v3.0+ on the .com side) and have diverged since. Each line has added features the other lacks, and both have renamed and restructured things. Case files written for one often do not run unchanged on the other. Some examples of the differences users run into:

So “which OpenFOAM?” always needs two answers: which line, and which version. The solver log answers both after the fact. Its header shows the version (Version: 12 or Version: v2406) and the build. Ask users to keep logs with their results. That makes a result traceable to the exact software that produced it.

Naming modules so nobody confuses them

Put the line into the module name, not only the version:

openfoam-org/11
openfoam-org/12
openfoam-com/v2312
openfoam-com/v2406

With a single openfoam module and versions 12 and v2406 side by side, module load openfoam picks whichever sorts as the default, and users do not notice they switched lines. Separate names make the choice explicit. Set a default per line, and let module avail openfoam show both lines next to each other.

Mark the two lines as conflicting (a family in Lmod, a conflict in Environment Modules), so loading one unloads (or refuses) the other. Both set the same environment variables (WM_PROJECT_DIR, FOAM_APPBIN and many more), and a shell with parts of both is the source of the strangest errors.

Writing the module file

OpenFOAM’s environment comes from sourcing etc/bashrc in the installation. It sets dozens of variables and depends on the compiler and MPI it was built with. Rewriting all of that by hand in a module file is error-prone, and it breaks on the next version. Both common module systems can instead run the shell script and capture the changes it makes.

Lmod (version 8.6 or later), in openfoam-com/v2406.lua:

family("openfoam")         -- only one OpenFOAM module of either line at a time
depends_on("openmpi/4.1")  -- the MPI OpenFOAM was built against
source_sh("bash", "/opt/openfoam/v2406/etc/bashrc")

Environment Modules (version 4.6 or later), in openfoam-org/12:

#%Module
conflict openfoam-org openfoam-com
module load openmpi/4.1
source-sh bash /opt/openfoam/12/etc/bashrc

Both evaluate the script when the module is loaded, record what changed, and undo exactly those changes on unload. Load the MPI module first, the one OpenFOAM was compiled with. OpenFOAM’s bashrc picks up the MPI from the environment when WM_MPLIB is set to the system MPI, and a different MPI at run time than at build time leads to crashes at start-up.

Test the module the way users will use it: in a fresh login shell, load it, run a tutorial in parallel through Slurm, unload it, and check that env | grep -i foam is empty afterwards.

When containers are the better choice

Building every version of both lines from source, for every compiler and MPI combination, is a lot of work for admins. Old versions are worse still, because they no longer build with current compilers. Containers solve this. On HPC systems, that means Apptainer (formerly Singularity), which runs images without root privileges.

Both lines publish container images or recipes. OpenCFD provides images for the .com line on Docker Hub, and CFD Direct documents container use for the .org line. An image can be pulled and converted in one step. For example:

apptainer pull openfoam-v2406.sif docker://opencfd/openfoam-default:2406

Check each project’s documentation for the current image names and tags.

apptainer exec does not start a login shell, so the OpenFOAM environment inside the image must be sourced explicitly. Find where the image installs OpenFOAM (for example with apptainer exec openfoam-v2406.sif ls /usr/lib/openfoam), then:

apptainer exec openfoam-v2406.sif \
    bash -c 'source /usr/lib/openfoam/openfoam2406/etc/bashrc && blockMesh'

MPI across nodes

For parallel runs across nodes, the usual method is the hybrid model: the host’s mpirun (or srun) starts one container process per rank, and the MPI library inside the container talks to the host’s process manager and network:

mpirun -np "${SLURM_NTASKS}" apptainer exec openfoam-v2406.sif \
    bash -c 'source /usr/lib/openfoam/openfoam2406/etc/bashrc && simpleFoam -parallel'

This only works if the MPI inside the image is compatible with the one on the host, typically the same implementation and a close version, and if the image’s MPI can use the cluster’s network. Run a small two-node job and check nProcs in the log and the time per step before declaring a container ready. If the interconnect is not used, the run still works but communicates over plain TCP and scales badly. The alternative is to bind-mount the host’s MPI into the container, which gives full network performance at the cost of a more fragile setup.

A module file can hide all of this. A module openfoam-com/v2406-container can define a shell function or a wrapper script that runs commands through the image, so users write simpleFoam as usual.

A short policy that works