Sizing an OpenFOAM job: cells per core, memory per million cells, and when more cores stop helping
Published
“How many cores should I ask for?” is the most common question new OpenFOAM users ask cluster support, and the most common mistake is asking for too many. Beyond a certain point, extra cores make a run no faster, or even slower, while they still count against the user’s allocation and block the queue for others. This page gives rules of thumb to start from, a way to measure the real answer, and suggestions for admins who want to turn that into limits and guidance.
Cells per core
OpenFOAM runs scale well as long as each process has enough cells to keep busy between communication steps. The useful measure is cells per core:
| Cells per core | What to expect |
|---|---|
| more than 200,000 | Scales fine, but the run is slower than it needs to be. More cores would help. |
| 50,000 to 200,000 | The usual sweet spot on most clusters. Good efficiency, reasonable wall time. |
| 10,000 to 50,000 | Can still scale on a fast, low-latency interconnect (InfiniBand, Omni-Path, Slingshot). Efficiency starts to drop. |
| fewer than 10,000 | Communication dominates. More cores rarely help, and often hurt. |
These ranges are starting points from common experience, not guarantees. Where your cluster sits in them depends on the interconnect, the memory bandwidth per core, the solver and the mesh. A pressure-based incompressible solver spends much of its time in the pressure equation, whose linear solver communicates a lot, and stops scaling earlier than an explicit or chemistry-heavy solver, which does more work per cell between messages.
For a quick estimate: a 5-million-cell case at 50,000-100,000 cells per core suggests 50-100 cores, so one or two nodes of a typical modern cluster.
Memory per million cells
Memory use depends mostly on the cell count, the solver and the number of fields. Rough planning figures:
- Incompressible steady or transient solvers (
simpleFoam,pimpleFoamwith a two-equation turbulence model): around 1-2 GB per million cells, summed over all processes. - Compressible, multiphase or reacting solvers carry more fields and need more: plan for 2-4 GB per million cells, and more for detailed chemistry with many species.
- Meshing with
snappyHexMeshcan use several times what the solver later needs for the same mesh, because it keeps intermediate meshes. It is the step that most often runs out of memory. decomposeParandreconstructParrun on one process and hold the whole mesh, plus the fields of one time step. For large meshes, that one process needs a node with enough memory on its own.
Memory per core is rarely the limit for the solver, because the work is spread across all processes. It becomes the limit for serial utilities and meshing. Measure your own numbers rather than relying on these figures. After a job finishes, Slurm reports what it used:
sacct -j <jobid> --format=JobID,Elapsed,NTasks,MaxRSS,MaxRSSNode,State
seff <jobid> # if your site has it: CPU and memory efficiency in one summary
MaxRSS is the peak memory of the largest single process. Multiply by the number of processes for
a rough total, then divide by the cell count in millions.
Measuring a speed-up curve
The rules of thumb tell you where to look. The speed-up curve tells you where to stop. It takes a handful of short jobs:
- Pick a representative case, ideally started from a developed solution.
- Choose 4-6 core counts, doubling each time, from one node down to a fraction of a node and up to where you expect it to stop helping. For example: 16, 32, 64, 128, 256, 512.
- For each count, decompose (with the same method, preferably
scotch) and run a fixed number of steps with no writing, as in the benchmark in the decomposition note. - Record the wall time per step, excluding the first step.
Then compute, relative to the smallest count:
speed-up S(N) = T(N0) / T(N)
efficiency E(N) = S(N) × N0 / N
A small table makes the point obvious. An illustrative example for a 4-million-cell incompressible case (your numbers will differ):
| Cores | Cells per core | Time per step | Speed-up | Efficiency |
|---|---|---|---|---|
| 32 | 125,000 | 4.0 s | 1.0 | 100% |
| 64 | 62,500 | 2.1 s | 1.9 | 95% |
| 128 | 31,250 | 1.2 s | 3.3 | 83% |
| 256 | 15,600 | 0.85 s | 4.7 | 59% |
| 512 | 7,800 | 0.80 s | 5.0 | 31% |
Going from 256 to 512 cores in this example costs twice the core-hours for a 6% gain in speed. A common rule is to stop where efficiency drops below about 70%. Here that is between 128 and 256 cores.
It can also pay to run more cases rather than bigger cases: four independent runs on 128 cores each get far more done than one run on 512. For parameter studies, see Slurm job arrays for OpenFOAM.
Things that distort the measurement
- Partly filled nodes. Eight processes on a 64-core node get far more memory bandwidth each than 64 processes do. Small counts then look unrealistically fast, and the curve looks worse than it is. Compare whole nodes where you can, or fill nodes the same way for every count.
- Writing. A run that writes during the measurement measures the file system. Set
writeIntervalbeyond the end of the test. For production runs, the collated file format reduces I/O cost at high core counts. - Start-up. Reading the mesh and fields, and the first iterations, are not representative. Leave them out.
- Other jobs. Shared interconnects and file systems vary over the day. Run each point twice.
Turning this into queue limits and guidance
For admins, the measurements give a defensible basis for policy:
-
Publish a table of measured cells per core for the common solvers on your hardware. Users can then pick a core count from their cell count without running their own study.
-
Add a check to the job template. The mesh size is in
constant/polyMesh/owner, whose headernotecontainsnCells. A job script can read it and warn when the request drops below your threshold:ncells=$(grep -m1 -o 'nCells: *[0-9]*' constant/polyMesh/owner | grep -o '[0-9]*$') per_core=$(( ncells / SLURM_NTASKS )) if (( per_core < 20000 )); then echo "Warning: only ${per_core} cells per core; fewer cores would likely be as fast." >&2 fiThe header stays readable in binary format. For a compressed mesh (
owner.gz), usezgrep, or read the count fromcheckMesh, at the cost of loading the whole mesh. -
Set partition limits with the speed-up curve in mind. A partition for CFD that allows 2,000 cores per job only makes sense if users have cases big enough to use them.
-
Use
sefforsacctreports to find jobs with low CPU efficiency, and talk to those users. An OpenFOAM job at 30% efficiency on 512 cores frees a lot of capacity once it moves to 128.
Choosing the core count is also the moment to set it once in the job script, so the decomposition cannot drift from it. See the Slurm job script for OpenFOAM.