Sizing an OpenFOAM job: cells per core, memory per million cells, and when more cores stop helping

Published

“How many cores should I ask for?” is the most common question new OpenFOAM users ask cluster support, and the most common mistake is asking for too many. Beyond a certain point, extra cores make a run no faster, or even slower, while they still count against the user’s allocation and block the queue for others. This page gives rules of thumb to start from, a way to measure the real answer, and suggestions for admins who want to turn that into limits and guidance.

Cells per core

OpenFOAM runs scale well as long as each process has enough cells to keep busy between communication steps. The useful measure is cells per core:

Cells per core What to expect
more than 200,000 Scales fine, but the run is slower than it needs to be. More cores would help.
50,000 to 200,000 The usual sweet spot on most clusters. Good efficiency, reasonable wall time.
10,000 to 50,000 Can still scale on a fast, low-latency interconnect (InfiniBand, Omni-Path, Slingshot). Efficiency starts to drop.
fewer than 10,000 Communication dominates. More cores rarely help, and often hurt.

These ranges are starting points from common experience, not guarantees. Where your cluster sits in them depends on the interconnect, the memory bandwidth per core, the solver and the mesh. A pressure-based incompressible solver spends much of its time in the pressure equation, whose linear solver communicates a lot, and stops scaling earlier than an explicit or chemistry-heavy solver, which does more work per cell between messages.

For a quick estimate: a 5-million-cell case at 50,000-100,000 cells per core suggests 50-100 cores, so one or two nodes of a typical modern cluster.

Memory per million cells

Memory use depends mostly on the cell count, the solver and the number of fields. Rough planning figures:

Memory per core is rarely the limit for the solver, because the work is spread across all processes. It becomes the limit for serial utilities and meshing. Measure your own numbers rather than relying on these figures. After a job finishes, Slurm reports what it used:

sacct -j <jobid> --format=JobID,Elapsed,NTasks,MaxRSS,MaxRSSNode,State
seff <jobid>     # if your site has it: CPU and memory efficiency in one summary

MaxRSS is the peak memory of the largest single process. Multiply by the number of processes for a rough total, then divide by the cell count in millions.

Measuring a speed-up curve

The rules of thumb tell you where to look. The speed-up curve tells you where to stop. It takes a handful of short jobs:

  1. Pick a representative case, ideally started from a developed solution.
  2. Choose 4-6 core counts, doubling each time, from one node down to a fraction of a node and up to where you expect it to stop helping. For example: 16, 32, 64, 128, 256, 512.
  3. For each count, decompose (with the same method, preferably scotch) and run a fixed number of steps with no writing, as in the benchmark in the decomposition note.
  4. Record the wall time per step, excluding the first step.

Then compute, relative to the smallest count:

speed-up    S(N) = T(N0) / T(N)
efficiency  E(N) = S(N) × N0 / N

A small table makes the point obvious. An illustrative example for a 4-million-cell incompressible case (your numbers will differ):

Cores Cells per core Time per step Speed-up Efficiency
32 125,000 4.0 s 1.0 100%
64 62,500 2.1 s 1.9 95%
128 31,250 1.2 s 3.3 83%
256 15,600 0.85 s 4.7 59%
512 7,800 0.80 s 5.0 31%

Going from 256 to 512 cores in this example costs twice the core-hours for a 6% gain in speed. A common rule is to stop where efficiency drops below about 70%. Here that is between 128 and 256 cores.

It can also pay to run more cases rather than bigger cases: four independent runs on 128 cores each get far more done than one run on 512. For parameter studies, see Slurm job arrays for OpenFOAM.

Things that distort the measurement

Turning this into queue limits and guidance

For admins, the measurements give a defensible basis for policy:

Choosing the core count is also the moment to set it once in the job script, so the decomposition cannot drift from it. See the Slurm job script for OpenFOAM.