Taylor–Green vortex grid convergence: Palabos and OpenFOAM from 64³ to 512³
An update to our post Palabos vs OpenFOAM on cloudHPC: lattice Boltzmann and finite volume on two classic benchmarks.
TL;DR
- The question: a reader pointed out that the DNS reference for the Taylor–Green vortex at Re = 1600 is a 512³ simulation, and asked how our results behave on coarser and finer grids. We ran the full ladder: 64³, 128³, 256³ and 512³.
- Palabos converges cleanly. RMS error of the dissipation rate against the reference: 6.6 % → 2.4 % → 1.0 % → 0.6 %. Going from 256³ to 512³ changes the Palabos curve by only 1.1 % RMS (3.2 % at most): 256³ was already close to converged.
- OpenFOAM is still converging at 256³. Error 20.3 % → 8.1 % → 3.9 %, and the curve still moves 7.0 % RMS between 128³ and 256³. At 64³ it peaks at t = 5.4 instead of 8.9.
- At 64³, plain BGK blows up. Palabos with the standard BGK collision went unstable (energy grows from t ≈ 6.9, overflow at t ≈ 7.9). The regularized BGK model ran the same grid to the end with 6.6 % error.
- Cost: the 512³ Palabos run (134 M nodes) took 47 min of solver time on 96 vCPU for €19.11. The whole update cost €19.40.
Why this update
In the first post we compared Palabos (lattice Boltzmann) and OpenFOAM (finite volume) on two benchmarks. On the 3D Taylor–Green vortex at Re = 1600 we wrote that, on a 256³ grid, “Palabos matches the DNS within 1.0 %, OpenFOAM 3.9 %”.
A reader asked a fair question: the reference data for this case come from spectral DNS at 512³, so what happens on coarser meshes, and does the ranking hold if you refine further? A single grid gives a number, not a trend. A grid-convergence study shows whether that number means anything.
So we added three runs to the two grids we already had:
- 64³ with both codes (the coarse end).
- 512³ with Palabos only.
Everything else is identical to the first post: same physics, same codes, same numerical settings, same cloudHPC hardware.
Results at a glance
| Grid | Nodes / cells | Palabos: RMS error vs reference | Palabos: peak ε (t) | OpenFOAM: RMS error vs reference | OpenFOAM: peak ε (t) |
|---|---|---|---|---|---|
| 64³ | 0.26 M | 6.6 % (regularized BGK) plain BGK: unstable | 0.01208 (8.88) | 20.3 % | 0.00994 (5.44) |
| 128³ | 2.1 M | 2.4 % | 0.01252 (8.54) | 8.1 % | 0.01349 (8.45) |
| 256³ | 16.8 M | 1.0 % | 0.01315 (8.91) | 3.9 % | 0.01179 (8.40) |
| 512³ | 134 M | 0.6 % | 0.01286 (8.95) | not run | — |
| Reference | — | peak ε = 0.01306 at t = 8.94 | |||
RMS error = root-mean-square difference of the dissipation rate ε(t) from the reference over 1 < t < 9.9, divided by the reference peak.
How the runs were set up
- Case: 3D Taylor–Green vortex, periodic cube [0, 2π]³, V₀ = 1, L = 1, Re = V₀L/ν = 1600, simulated to t = 10. Initial field u = sin x cos y cos z, v = −cos x sin y cos z, w = 0, with the matching pressure. No turbulence model in either code: the grid alone resolves (or fails to resolve) the transition to turbulence.
- Metric: kinetic energy Eₖ = ⟨|u|²⟩/2 written every ~0.05 time units, dissipation rate ε = −dEₖ/dt, compared over 1 < t < 9.9.
- Palabos 2.3.0: D3Q19 lattice, BGK collision (regularized BGK at 64³, see below), acoustic scaling with lattice velocity u_LB = 0.05, so the time step halves when the grid spacing halves.
- OpenFOAM v2606 (ESI): pimpleFoam, second-order central (Gauss linear) convection, backward time stepping, fixed Δt for a Courant number of about 0.2, GAMG pressure solver, cyclic patches on all faces.
- Hardware: cloudHPC hypercore instances (physical cores only). The 64³ runs used 8 vCPU on spot instances; the 512³ run used 96 vCPU on a regular instance, so timing is not affected by preemption. The 128³ and 256³ runs are those of the first post.
Which reference curve we used
We compare against the Re = 1600 dissipation curve of Brachet et al. (1983), in the digitised form distributed with the OpenLB tgv3d example (73 points, about one every 0.14 time units). It is the same curve as in the first post.
The reference most groups use today is the 512³ pseudo-spectral DNS of the HiOCFD workshop (case C3.5, van Rees et al. 2011). We did not have that dataset for this update. Two consequences:
- Below about 1 % the comparison is limited by the reference itself (a 73-point curve with interpolation between points), not by the simulation.
- For that reason we also report grid-to-grid differences (self-convergence): how much each curve moves when the grid is doubled. That number needs no reference at all.
Results: the dissipation curves
Palabos. From 256³ up the curves sit on top of each other until t ≈ 7.5 (128³ is within about 3 % of the peak value there). All the grid sensitivity is around the dissipation peak (t ≈ 8–10), where the smallest eddies form. The 64³ run smooths the curve and over-dissipates between t = 4 and t = 7.5, then under-predicts the peak by 7.6 %. The peak time is within 0.4 time units of the reference at every grid (8.54–8.95 against 8.94).
OpenFOAM. The coarse grids dissipate too much, too early. At 64³ the peak comes at t = 5.4, more than three time units early and 24 % too low. At 128³ the peak timing is close (8.45) but it overshoots by 3.3 %, and at 256³ it undershoots by 9.8 %. The trend is right, but the curve is still moving between grids.
Results: error, self-convergence and cost
| Grid step | Palabos: RMS change | Palabos: max change | OpenFOAM: RMS change | OpenFOAM: max change |
|---|---|---|---|---|
| 64³ → 128³ | 6.3 % | 11.7 % | 14.5 % | 31.7 % |
| 128³ → 256³ | 3.0 % | 9.9 % | 7.0 % | 14.9 % |
| 256³ → 512³ | 1.1 % | 3.2 % | not run | — |
Change = difference in ε(t) between two successive grids over 1 < t < 9.9, as a percentage of the finer grid’s peak.
What the numbers say:
- Palabos cuts its error by 2–3× per refinement up to 256³ (6.6 → 2.4 → 1.0 %), then to 0.6 % at 512³. The grid-to-grid change falls from 6.3 to 3.0 to 1.1 %, an observed order between one and two. The last step against the reference flattens, probably because the reference curve itself becomes the limit. The grid-to-grid change (1.1 % RMS) confirms that 256³ is close to converged.
- The 1.0 % at 256³ in the first post holds up. The 512³ run lands at 0.6 % and agrees with the 256³ curve within 1.1 % RMS. Its peak (0.01286) is 1.6 % below the reference, while the 256³ peak was 0.7 % above it. The two finest grids differ by 2.2 % at the peak, which is about the uncertainty of a 73-point reference curve.
- OpenFOAM more than halves its error per refinement too (20.3 → 8.1 → 3.9 %), but from a higher level. With central differencing and no subgrid model, the under-resolved small scales show up as an early, distorted dissipation peak. Its 256³ result (3.9 %) is still behind Palabos at 128³ (2.4 %).
- Accuracy per euro favours Palabos. Palabos at 256³ on 8 vCPU (€0.90, 1.0 %) beats OpenFOAM at 256³ on 32 vCPU (€27.03, 3.9 %). The 64³ runs cost the same for both codes (€0.14–0.15), but the Palabos error is three times smaller.
What didn’t work: plain BGK at 64³
Our first Palabos 64³ run used the same BGK collision as all the other grids. It failed:
- The kinetic energy fell normally until t ≈ 6.9, then started to rise: the scheme was creating energy, which is unphysical in a decaying flow.
- By t ≈ 7.9 the energy had overflowed and the run produced NaN at t = 8.84. The job still finished with status COMPLETED, so the failure only showed in the energy file.
The cause is the relaxation time. With acoustic scaling at u_LB = 0.05, the BGK relaxation time is τ = 0.5 + 3ν_LB, and ν_LB shrinks with the grid spacing:
| Grid | τ (Palabos, u_LB = 0.05) | BGK result |
|---|---|---|
| 64³ | 0.50096 | unstable |
| 128³ | 0.50191 | stable |
| 256³ | 0.50382 | stable |
| 512³ | 0.50764 | stable |
At 64³, τ is less than 0.001 above the stability limit of 0.5, and the grid cannot resolve the small eddies that form near the peak. BGK has almost no numerical damping in that regime. The regularized BGK collision filters the non-hydrodynamic part of the distribution functions. With it, the same 64³ grid ran to t = 10 with a 6.6 % error. This is a known trade-off in LBM: plain BGK is the most accurate collision model when the flow is resolved, and the first to fail when it is not. For coarse or under-resolved runs, use a regularized, MRT or other stabilised collision model, or lower u_LB.
Run times and costs of this update
| Run | Instance | Solve time | Billed cost |
|---|---|---|---|
| Palabos 512³ (BGK) | hypercore 96 vCPU, regular | 2796 s (47 min), 782 MLUPS | €19.11 |
| Palabos 64³ (BGK and regularized BGK, one job) | hypercore 8 vCPU, spot | — | €0.15 |
| OpenFOAM 64³ | hypercore 8 vCPU, spot | 46 s | €0.14 |
| Total for this update | €19.40 |
The 128³ and 256³ runs and their costs are in the first post. Billed costs include boot, end-of-run detection and result upload. For the short 64³ runs these take most of the ~10 billed minutes.
How to run this grid study on cloudHPC
- Download the case files (Palabos
tgv.cppandparam.xmlper grid, the OpenFOAM case generatormake_of_tgv.py). - Change only the grid size:
<N>inparam.xmlfor Palabos, the argument ofmake_of_tgv.pyfor OpenFOAM. Everything else follows from N (time step, output interval). - Upload each grid to its own sub-folder (e.g.
tgv/N64,tgv/N512) and launch: palabos-v2.3.0 and openFoam-v2606 on hypercore instances. - Size the machine to the grid: 64³ fits on 8 vCPU (runs in under a minute); 512³ Palabos needs a 96 vCPU machine for a sub-hour run. Use regular instances for timing, spot instances for everything else.
- Check the energy file, not only the run status: a diverged LBM run can still end as COMPLETED.
- Download your results: cloudHPC storage is cleaned automatically after 60 days.
FAQ
What is a grid convergence study in CFD?
You run the same case on successively finer grids (here each one has twice the resolution of the last) and check that the result stops changing. The difference between the two finest grids estimates the remaining discretisation error, without needing an exact solution.
What grid do I need for the Taylor–Green vortex at Re = 1600?
For about 1 % accuracy on the dissipation rate: 256³ with Palabos (BGK, D3Q19). With OpenFOAM pimpleFoam and central differencing, 256³ gave 3.9 %, still changing by 7 % between grids. 64³ is enough for a quick qualitative check only, and it needs a stabilised collision model in LBM.
Why did Palabos BGK diverge at 64³ but not on finer grids?
With a fixed lattice velocity, the relaxation time τ approaches 0.5 as the grid gets coarser (0.50096 at 64³). BGK has almost no numerical damping there, and the unresolved small eddies near the dissipation peak make it unstable. Regularized BGK ran the same grid to the end.
Is lattice Boltzmann more accurate than finite volume?
On this periodic, wall-free case, Palabos was more accurate on every grid tested (6.6 vs 20.3 % at 64³, 1.0 vs 3.9 % at 256³) and at the same or lower cost per run. On curved walls the picture is different: in the cylinder benchmark of the first post, OpenFOAM matched the reference with fewer cells.
How much does a 512³ lattice Boltzmann simulation cost in the cloud?
On cloudHPC, the 134-million-node Palabos run (16,297 time steps) took 47 minutes of solver time (54 min billed) on 96 hypercore vCPU and cost €19.11.
References
- M. E. Brachet, D. I. Meiron, S. A. Orszag, B. G. Nickel, R. H. Morf, U. Frisch, Small-scale structure of the Taylor–Green vortex, Journal of Fluid Mechanics 130 (1983) 411–452. doi:10.1017/S0022112083001159
- W. M. van Rees, A. Leonard, D. I. Pullin, P. Koumoutsakos, A comparison of vortex and pseudo-spectral methods for the simulation of periodic vortical flows at high Reynolds numbers, Journal of Computational Physics 230 (2011) 2794–2805. doi:10.1016/j.jcp.2010.11.031
- Z. J. Wang et al., High-order CFD methods: current status and perspective, International Journal for Numerical Methods in Fluids 72 (2013) 811–845. doi:10.1002/fld.3767
- HiOCFD workshop, case C3.5, DNS of the Taylor–Green vortex at Re = 1600: cfd.ku.edu/hiocfd/case_c3.5.pdf
- J. Latt et al., Palabos: Parallel Lattice Boltzmann Solver, Computers & Mathematics with Applications 81 (2021) 334–350. doi:10.1016/j.camwa.2020.03.022
- Brachet et al. Re = 1600 dissipation curve as distributed with the OpenLB
tgv3dexample: openlb.net - cloudHPC, Palabos vs OpenFOAM on cloudHPC: lattice Boltzmann and finite volume on two classic benchmarks (the first post): cloudhpc.cloud
CloudHPC is a HPC provider to run engineering simulations on the cloud. CloudHPC provides from 1 to 224 vCPUs for each process in several configuration of HPC infrastructure - both multi-thread and multi-core. Current software ranges includes several CAE, CFD, FEA, FEM software among which OpenFOAM, FDS, Blender and several others.
New users benefit of a FREE trial of 300 vCPU/Hours to be used on the platform in order to test the platform, all each features and verify if it is suitable for their needs
