Introduction

ORB5X is a C++ refactorization of ORB5 (Fortran) which is a global, gyrokinetic, electromagnetic, multi-species code based on a Lagrangian variational description, discretized with a particle representation of the distribution functions and a finite element representation of the fields. It can be considered as a variation of the Particle-In-Cell (PIC) method.

Compilation

In this project, we use

- cmake/3.25.2
- openmpi/4.1.5
- gcc/12.3.0
- cuda/12.8.1

One can build from the repository root with the general workflow

# From root
cmake -B <build directory>  # Takes care of creating the build directory and configuring the project inside of it
cmake --build <build directory>    # Builds the project according to the build tool chosen at configuration time

For example, the cmake command for the CPU build (MPI + serial Kokkos backend) is

cmake -B build \
  -DKokkos_ENABLE_SERIAL=ON
cmake --build build -j 4

As another example, the cmake for compilation on A100 would be

cmake -B build_cuda \
  -DKokkos_ENABLE_CUDA=ON \
  -DKokkos_ARCH_AMPERE80=ON \
  -DKokkos_ENABLE_SERIAL=ON
cmake --build build_cuda -j 4

ORB5X defaults to using installed dependencies available on the system. During configuration, CMake first tries to find:

Kokkos
MPI
HDF5
FFTW3
LAPACK

If a dependency is found but does not satisfy ORB5X requirements, configuration stops with a clear error message explaining what is wrong and which source-build flag can be enabled instead.

The current dependency checks are:

Kokkos: must provide the Kokkos::kokkos target.
HDF5: must provide parallel HDF5 and the C++ bindings.
FFTW3: must provide fftw3.h and fftw_plan_dft_2d.
LAPACK: must provide the symbols used by ORB5X such as dgetrf_, zgetrf_, and zpbtrf_.

If you want CMake to build a dependency from source, enable the corresponding flag:

-DORB5X_BUILD_KOKKOS_FROM_SOURCE=ON
-DORB5X_BUILD_HDF5_FROM_SOURCE=ON
-DORB5X_BUILD_FFTW3_FROM_SOURCE=ON
-DORB5X_BUILD_LAPACK_FROM_SOURCE=ON

You can also override source-build versions with:

-DKokkos_VERSION=...
-DORB5X_HDF5_VERSION=...
-DORB5X_FFTW3_VERSION=...
-DORB5X_OPENBLAS_VERSION=...

When using installed packages, you can help cmake by pointing to non-standard prefixes with variables such as CMAKE_PREFIX_PATH, Kokkos_DIR, HDF5_DIR, or FFTW3_ROOT. For MPI-enabled builds, it is often safest to configure with the MPI wrappers explicitly via -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx.

Here is an example of cmake command that builds Kokkos/FFTW/HDF5/LAPACK as dependencies for H100 GPU:

cmake .. \
    -DKokkos_ENABLE_CUDA=ON \
    -DKokkos_ARCH_HOPPER90=ON \
    -DORB5X_BUILD_KOKKOS_FROM_SOURCE=ON \
    -DORB5X_BUILD_HDF5_FROM_SOURCE=ON \
    -DORB5X_BUILD_FFTW3_FROM_SOURCE=ON \
    -DORB5X_BUILD_LAPACK_FROM_SOURCE=ON

Depending on the Architecture, modify the relevant kokkos flag. For example use -DKokkos_ARCH_BLACKWELL120=ON for Geforce RTX 5070 with Blackwell architecture, and use -DKokkos_ARCH_HOPPER90=ON for H100 with Hopper architecture. For the full list of kokkos flags for a given architecture, see Kokkos GPU architectures configuration guide.

Configurable CMake flags used by this project

Flag	Default	Description
CMAKE_BUILD_TYPE	Release	Build type. Supported values in this project are Debug, Release, RelWithDebInfo, and MinSizeRel.
ENABLE_TESTS	OFF	Enables the unit tests under tests/.
ORB5X_PROFILE_SUMMARY_DEFAULT	OFF	Enables the built-in per-step profile summary by default.
ORB5X_BUILD_KOKKOS_FROM_SOURCE	OFF	Builds Kokkos from source instead of using an installed package.
ORB5X_BUILD_HDF5_FROM_SOURCE	OFF	Builds HDF5 from source instead of using an installed package.
ORB5X_BUILD_FFTW3_FROM_SOURCE	OFF	Builds FFTW3 from source instead of using an installed package.
ORB5X_BUILD_LAPACK_FROM_SOURCE	OFF	Builds LAPACK support from source via OpenBLAS instead of using an installed package.
Kokkos_VERSION	4.7.01	Selects the Kokkos version or git label fetched through FetchContent.
FFTW3_ROOT	auto-detected if possible	Root directory for an existing FFTW3 installation when it is not found automatically.
Kokkos_ENABLE_SERIAL	backend-dependent	Enables the Kokkos Serial backend. Common for CPU builds.
Kokkos_ENABLE_OPENMP	backend-dependent	Enables the Kokkos OpenMP backend. If enabled, ORB5X also tries to link host OpenMP.
Kokkos_ENABLE_CUDA	backend-dependent	Enables the Kokkos CUDA backend for NVIDIA GPU builds.
Kokkos_ARCH_<ARCH>	unset	Selects the target Kokkos architecture, for example Kokkos_ARCH_AMPERE80=ON, Kokkos_ARCH_HOPPER90=ON, or Kokkos_ARCH_BLACKWELL120=ON.

For the setup on supported HPC machines, see Supported_HPC_machines.md.

Documentation

The documentation combines a scientific/user guide with the C++ API reference. The project overview is available at paidynamics.ch/orb5x, and the generated API reference is hosted from this repository at orb5x-docs.pages.dev.

To build it locally, install Doxygen (Graphviz is recommended), configure ORB5X, and build the dedicated target:

cmake -S . -B build -DKokkos_ENABLE_SERIAL=ON

cmake --build build --target docs

Open build/docs/html/index.html. The guide sources live in docs/, the configuration in DOC/Doxygen/, and public API explanations are kept beside declarations in the C++ headers. The ORB5X GitHub repository is private; request access from info@paidynamics.ch and include your name, affiliation, intended use, and GitHub username.

Regression Helpers

For a fresh 2-rank CPU comparison of ORB5X against the Fortran reference on the standard ITG, TAE, and chirping cases, run

bash regression/run_multirank_cpu_regression.sh

This script builds both codes, launches fresh MPI runs with mpirun -np 2, and compares the resulting orb5_res.h5 outputs for:

ITG.in
input_ITPA-TAE.in
input_chirping_AEs.in

Useful options:

ORB5_REG_CASES=ITG bash regression/run_multirank_cpu_regression.sh
ORB5_REG_CASES="TAE chirping" ORB5_REG_NP=2 bash regression/run_multirank_cpu_regression.sh
ORB5_REG_BUILD=0 ORB5_REG_CASES=ITG bash regression/run_multirank_cpu_regression.sh