ImpactX

ImpactX is an s-based beam dynamics code including space charge effects. This is the next generation of the IMPACT-Z code.

Please contact us with any questions on it or if you like to contribute to its development.

Contact us

If you are starting using ImpactX, or if you have a user question, please pop in our discussions page and get in touch with the community.

The ImpactX GitHub repo is the main communication platform. Have a look at the action icons on the top right of the web page: feel free to watch the repo if you want to receive updates, or to star the repo to support the project. For bug reports or to request new features, you can also open a new issue.

We also have a discussion page on which you can find already answered questions, add new questions, get help with installation procedures, discuss ideas or share comments.

Code of Conduct

Our Pledge

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

Our Standards

Examples of behavior that contributes to creating a positive environment include:

Using welcoming and inclusive language
Being respectful of differing viewpoints and experiences
Gracefully accepting constructive criticism
Focusing on what is best for the community
Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

The use of sexualized language or imagery and unwelcome sexual attention or advances
Trolling, insulting/derogatory comments, and personal or political attacks
Public or private harassment
Publishing others’ private information, such as a physical or electronic address, without explicit permission
Other conduct which could reasonably be considered inappropriate in a professional setting

Our Responsibilities

Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.

Scope

This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.

Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at warpx-coc@lbl.gov. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project’s leadership.

Attribution

This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

For answers to common questions about this code of conduct, see https://www.contributor-covenant.org/faq

Acknowledge ImpactX

Please acknowledge the role that ImpactX played in your research.

In presentations

Note

TODO :-)

In publications

Please add the following sentence to your publications, it helps contributors keep in touch with the community and promote the project.

Plain text:

This research used the open-source particle-in-cell code ImpactX https://github.com/ECP-WarpX/impactx. We acknowledge all ImpactX contributors.

Latex:

\usepackage{hyperref}
This research used the open-source particle-in-cell code ImpactX \url{https://github.com/ECP-WarpX/impactx}.
We acknowledge all ImpactX contributors.

Main ImpactX Reference

If your project leads to a scientific publication, please consider citing the paper below.

Huebl A, Lehe R, Mitchell C E, Qiang J, Ryne R D, Sandberg R T, Vay JL. Next Generation Computational Tools for the Modeling and Design of Particle Accelerators at Exascale. 2022 North American Particle Accelerator Conference (NAPAC’22), TUYE2, pp. 302-306, 2022. arXiv:2208.02382, DOI:10.18429/JACoW-NAPAC2022-TUYE2

Further ImpactX References

Sandberg R T, Lehe R, Mitchell C E, Garten M, Myers A, Qiang J, Vay J-L and Huebl A. Synthesizing Particle-in-Cell Simulations Through Learning and GPU Computing for Hybrid Particle Accelerator Beamlines. Proc. of Platform for Advanced Scientific Computing (PASC’24), submitted, 2024. arXiv:2402.17248
Sandberg R T, Lehe R, Mitchell C E, Garten M, Qiang J, Vay J-L and Huebl A. Hybrid Beamline Element ML-Training for Surrogates in the ImpactX Beam-Dynamics Code. 14th International Particle Accelerator Conference (IPAC’23), WEPA101, 2023. DOI:10.18429/JACoW-IPAC2023-WEPA101
Huebl A, Lehe R, Zoni E, Shapoval O, Sandberg R T, Garten M, Formenti A, Jambunathan R, Kumar P, Gott K, Myers A, Zhang W, Almgren A, Mitchell C E, Qiang J, Sinn A, Diederichs S, Thevenet M, Grote D, Fedeli L, Clark T, Zaim N, Vincenti H, Vay JL. From Compact Plasma Particle Sources to Advanced Accelerators with Modeling at Exascale. Proceedings of the 20th Advanced Accelerator Concepts Workshop (AAC’22), in print, 2023. arXiv:2303.12873

Installation

Users

Our community is here to help. Please report installation problems in case you should get stuck.

Choose one of the installation methods below to get started:

HPC Systems

If want to use ImpactX on a specific high-performance computing (HPC) systems, jump directly to our HPC system-specific documentation.

Using the Conda Package

A package for ImpactX is available via the Conda package manager.

Tip

We recommend to configure your conda to use the faster libmamba dependency solver.

conda update -y -n base conda
conda install -y -n base conda-libmamba-solver
conda config --set solver libmamba

We recommend to deactivate that conda self-activates its base environment. This avoids interference with the system and other package managers.

conda config --set auto_activate_base false

conda create -n impactx -c conda-forge impactx
conda activate impactx

Note

The impactx conda package does not yet provide GPU support.

Using the Spack Package

Note

Coming soon.

Using the PyPI Package

Note

Coming soon.

Using the Brew Package

Note

Coming soon.

From Source with CMake

After installing the ImpactX dependencies, you can also install ImpactX from source with CMake:

# get the source code
git clone https://github.com/ECP-WarpX/impactx.git $HOME/src/impactx
cd $HOME/src/impactx

# configure
cmake -S . -B build

# optional: change configuration
ccmake build

# compile
#   on Windows:          --config Release
cmake --build build -j 4

# executables for ImpactX are now in build/bin/

We document the details in the developer installation.

Tips for macOS Users

Tip

Before getting started with package managers, please check what you manually installed in /usr/local. If you find entries in bin/, lib/ et al. that look like you manually installed MPI, HDF5 or other software in the past, then remove those files first.

If you find software such as MPI in the same directories that are shown as symbolic links then it is likely you brew installed software before. If you are trying annother package manager than brew, run brew unlink … on such packages first to avoid software incompatibilities.

Developers

CMake is our primary build system. If you are new to CMake, this short tutorial from the HEP Software foundation is the perfect place to get started. If you just want to use CMake to build the project, jump into sections 1. Introduction, 2. Building with CMake and 9. Finding Packages.

Dependencies

Before you start, you will need a copy of the ImpactX source code:

git clone https://github.com/ECP-WarpX/impactx.git $HOME/src/impactx
cd $HOME/src/impact

ImpactX depends on popular third party software.

On your development machine, follow the instructions here.
If you are on an HPC machine, follow the instructions here.

Dependencies

ImpactX depends on the following popular third party software. Please see installation instructions below.

a mature C++17 compiler, e.g., GCC 8.4+, Clang 7, NVCC 11.0, MSVC 19.15 or newer
CMake 3.20.0+
Git 2.18+
AMReX: we automatically download and compile a copy
ABLASTR/WarpX: we automatically download and compile a copy

Optional dependencies include:

MPI 3.0+: for multi-node and/or multi-GPU execution
for on-node accelerated compute one of either:
- OpenMP 3.1+: for threaded CPU execution or
- CUDA Toolkit 11.0+ (11.3+ recommended): for Nvidia GPU support (see matching host-compilers) or
- ROCm 5.2+ (5.5+ recommended): for AMD GPU support
openPMD-api 0.15.1+: we automatically download and compile a copy of openPMD-api for openPMD I/O support
- see optional I/O backends
CCache: to speed up rebuilds (For CUDA support, needs version 3.7.9+ and 4.2+ is recommended)
Ninja: for faster parallel compiles
Python 3.8+
- mpi4py
- numpy
- quantiphy
- openPMD-api
- see our requirements.txt file for compatible versions

If you are on a high-performance computing (HPC) system, then please see our separate HPC documentation.

For all other systems, we recommend to use a package dependency manager: Pick one of the installation methods below to install all dependencies for ImpactX development in a consistent manner.

Conda (Linux/macOS/Windows)

Conda/Mamba are cross-compatible, user-level package managers.

Tip

We recommend to configure your conda to use the faster libmamba dependency solver.

conda update -y -n base conda
conda install -y -n base conda-libmamba-solver
conda config --set solver libmamba

We recommend to deactivate that conda self-activates its base environment. This avoids interference with the system and other package managers.

conda config --set auto_activate_base false

With MPI (only Linux/macOS)

conda create -n impactx-cpu-mpich-dev -c conda-forge blaspp boost ccache cmake compilers git lapackpp "openpmd-api=*=mpi_mpich*" python numpy pandas quantiphy scipy yt pkg-config matplotlib mamba ninja mpich pip virtualenv
conda activate impactx-cpu-mpich-dev

# compile ImpactX with -DImpactX_MPI=ON
# for pip, use: export IMPACTX_MPI=ON

Without MPI

conda create -n impactx-cpu-dev -c conda-forge blaspp boost ccache cmake compilers git lapackpp openpmd-api python numpy pandas quantiphy scipy yt pkg-config matplotlib mamba ninja pip virtualenv
conda activate impactx-cpu-dev

# compile ImpactX with -DImpactX_MPI=OFF
# for pip, use: export IMPACTX_MPI=OFF

For OpenMP support, you will further need:

Linux

conda install -c conda-forge libgomp

macOS or Windows

conda install -c conda-forge llvm-openmp

Spack (Linux/macOS)

Spack is a user-level package manager. It is primarily written for Linux, with slightly less support for macOS, and future support for Windows.

First, download a WarpX Spack desktop development environment of your choice (which will also work for ImpactX). For most desktop developments, pick the OpenMP environment for CPUs unless you have a supported GPU.

Debian/Ubuntu Linux:
- OpenMP: system=ubuntu; compute=openmp (CPUs)
- CUDA: system=ubuntu; compute=cuda (Nvidia GPUs)
- ROCm: system=ubuntu; compute=rocm (AMD GPUs)
- SYCL: todo (Intel GPUs)
macOS: first, prepare with brew install gpg2; brew install gcc
- OpenMP: system=macos; compute=openmp

If you already installed Spack, we recommend to activate its binary caches for faster builds:

spack mirror add rolling https://binaries.spack.io/develop
spack buildcache keys --install --trust

Now install the WarpX/ImpactX dependencies in a new development environment:

# download environment file
curl -sLO https://raw.githubusercontent.com/ECP-WarpX/WarpX/development/Tools/machines/desktop/spack-${system}-${compute}.yaml

# create new development environment
spack env create impactx-${compute}-dev spack-${system}-${compute}.yaml
spack env activate impactx-${compute}-dev

# installation
spack install
python3 -m pip install jupyter matplotlib numpy openpmd-api openpmd-viewer pandas quantiphy scipy virtualenv yt

In new terminal sessions, re-activate the environment with

spack env activate impactx-openmp-dev

again. Replace openmp with the equivalent you chose.

Compile ImpactX with -DImpactX_MPI=ON. For pip, use export IMPACTX_MPI=ON.

Brew (macOS/Linux)

Homebrew (Brew) is a user-level package manager primarily for Apple macOS, but also supports Linux.

brew update
brew tap openpmd/openpmd
brew install adios2      # for openPMD
brew install ccache
brew install cmake
brew install git
brew install hdf5-mpi    # for openPMD
brew install libomp
brew unlink gcc
brew link --force libomp
brew install open-mpi
brew install openblas    # for PSATD in RZ
brew install openpmd-api # for openPMD

If you also want to compile with PSATD in RZ, you need to manually install BLAS++ and LAPACK++:

sudo mkdir -p /usr/local/bin/
sudo curl -L -o /usr/local/bin/cmake-easyinstall https://raw.githubusercontent.com/ax3l/cmake-easyinstall/main/cmake-easyinstall
sudo chmod a+x /usr/local/bin/cmake-easyinstall

cmake-easyinstall --prefix=/usr/local git+https://github.com/icl-utk-edu/blaspp.git \
    -Duse_openmp=OFF -Dbuild_tests=OFF -DCMAKE_VERBOSE_MAKEFILE=ON
cmake-easyinstall --prefix=/usr/local git+https://github.com/icl-utk-edu/lapackpp.git \
    -Duse_cmake_find_lapack=ON -Dbuild_tests=OFF -DCMAKE_VERBOSE_MAKEFILE=ON

Compile ImpactX with -DImpactX_MPI=ON. For pip, use export IMPACTX_MPI=ON.

APT (Debian/Ubuntu Linux)

The Advanced Package Tool (APT) is a system-level package manager on Debian-based Linux distributions, including Ubuntu.

With MPI (only Linux/macOS)

sudo apt update
sudo apt install build-essential ccache cmake g++ git libhdf5-openmpi-dev libopenmpi-dev pkg-config python3 python3-matplotlib python3-numpy python3-pandas python3-pip python3-scipy python3-venv

# optional:
# for CUDA, either install
#   https://developer.nvidia.com/cuda-downloads (preferred)
# or, if your Debian/Ubuntu is new enough, use the packages
#   sudo apt install nvidia-cuda-dev libcub-dev

# compile ImpactX with -DImpactX_MPI=ON
# for pip, use: export IMPACTX_MPI=ON

Without MPI

sudo apt update
sudo apt install build-essential ccache cmake g++ git libhdf5-dev pkg-config python3 python3-matplotlib python3-numpy python3-pandas python3-pip python3-scipy python3-venv

# optional:
# for CUDA, either install
#   https://developer.nvidia.com/cuda-downloads (preferred)
# or, if your Debian/Ubuntu is new enough, use the packages
#   sudo apt install nvidia-cuda-dev libcub-dev

# compile ImpactX with -DImpactX_MPI=OFF
# for pip, use: export IMPACTX_MPI=OFF

Note

Preparation: make sure you work with up-to-date Python tooling.

python3 -m pip install -U pip
python3 -m pip install -U build packaging setuptools wheel pytest
python3 -m pip install -U -r examples/requirements.txt

Compile

From the base of the ImpactX source directory, execute:

# find dependencies & configure
#   see additional options below, e.g.
#                   -DCMAKE_INSTALL_PREFIX=$HOME/sw/impactX
cmake -S . -B build -DImpactX_PYTHON=ON

# compile, here we use four threads
cmake --build build -j 4

That’s all! ImpactX binaries are now in build/bin/. Most people execute these binaries directly or copy them out.

If you want to install the executables in a programmatic way, run this:

# for default install paths, you will need administrator rights, e.g. with sudo:
#   this installs the application
cmake --build build --target install

#   this installs the Python bindings via "python3 -m pip install ..."
cmake --build build --target pip_install -j 4

You can inspect and modify build options after running cmake -S . -B build with either

ccmake build

or by adding arguments with -D<OPTION>=<VALUE> to the first CMake call, e.g.:

cmake -S . -B build -DImpactX_PYTHON=ON -DImpactX_COMPUTE=CUDA

That’s it! You can now run a first example.

Developers could now change the ImpactX source code and then call the install lines again to refresh the installation.

Tip

If you do not develop with a user-level package manager, e.g., because you rely on a HPC system’s environment modules, then consider to set up a virtual environment via Python venv. Otherwise, without a virtual environment, you likely need to add the CMake option -DPYINSTALLOPTIONS="--user".

Build Options

CMake Option	Default & Values	Description
`BUILD_TESTING`	ON/OFF	Build tests
`CMAKE_BUILD_TYPE`	RelWithDebInfo/Release/Debug	Type of build, symbols & optimizations
`CMAKE_INSTALL_PREFIX`	system-dependent path	Install path prefix
`CMAKE_VERBOSE_MAKEFILE`	ON/OFF	Print all compiler commands to the terminal during build
`ImpactX_APP`	ON/OFF	Build the ImpactX executable application
`ImpactX_COMPUTE`	NOACC/OMP/CUDA/SYCL/HIP	On-node, accelerated computing backend
`ImpactX_IPO`	ON/OFF	Compile ImpactX with interprocedural optimization (aka LTO)
`ImpactX_MPI`	ON/OFF	Multi-node support (message-passing)
`ImpactX_MPI_THREAD_MULTIPLE`	ON/OFF	MPI thread-multiple support, i.e. for `async_io`
`ImpactX_OPENPMD`	ON/OFF	openPMD I/O (HDF5, ADIOS)
`ImpactX_PRECISION`	SINGLE/DOUBLE	Floating point precision (single/double)
`ImpactX_PYTHON`	ON/OFF	Python bindings
`Python_EXECUTABLE`	(newest found)	Path to Python executable

ImpactX can be configured in further detail with options from AMReX, which are documented in the AMReX manual.

Developers might be interested in additional options that control dependencies of ImpactX. By default, the most important dependencies of ImpactX are automatically downloaded for convenience:

CMake Option	Default & Values	Description
`BUILD_SHARED_LIBS`	ON/OFF	Build shared libraries for dependencies
`ImpactX_CCACHE`	ON/OFF	Search and use CCache to speed up rebuilds.
`ImpactX_ablastr_src`	None	Path to ABLASTR source directory (preferred if set)
`ImpactX_ablastr_repo`	`https://github.com/ECP-WarpX/WarpX.git`	Repository URI to pull and build ABLASTR from
`ImpactX_ablastr_branch`	we set and maintain a compatible commit	Repository branch for `ImpactX_ablastr_repo`
`ImpactX_ablastr_internal`	ON/OFF	Needs a pre-installed ABLASTR library if set to `OFF`
`ImpactX_amrex_src`	None	Path to AMReX source directory (preferred if set)
`ImpactX_amrex_repo`	`https://github.com/AMReX-Codes/amrex.git`	Repository URI to pull and build AMReX from
`ImpactX_amrex_branch`	we set and maintain a compatible commit	Repository branch for `ImpactX_amrex_repo`
`ImpactX_amrex_internal`	ON/OFF	Needs a pre-installed AMReX library if set to `OFF`
`ImpactX_openpmd_src`	None	Path to openPMD-api source directory (preferred if set)
`ImpactX_openpmd_repo`	`https://github.com/openPMD/openPMD-api.git`	Repository URI to pull and build openPMD-api from
`ImpactX_openpmd_branch`	we set and maintain a compatible commit	Repository branch for `ImpactX_openpmd_repo`
`ImpactX_openpmd_internal`	ON/OFF	Needs a pre-installed openPMD-api library if set to `OFF`
`ImpactX_pyamrex_src`	None	Path to AMReX source directory (preferred if set)
`ImpactX_pyamrex_repo`	`https://github.com/AMReX-Codes/pyamrex.git`	Repository URI to pull and build pyAMReX from
`ImpactX_pyamrex_branch`	we set and maintain a compatible commit	Repository branch for `ImpactX_pyamrex_repo`
`ImpactX_pyamrex_internal`	ON/OFF	Needs a pre-installed pyAMReX module if set to `OFF`
`ImpactX_PYTHON_IPO`	ON/OFF	Build Python w/ interprocedural/link optimization (IPO/LTO)
`ImpactX_pybind11_src`	None	Path to pybind11 source directory (preferred if set)
`ImpactX_pybind11_repo`	`https://github.com/pybind/pybind11.git`	Repository URI to pull and build pybind11 from
`ImpactX_pybind11_branch`	we set and maintain a compatible commit	Repository branch for `ImpactX_pybind11_repo`
`ImpactX_pybind11_internal`	ON/OFF	Needs a pre-installed pybind11 library if set to `OFF`

For example, one can also build against a local AMReX copy. Assuming AMReX’ source is located in $HOME/src/amrex, add the cmake argument -DImpactX_amrex_src=$HOME/src/amrex. Relative paths are also supported, e.g. -DImpactX_amrex_src=../amrex.

Or build against an AMReX feature branch of a colleague. Assuming your colleague pushed AMReX to https://github.com/WeiqunZhang/amrex/ in a branch new-feature then pass to cmake the arguments: -DImpactX_amrex_repo=https://github.com/WeiqunZhang/amrex.git -DImpactX_amrex_branch=new-feature.

If you want to develop against local versions of ABLASTR (from WarpX) and AMReX at the same time, pass for instance -DImpactX_ablastr_src=$HOME/src/warpx -DImpactX_amrex_src=$HOME/src/amrex.

You can speed up the install further if you pre-install these dependencies, e.g. with a package manager. Set -DImpactX_<dependency-name>_internal=OFF and add installation prefix of the dependency to the environment variable CMAKE_PREFIX_PATH. Please see the introduction to CMake if this sounds new to you.

If you re-compile often, consider installing the Ninja build system. Pass -G Ninja to the CMake configuration call to speed up parallel compiles.

Configure Your Compiler

If you don’t want to use your default compiler, you can set the following environment variables. For example, using a Clang/LLVM:

export CC=$(which clang)
export CXX=$(which clang++)

If you also want to select a CUDA compiler:

export CUDACXX=$(which nvcc)
export CUDAHOSTCXX=$(which clang++)

Note

Please clean your build directory with rm -rf build/ after changing the compiler. Now call cmake -S . -B build (+ further options) again to re-initialize the build configuration.

Run

The ImpactX Python bindings, which provide the imports impactx and amrex (from pyAMReX), are automatically packaged and installed when calling the pip_install CMake target.

An executable ImpactX application binary with the current compile-time options encoded in its file name will be created in build/bin/. Additionally, a symbolic link named impactx can be found in that directory, which points to the last built ImpactX executable.

HPC

On selected high-performance computing (HPC) systems, ImpactX has documented or even pre-build installation routines. Follow the guide here instead of the generic installation routines for optimal stability and best performance.

impactx.profile

Use a impactx.profile file to set up your software environment without colliding with other software. Ideally, store that file directly in your $HOME/ and source it after connecting to the machine:

source $HOME/impactx.profile

We list example impactx.profile files below, which can be used to set up ImpactX on various HPC systems.

HPC Systems

Perlmutter (NERSC)

The Perlmutter cluster is located at NERSC.

Introduction

If you are new to this system, please see the following resources:

NERSC user guide
Batch system: Slurm
Jupyter service (documentation)
Filesystems:
- $HOME: per-user directory, use only for inputs, source and scripts; backed up (40GB)
- ${CFS}/m3239/: community file system for users in the project m3239 (or equivalent); moderate performance (20TB default)
- $PSCRATCH: per-user production directory; very fast for parallel jobs; purged every 8 weeks (20TB default)

Preparation

Use the following commands to download the ImpactX source code:

git clone https://github.com/ECP-WarpX/impactx.git $HOME/src/impactx

On Perlmutter, you can run either on GPU nodes with fast A100 GPUs (recommended) or CPU nodes.

A100 GPUs

We use system software modules, add environment hints and further dependencies via the file $HOME/perlmutter_gpu_impactx.profile. Create it now:

cp $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu_impactx.profile.example $HOME/perlmutter_gpu_impactx.profile

Script Details

# please set your project account
export proj=""  # change me! GPU projects must end in "..._g"

# remembers the location of this script
export MY_PROFILE=$(cd $(dirname $BASH_SOURCE) && pwd)"/"$(basename $BASH_SOURCE)
if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your $MY_PROFILE file! Please edit its line 2 to continue!"; return; fi

# required dependencies
module load cmake/3.24.3

# optional: for QED support with detailed tables
export BOOST_ROOT=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/boost-1.82.0-ow5r5qrgslcwu33grygouajmuluzuzv3

# optional: for openPMD and PSATD+RZ support
module load cray-hdf5-parallel/1.12.2.9
export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/adios2-2.8.3:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/blaspp-master:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/lapackpp-master:$CMAKE_PREFIX_PATH

export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/adios2-2.8.3/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/blaspp-master/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/lapackpp-master/lib64:$LD_LIBRARY_PATH

export PATH=${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/adios2-2.8.3/bin:${PATH}

# optional: CCache
export PATH=/global/common/software/spackecp/perlmutter/e4s-23.08/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/ccache-4.8.2-cvooxdw5wgvv2g3vjxjkrpv6dopginv6/bin:$PATH

# optional: for Python bindings or libEnsemble
module load cray-python/3.11.5

if [ -d "${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/venvs/impactx" ]
then
  source ${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/venvs/impactx/bin/activate
fi

# an alias to request an interactive batch node for one hour
#   for parallel execution, start on the batch node: srun <command>
alias getNode="salloc -N 1 --ntasks-per-node=4 -t 1:00:00 -q interactive -C gpu --gpu-bind=single:1 -c 32 -G 4 -A $proj"
# an alias to run a command on a batch node for up to 30min
#   usage: runNode <command>
alias runNode="srun -N 1 --ntasks-per-node=4 -t 0:30:00 -q interactive -C gpu --gpu-bind=single:1 -c 32 -G 4 -A $proj"

# necessary to use CUDA-Aware MPI and run a job
export CRAY_ACCEL_TARGET=nvidia80

# optimize CUDA compilation for A100
export AMREX_CUDA_ARCH=8.0

# optimize CPU microarchitecture for AMD EPYC 3rd Gen (Milan/Zen3)
# note: the cc/CC/ftn wrappers below add those
export CXXFLAGS="-march=znver3"
export CFLAGS="-march=znver3"

# compiler environment hints
export CC=cc
export CXX=CC
export FC=ftn
export CUDACXX=$(which nvcc)
export CUDAHOSTCXX=CC

Edit the 2nd line of this script, which sets the export proj="" variable. Perlmutter GPU projects must end in ..._g. For example, if you are member of the project m3239, then run nano $HOME/perlmutter_gpu_impactx.profile and edit line 2 to read:

export proj="m3239_g"

Exit the nano editor with Ctrl + O (save) and then Ctrl + X (exit).

Important

Now, and as the first step on future logins to Perlmutter, activate these environment settings:

source $HOME/perlmutter_gpu_impactx.profile

Finally, since Perlmutter does not yet provide software modules for some of our dependencies, install them once:

bash $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/install_gpu_dependencies.sh
source ${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu/venvs/impactx/bin/activate

Script Details

#!/bin/bash
#
# Copyright 2023 The ImpactX Community
#
# This file is part of ImpactX.
#
# Author: Axel Huebl
# License: BSD-3-Clause-LBNL

# Exit on first error encountered #############################################
#
set -eu -o pipefail


# Check: ######################################################################
#
#   Was perlmutter_gpu_impactx.profile sourced and configured correctly?
if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your perlmutter_gpu_impactx.profile file! Please edit its line 2 to continue!"; exit 1; fi


# Check $proj variable is correct and has a corresponding CFS directory #######
#
if [ ! -d "${CFS}/${proj%_g}/" ]
then
    echo "WARNING: The directory ${CFS}/${proj%_g}/ does not exist!"
    echo "Is the \$proj environment variable of value \"$proj\" correctly set? "
    echo "Please edit line 2 of your perlmutter_gpu_impactx.profile file to continue!"
    exit
fi


# Remove old dependencies #####################################################
#
SW_DIR="${CFS}/${proj%_g}/${USER}/sw/perlmutter/gpu"
rm -rf ${SW_DIR}
mkdir -p ${SW_DIR}

# remove common user mistakes in python, located in .local instead of a venv
python3 -m pip uninstall -qq -y impactx
python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true


# General extra dependencies ##################################################
#

# tmpfs build directory: avoids issues often seen with $HOME and is faster
build_dir=$(mktemp -d)

# c-blosc (I/O compression)
if [ -d $HOME/src/c-blosc ]
then
  cd $HOME/src/c-blosc
  git fetch --prune
  git checkout v1.21.1
  cd -
else
  git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git $HOME/src/c-blosc
fi
cmake -S $HOME/src/c-blosc -B ${build_dir}/c-blosc-pm-gpu-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1
cmake --build ${build_dir}/c-blosc-pm-gpu-build --target install --parallel 16
rm -rf ${build_dir}/c-blosc-pm-gpu-build

# ADIOS2
if [ -d $HOME/src/adios2 ]
then
  cd $HOME/src/adios2
  git fetch --prune
  git checkout v2.8.3
  cd -
else
  git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git $HOME/src/adios2
fi
cmake -S $HOME/src/adios2 -B ${build_dir}/adios2-pm-gpu-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3
cmake --build ${build_dir}/adios2-pm-gpu-build --target install -j 16
rm -rf ${build_dir}/adios2-pm-gpu-build

# BLAS++ (for PSATD+RZ)
if [ -d $HOME/src/blaspp ]
then
  cd $HOME/src/blaspp
  git fetch --prune
  git checkout master
  git pull
  cd -
else
  git clone https://github.com/icl-utk-edu/blaspp.git $HOME/src/blaspp
fi
CXX=$(which CC) cmake -S $HOME/src/blaspp -B ${build_dir}/blaspp-pm-gpu-build -Duse_openmp=OFF -Dgpu_backend=cuda -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master
cmake --build ${build_dir}/blaspp-pm-gpu-build --target install --parallel 16
rm -rf ${build_dir}/blaspp-pm-gpu-build

# LAPACK++ (for PSATD+RZ)
if [ -d $HOME/src/lapackpp ]
then
  cd $HOME/src/lapackpp
  git fetch --prune
  git checkout master
  git pull
  cd -
else
  git clone https://github.com/icl-utk-edu/lapackpp.git $HOME/src/lapackpp
fi
CXX=$(which CC) CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S $HOME/src/lapackpp -B ${build_dir}/lapackpp-pm-gpu-build -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master
cmake --build ${build_dir}/lapackpp-pm-gpu-build --target install --parallel 16
rm -rf ${build_dir}/lapackpp-pm-gpu-build


# Python ######################################################################
#
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade virtualenv
python3 -m pip cache purge
rm -rf ${SW_DIR}/venvs/impactx
python3 -m venv ${SW_DIR}/venvs/impactx
source ${SW_DIR}/venvs/impactx/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade build
python3 -m pip install --upgrade packaging
python3 -m pip install --upgrade wheel
python3 -m pip install --upgrade setuptools
python3 -m pip install --upgrade cython
python3 -m pip install --upgrade numpy
python3 -m pip install --upgrade pandas
python3 -m pip install --upgrade scipy
MPICC="cc -target-accel=nvidia80 -shared" python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py
python3 -m pip install --upgrade openpmd-api
python3 -m pip install --upgrade matplotlib
python3 -m pip install --upgrade yt
# install or update impactx dependencies
python3 -m pip install --upgrade -r $HOME/src/impactx/requirements.txt
python3 -m pip install --upgrade cupy-cuda12x  # CUDA 12 compatible wheel
python3 -m pip install --upgrade torch  # CUDA 12 compatible wheel


# remove build temporary directory
rm -rf ${build_dir}

CPU Nodes

We use system software modules, add environment hints and further dependencies via the file $HOME/perlmutter_cpu_impactx.profile. Create it now:

cp $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu_impactx.profile.example $HOME/perlmutter_cpu_impactx.profile

Script Details

# please set your project account
export proj=""  # change me!

# remembers the location of this script
export MY_PROFILE=$(cd $(dirname $BASH_SOURCE) && pwd)"/"$(basename $BASH_SOURCE)
if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your $MY_PROFILE file! Please edit its line 2 to continue!"; return; fi

# required dependencies
module load cpu
module load cmake/3.24.3
module load cray-fftw/3.3.10.6

# optional: for QED support with detailed tables
export BOOST_ROOT=/global/common/software/spackecp/perlmutter/e4s-23.05/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/boost-1.82.0-ow5r5qrgslcwu33grygouajmuluzuzv3

# optional: for openPMD and PSATD+RZ support
module load cray-hdf5-parallel/1.12.2.9
export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/adios2-2.8.3:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/blaspp-master:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/lapackpp-master:$CMAKE_PREFIX_PATH

export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/c-blosc-1.21.1/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/adios2-2.8.3/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/blaspp-master/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/lapackpp-master/lib64:$LD_LIBRARY_PATH

export PATH=${CFS}/${proj}/${USER}/sw/perlmutter/cpu/adios2-2.8.3/bin:${PATH}

# optional: CCache
export PATH=/global/common/software/spackecp/perlmutter/e4s-23.08/default/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/ccache-4.8.2-cvooxdw5wgvv2g3vjxjkrpv6dopginv6/bin:$PATH

# optional: for Python bindings or libEnsemble
module load cray-python/3.11.5

if [ -d "${CFS}/${proj}/${USER}/sw/perlmutter/cpu/venvs/impactx" ]
then
  source ${CFS}/${proj}/${USER}/sw/perlmutter/cpu/venvs/impactx/bin/activate
fi

# an alias to request an interactive batch node for one hour
#   for parallel execution, start on the batch node: srun <command>
alias getNode="salloc --nodes 1 --qos interactive --time 01:00:00 --constraint cpu --account=$proj"
# an alias to run a command on a batch node for up to 30min
#   usage: runNode <command>
alias runNode="srun --nodes 1 --qos interactive --time 01:00:00 --constraint cpu $proj"

# optimize CPU microarchitecture for AMD EPYC 3rd Gen (Milan/Zen3)
# note: the cc/CC/ftn wrappers below add those
export CXXFLAGS="-march=znver3"
export CFLAGS="-march=znver3"

# compiler environment hints
export CC=cc
export CXX=CC
export FC=ftn

Edit the 2nd line of this script, which sets the export proj="" variable. For example, if you are member of the project m3239, then run nano $HOME/perlmutter_cpu_impactx.profile and edit line 2 to read:

export proj="m3239"

Exit the nano editor with Ctrl + O (save) and then Ctrl + X (exit).

Important

Now, and as the first step on future logins to Perlmutter, activate these environment settings:

source $HOME/perlmutter_cpu_impactx.profile

Finally, since Perlmutter does not yet provide software modules for some of our dependencies, install them once:

bash $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/install_cpu_dependencies.sh
source ${CFS}/${proj}/${USER}/sw/perlmutter/cpu/venvs/impactx/bin/activate

Script Details

#!/bin/bash
#
# Copyright 2023 The ImpactX Community
#
# This file is part of ImpactX.
#
# Author: Axel Huebl
# License: BSD-3-Clause-LBNL

# Exit on first error encountered #############################################
#
set -eu -o pipefail


# Check: ######################################################################
#
#   Was perlmutter_cpu_impactx.profile sourced and configured correctly?
if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your perlmutter_cpu_impactx.profile file! Please edit its line 2 to continue!"; exit 1; fi


# Check $proj variable is correct and has a corresponding CFS directory #######
#
if [ ! -d "${CFS}/${proj}/" ]
then
    echo "WARNING: The directory ${CFS}/${proj}/ does not exist!"
    echo "Is the \$proj environment variable of value \"$proj\" correctly set? "
    echo "Please edit line 2 of your perlmutter_cpu_impactx.profile file to continue!"
    exit
fi


# Remove old dependencies #####################################################
#
SW_DIR="${CFS}/${proj}/${USER}/sw/perlmutter/cpu"
rm -rf ${SW_DIR}
mkdir -p ${SW_DIR}

# remove common user mistakes in python, located in .local instead of a venv
python3 -m pip uninstall -qq -y impactx
python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true


# General extra dependencies ##################################################
#

# tmpfs build directory: avoids issues often seen with $HOME and is faster
build_dir=$(mktemp -d)

# c-blosc (I/O compression)
if [ -d $HOME/src/c-blosc ]
then
  cd $HOME/src/c-blosc
  git fetch --prune
  git checkout v1.21.1
  cd -
else
  git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git $HOME/src/c-blosc
fi
cmake -S $HOME/src/c-blosc -B ${build_dir}/c-blosc-pm-cpu-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1
cmake --build ${build_dir}/c-blosc-pm-cpu-build --target install --parallel 16
rm -rf ${build_dir}/c-blosc-pm-cpu-build

# ADIOS2
if [ -d $HOME/src/adios2 ]
then
  cd $HOME/src/adios2
  git fetch --prune
  git checkout v2.8.3
  cd -
else
  git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git $HOME/src/adios2
fi
cmake -S $HOME/src/adios2 -B ${build_dir}/adios2-pm-cpu-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_CUDA=OFF -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3
cmake --build ${build_dir}/adios2-pm-cpu-build --target install -j 16
rm -rf ${build_dir}/adios2-pm-cpu-build

# BLAS++ (for PSATD+RZ)
if [ -d $HOME/src/blaspp ]
then
  cd $HOME/src/blaspp
  git fetch --prune
  git checkout master
  git pull
  cd -
else
  git clone https://github.com/icl-utk-edu/blaspp.git $HOME/src/blaspp
fi
CXX=$(which CC) cmake -S $HOME/src/blaspp -B ${build_dir}/blaspp-pm-cpu-build -Duse_openmp=ON -Dgpu_backend=OFF -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master
cmake --build ${build_dir}/blaspp-pm-cpu-build --target install --parallel 16
rm -rf ${build_dir}/blaspp-pm-cpu-build

# LAPACK++ (for PSATD+RZ)
if [ -d $HOME/src/lapackpp ]
then
  cd $HOME/src/lapackpp
  git fetch --prune
  git checkout master
  git pull
  cd -
else
  git clone https://github.com/icl-utk-edu/lapackpp.git $HOME/src/lapackpp
fi
CXX=$(which CC) CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S $HOME/src/lapackpp -B ${build_dir}/lapackpp-pm-cpu-build -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master
cmake --build ${build_dir}/lapackpp-pm-cpu-build --target install --parallel 16
rm -rf ${build_dir}/lapackpp-pm-cpu-build


# Python ######################################################################
#
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade virtualenv
python3 -m pip cache purge
rm -rf ${SW_DIR}/venvs/impactx
python3 -m venv ${SW_DIR}/venvs/impactx
source ${SW_DIR}/venvs/impactx/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade build
python3 -m pip install --upgrade packaging
python3 -m pip install --upgrade wheel
python3 -m pip install --upgrade setuptools
python3 -m pip install --upgrade cython
python3 -m pip install --upgrade numpy
python3 -m pip install --upgrade pandas
python3 -m pip install --upgrade scipy
MPICC="cc -shared" python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py
python3 -m pip install --upgrade openpmd-api
python3 -m pip install --upgrade matplotlib
python3 -m pip install --upgrade yt
# install or update impactx dependencies
python3 -m pip install --upgrade -r $HOME/src/impactx/requirements.txt
python3 -m pip install --upgrade torch --index-url https://download.pytorch.org/whl/cpu


# remove build temporary directory
rm -rf ${build_dir}

Compilation

Use the following cmake commands to compile the application executable:

A100 GPUs

cd $HOME/src/impactx
rm -rf build_pm_gpu

cmake -S . -B build_pm_gpu -DImpactX_COMPUTE=CUDA
cmake --build build_pm_gpu -j 16

The ImpactX application executables are now in $HOME/src/impactx/build_pm_gpu/bin/. Additionally, the following commands will install ImpactX as a Python module:

cd $HOME/src/impactx
rm -rf build_pm_gpu_py

cmake -S . -B build_pm_gpu_py -DImpactX_COMPUTE=CUDA -DImpactX_APP=OFF -DImpactX_PYTHON=ON
cmake --build build_pm_gpu_py -j 16 --target pip_install

CPU Nodes

cd $HOME/src/impactx
rm -rf build_pm_cpu

cmake -S . -B build_pm_cpu -DImpactX_COMPUTE=OMP
cmake --build build_pm_cpu -j 16

The ImpactX application executables are now in $HOME/src/impactx/build_pm_cpu/bin/. Additionally, the following commands will install ImpactX as a Python module:

rm -rf build_pm_cpu_py

cmake -S . -B build_pm_cpu_py -DImpactX_COMPUTE=OMP -DImpactX_APP=OFF -DImpactX_PYTHON=ON
cmake --build build_pm_cpu_py -j 16 --target pip_install

Now, you can submit Perlmutter compute jobs for ImpactX Python (PICMI) scripts (example scripts). Or, you can use the ImpactX executables to submit Perlmutter jobs (example inputs). For executables, you can reference their location in your job script or copy them to a location in $PSCRATCH.

Update ImpactX & Dependencies

If you already installed ImpactX in the past and want to update it, start by getting the latest source code:

cd $HOME/src/impactx

# read the output of this command - does it look ok?
git status

# get the latest ImpactX source code
git fetch
git pull

# read the output of these commands - do they look ok?
git status
git log # press q to exit

And, if needed,

update the perlmutter_gpu_impactx.profile or perlmutter_cpu_impactx files,
log out and into the system, activate the now updated environment profile as usual,
execute the dependency install scripts.

As a last step, clean the build directory rm -rf $HOME/src/impactx/build_pm_* and rebuild ImpactX.

Running

A100 (40GB) GPUs

The batch script below can be used to run a ImpactX simulation on multiple nodes (change -N accordingly) on the supercomputer Perlmutter at NERSC. This partition as up to 1536 nodes.

Replace descriptions between chevrons <> by relevant values, for instance <input file> could be plasma_mirror_inputs. Note that we run one MPI rank per GPU.

You can copy this file from $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_gpu.sbatch.

#!/bin/bash -l

# Copyright 2021-2023 Axel Huebl, Kevin Gott
#
# This file is part of ImpactX.
#
# License: BSD-3-Clause-LBNL

#SBATCH -t 00:10:00
#SBATCH -N 2
#SBATCH -J ImpactX
#    note: <proj> must end on _g
#SBATCH -A <proj>
#SBATCH -q regular
# A100 40GB (most nodes)
#SBATCH -C gpu
# A100 80GB (256 nodes)
#S BATCH -C gpu&hbm80g
#SBATCH --exclusive
# ideally single:1, but NERSC cgroups issue
#SBATCH --gpu-bind=none
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-node=4
#SBATCH -o ImpactX.o%j
#SBATCH -e ImpactX.e%j

# executable & inputs file or python interpreter & PICMI script here
EXE=./impactx
INPUTS=inputs

# pin to closest NIC to GPU
export MPICH_OFI_NIC_POLICY=GPU

# threads for OpenMP and threaded compressors per MPI rank
#   note: 16 avoids hyperthreading (32 virtual cores, 16 physical)
export SRUN_CPUS_PER_TASK=16

# GPU-aware MPI optimizations
GPU_AWARE_MPI="amrex.use_gpu_aware_mpi=1"

# CUDA visible devices are ordered inverse to local task IDs
#   Reference: nvidia-smi topo -m
srun --cpu-bind=cores bash -c "
    export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID));
    ${EXE} ${INPUTS} ${GPU_AWARE_MPI}" \
  > output.txt

To run a simulation, copy the lines above to a file perlmutter_gpu.sbatch and run

sbatch perlmutter_gpu.sbatch

to submit the job.

A100 (80GB) GPUs

Perlmutter has 256 nodes that provide 80 GB HBM per A100 GPU. In the A100 (40GB) batch script, replace -C gpu with -C gpu&hbm80g to use these large-memory GPUs.

CPU Nodes

The Perlmutter CPU partition as up to 3072 nodes, each with 2x AMD EPYC 7763 CPUs.

You can copy this file from $HOME/src/impactx/docs/source/install/hpc/perlmutter-nersc/perlmutter_cpu.sbatch.

#!/bin/bash -l

# Copyright 2021-2023 ImpactX
#
# This file is part of ImpactX.
#
# Authors: Axel Huebl
# License: BSD-3-Clause-LBNL

#SBATCH -t 00:10:00
#SBATCH -N 2
#SBATCH -J ImpactX
#SBATCH -A <proj>
#SBATCH -q regular
#SBATCH -C cpu
#SBATCH --ntasks-per-node=16
#SBATCH --exclusive
#SBATCH -o ImpactX.o%j
#SBATCH -e ImpactX.e%j

# executable & inputs file or python interpreter & PICMI script here
EXE=./impactx
INPUTS=inputs_small

# each CPU node on Perlmutter (NERSC) has 64 hardware cores with
# 2x Hyperthreading/SMP
# https://en.wikichip.org/wiki/amd/epyc/7763
# https://www.amd.com/en/products/cpu/amd-epyc-7763
# Each CPU is made up of 8 chiplets, each sharing 32MB L3 cache.
# This will be our MPI rank assignment (2x8 is 16 ranks/node).

# threads for OpenMP and threaded compressors per MPI rank
export SRUN_CPUS_PER_TASK=16  # 8 cores per chiplet, 2x SMP
export OMP_PLACES=threads
export OMP_PROC_BIND=spread

srun --cpu-bind=cores \
  ${EXE} ${INPUTS} \
  > output.txt

Post-Processing

For post-processing, most users use Python via NERSC’s Jupyter service (documentation).

As a one-time preparatory setup, log into Perlmutter via SSH and do not source the ImpactX profile script above. Create your own Conda environment and Jupyter kernel for post-processing:

module load python

conda config --set auto_activate_base false

# create conda environment
rm -rf $HOME/.conda/envs/impactx-pm-postproc
conda create --yes -n impactx-pm-postproc -c conda-forge mamba conda-libmamba-solver
conda activate impactx-pm-postproc
conda config --set solver libmamba
mamba install --yes -c conda-forge python ipykernel ipympl matplotlib numpy pandas yt openpmd-viewer openpmd-api h5py fast-histogram dask dask-jobqueue pyarrow

# create Jupyter kernel
rm -rf $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/
python -m ipykernel install --user --name impactx-pm-postproc --display-name ImpactX-PM-PostProcessing
echo -e '#!/bin/bash\nmodule load python\nsource activate impactx-pm-postproc\nexec "$@"' > $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel-helper.sh
chmod a+rx $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel-helper.sh
KERNEL_STR=$(jq '.argv |= ["{resource_dir}/kernel-helper.sh"] + .' $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel.json | jq '.argv[1] = "python"')
echo ${KERNEL_STR} | jq > $HOME/.local/share/jupyter/kernels/impactx-pm-postproc/kernel.json

exit

When opening a Jupyter notebook on https://jupyter.nersc.gov, just select ImpactX-PM-PostProcessing from the list of available kernels on the top right of the notebook.

Additional software can be installed later on, e.g., in a Jupyter cell using !mamba install -y -c conda-forge .... Software that is not available via conda can be installed via !python -m pip install ....

LUMI (CSC)

The LUMI cluster is located at CSC (Finland). Each node contains 4 AMD MI250X GPUs, each with 2 Graphics Compute Dies (GCDs) for a total of 8 GCDs per node. You can think of the 8 GCDs as 8 separate GPUs, each having 64 GB of high-bandwidth memory (HBM2E).

Introduction

If you are new to this system, please see the following resources:

Lumi user guide
- Project Maintainance and SSH Key management
- Quotas and projects
Batch system: Slurm
Data analytics and visualization
Production directories:
- $HOME: single user, intended to store user configuration files and personal data (20GB default quota)
- /project/$proj: shared with all members of a project, purged at the end of a project (50 GB default quota)
- /scratch/$proj: temporary storage, main storage to be used for disk I/O needs when running simulations on LUMI, purged every 90 days (50TB default quota)

Preparation

Use the following commands to download the ImpactX source code:

git clone https://github.com/ECP-WarpX/impactx.git $HOME/src/impactx

We use system software modules, add environment hints and further dependencies via the file $HOME/lumi_impactx.profile. Create it now:

cp $HOME/src/impactx/docs/source/install/hpc/lumi-csc/lumi_impactx.profile.example $HOME/lumi_impactx.profile

Edit the 2nd line of this script, which sets the export proj="project_..." variable using a text editor such as nano, emacs, or vim (all available by default on LUMI login nodes). You can find out your project name by running lumi-ldap-userinfo on LUMI. For example, if you are member of the project project_465000962, then run nano $HOME/lumi_impactx.profile and edit line 2 to read:

export proj="project_465000962"

Exit the nano editor with Ctrl + O (save) and then Ctrl + X (exit).

Important

Now, and as the first step on future logins to LUMI, activate these environment settings:

source $HOME/lumi_impactx.profile

Finally, since LUMI does not yet provide software modules for some of our dependencies, install them once:

bash $HOME/src/impactx/docs/source/install/hpc/lumi-csc/install_dependencies.sh
source $HOME/sw/lumi/gpu/venvs/impactx-lumi/bin/activate

Script Details

#!/bin/bash
#
# Copyright 2023 The ImpactX Community
#
# This file is part of ImpactX.
#
# Author: Axel Huebl, Luca Fedeli
# License: BSD-3-Clause-LBNL

# Exit on first error encountered #############################################
#
set -eu -o pipefail


# Check: ######################################################################
#
#   Was lumi_warpx.profile sourced and configured correctly?
if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your lumi_impactx.profile file! Please edit its line 2 to continue!"; exit 1; fi


# Remove old dependencies #####################################################
#
SRC_DIR="${HOME}/src"
SW_DIR="${HOME}/sw/lumi/gpu"
rm -rf ${SW_DIR}
mkdir -p ${SW_DIR}
mkdir -p ${SRC_DIR}

# remove common user mistakes in python, located in .local instead of a venv
python3 -m pip uninstall -qq -y pyimpactx
python3 -m pip uninstall -qq -y impactx
python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true


# General extra dependencies ##################################################
#

# tmpfs build directory: avoids issues often seen with $HOME and is faster
build_dir=$(mktemp -d)

# c-blosc (I/O compression, for openPMD)
if [ -d ${SRC_DIR}/c-blosc ]
then
  cd ${SRC_DIR}/c-blosc
  git fetch --prune
  git checkout v1.21.1
  cd -
else
  git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git ${SRC_DIR}/c-blosc
fi
rm -rf ${SRC_DIR}/c-blosc-lu-build
cmake -S ${SRC_DIR}/c-blosc             \
      -B ${build_dir}/c-blosc-lu-build  \
      -DBUILD_TESTS=OFF                 \
      -DBUILD_BENCHMARKS=OFF            \
      -DDEACTIVATE_AVX2=OFF             \
      -DCMAKE_INSTALL_PREFIX=${HOME}/sw/lumi/gpu/c-blosc-1.21.1
cmake --build ${build_dir}/c-blosc-lu-build --target install --parallel 16
rm -rf ${build_dir}/c-blosc-lu-build

# HDF5 (for openPMD)
if [ -d ${SRC_DIR}/hdf5 ]
then
  cd ${SRC_DIR}/hdf5
  git fetch --prune
  git checkout hdf5-1_14_1-2
  cd -
else
  git clone -b hdf5-1_14_1-2 https://github.com/HDFGroup/hdf5.git ${SRC_DIR}/hdf5
fi
cmake -S ${SRC_DIR}/hdf5          \
      -B ${build_dir}/hdf5-build  \
      -DBUILD_TESTING=OFF         \
      -DHDF5_ENABLE_PARALLEL=ON   \
      -DCMAKE_INSTALL_PREFIX=${SW_DIR}/hdf5-1.14.1.2
cmake --build ${build_dir}/hdf5-build --target install --parallel 10
rm -rf ${build_dir}/hdf5-build

# ADIOS2 (for openPMD)
if [ -d ${SRC_DIR}/adios2 ]
then
  cd ${SRC_DIR}/adios2
  git fetch --prune
  git checkout v2.8.3
  cd -
else
  git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git ${SRC_DIR}/adios2
fi
rm -rf ${SRC_DIR}/adios2-lu-build
cmake -S ${SRC_DIR}/adios2             \
      -B ${build_dir}/adios2-lu-build  \
      -DADIOS2_USE_Blosc=ON            \
      -DADIOS2_USE_Fortran=OFF         \
      -DADIOS2_USE_HDF5=OFF            \
      -DADIOS2_USE_Python=OFF          \
      -DADIOS2_USE_ZeroMQ=OFF          \
      -DCMAKE_INSTALL_PREFIX=${HOME}/sw/lumi/gpu/adios2-2.8.3
cmake --build ${build_dir}/adios2-lu-build --target install -j 16
rm -rf ${build_dir}/adios2-lu-build


# Python ######################################################################
#
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade virtualenv
python3 -m pip cache purge
rm -rf ${SW_DIR}/venvs/impactx-lumi
python3 -m venv ${SW_DIR}/venvs/impactx-lumi
source ${SW_DIR}/venvs/impactx-lumi/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade build
python3 -m pip install --upgrade packaging
python3 -m pip install --upgrade wheel
python3 -m pip install --upgrade setuptools
python3 -m pip install --upgrade cython
python3 -m pip install --upgrade numpy
python3 -m pip install --upgrade pandas
python3 -m pip install --upgrade scipy
MPICC="cc -shared" python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py
python3 -m pip install --upgrade openpmd-api
python3 -m pip install --upgrade matplotlib
python3 -m pip install --upgrade yt
# install or update ImpactX dependencies
python3 -m pip install --upgrade -r ${SRC_DIR}/impactx/requirements.txt
# cupy: no ROCm 5.2 Python wheels
#python3 -m pip install --upgrade torch --index-url https://download.pytorch.org/whl/rocm5.4.2
#python3 -m pip install --upgrade optimas[all]

Compilation

Use the following cmake commands to compile the application executable:

cd $HOME/src/impactx
rm -rf build_lumi

cmake -S . -B build_lumi -DImpactX_COMPUTE=HIP
cmake --build build_lumi -j 16

The ImpactX application executables are now in $HOME/src/impactx/build_lumi/bin/. Additionally, the following commands will install ImpactX as a Python module:

rm -rf build_lumi_py

cmake -S . -B build_lumi_py -DImpactX_COMPUTE=HIP -DImpactX_PYTHON=ON
cmake --build build_lumi_py -j 16 --target pip_install

Update ImpactX & Dependencies

If you already installed ImpactX in the past and want to update it, start by getting the latest source code:

cd $HOME/src/impactx

# read the output of this command - does it look ok?
git status

# get the latest ImpactX source code
git fetch
git pull

# read the output of these commands - do they look ok?
git status
git log     # press q to exit

And, if needed,

update the lumi_impactx.profile file,
log out and into the system, activate the now updated environment profile as usual,
execute the dependency install scripts.

As a last step, clean the build directory rm -rf $HOME/src/impactx/build_lumi and rebuild ImpactX.

Running

MI250X GPUs (2x64 GB)

The GPU partition on the supercomputer LUMI at CSC has up to 2978 nodes, each with 8 Graphics Compute Dies (GCDs). ImpactX runs one MPI rank per Graphics Compute Die.

For interactive runs, simply use the aliases getNode or runNode ....

The batch script below can be used to run an ImpactX simulation on multiple nodes (change -N accordingly). Replace descriptions between chevrons <> by relevant values, for instance <project id> or the concete inputs file. Copy the executable or point to it via EXE and adjust the path for the INPUTS variable accordingly.

You can copy this file from $HOME/src/impactx/docs/source/install/hpc/lumi-csc/lumi.sbatch.

#!/bin/bash -l

#SBATCH -A <project id>
#SBATCH --nodes=1
#SBATCH --time=00:10:00
#SBATCH --job-name=impactx
#SBATCH --output=ImpactX.o%j
#SBATCH --error=ImpactX.e%j
#SBATCH --partition=standard-g
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8

# executable & inputs file or ...
EXE=./impactx
INPUTS=inputs
# ... python interpreter & PICMI script here
#EXE=python3
#INPUTS=run_fodo.py

date

# note (12-12-22)
# this environment setting is currently needed on LUMI to work-around a
# known issue with Libfabric
#export FI_MR_CACHE_MAX_COUNT=0  # libfabric disable caching
# or, less invasive:
export FI_MR_CACHE_MONITOR=memhooks  # alternative cache monitor

# Seen since August 2023 seen on OLCF (not yet seen on LUMI?)
# OLCFDEV-1597: OFI Poll Failed UNDELIVERABLE Errors
# https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#olcfdev-1597-ofi-poll-failed-undeliverable-errors
#export MPICH_SMP_SINGLE_COPY_MODE=NONE
#export FI_CXI_RX_MATCH_MODE=software

# note (9-2-22, OLCFDEV-1079)
# this environment setting is needed to avoid that rocFFT writes a cache in
# the home directory, which does not scale.
export ROCFFT_RTC_CACHE_PATH=/dev/null

# Seen since August 2023
# OLCFDEV-1597: OFI Poll Failed UNDELIVERABLE Errors
# https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#olcfdev-1597-ofi-poll-failed-undeliverable-errors
export MPICH_SMP_SINGLE_COPY_MODE=NONE
export FI_CXI_RX_MATCH_MODE=software

# LUMI documentation suggests using the following wrapper script
# to set the ROCR_VISIBLE_DEVICES to the value of SLURM_LOCALID
# see https://docs.lumi-supercomputer.eu/runjobs/scheduled-jobs/lumig-job/
cat << EOF > select_gpu
#!/bin/bash

export ROCR_VISIBLE_DEVICES=\$SLURM_LOCALID
exec \$*
EOF

chmod +x ./select_gpu

sleep 1

# LUMI documentation suggests using the following CPU bind
# in order to have 6 threads per GPU (blosc compression in adios2 uses threads)
# see https://docs.lumi-supercomputer.eu/runjobs/scheduled-jobs/lumig-job/
#
# WARNING: the following CPU_BIND options don't work on the dev-g partition.
#          If you want to run your simulation on dev-g, please comment them
#          out and replace them with CPU_BIND="map_cpu:49,57,17,25,1,9,33,41"
#
CPU_BIND="mask_cpu:7e000000000000,7e00000000000000"
CPU_BIND="${CPU_BIND},7e0000,7e000000"
CPU_BIND="${CPU_BIND},7e,7e00"
CPU_BIND="${CPU_BIND},7e00000000,7e0000000000"

export OMP_NUM_THREADS=6

export MPICH_GPU_SUPPORT_ENABLED=1

srun --cpu-bind=${CPU_BIND} ./select_gpu  \
  ${EXE} ${INPUTS}                        \
  | tee outputs.txt

rm -rf ./select_gpu

To run a simulation, copy the lines above to a file lumi.sbatch and run

sbatch lumi.sbatch

to submit the job.

Post-Processing

Note

TODO: Document any Jupyter or data services.

Known System Issues

Warning

December 12th, 2022: There is a caching bug in libFabric that causes ImpactX simulations to occasionally hang on LUMI on more than 1 node.

As a work-around, please export the following environment variable in your job scripts until the issue is fixed:

#export FI_MR_CACHE_MAX_COUNT=0  # libfabric disable caching
# or, less invasive:
export FI_MR_CACHE_MONITOR=memhooks  # alternative cache monitor

Warning

January, 2023: We discovered a regression in AMD ROCm, leading to 2x slower current deposition (and other slowdowns) in ROCm 5.3 and 5.4.

June, 2023: Although a fix was planned for ROCm 5.5, we still see the same issue in this release and continue to exchange with AMD and HPE on the issue.

Stay with the ROCm 5.2 module to avoid a 2x slowdown.

Warning

May 2023: rocFFT in ROCm 5.1-5.3 tries to write to a cache in the home area by default. This does not scale, disable it via:

export ROCFFT_RTC_CACHE_PATH=/dev/null

Tip

Your HPC system is not in the list? Our instructions are nearly identical to installing WarpX, documented here.

Also, please do not hesitate to open an issue and together we can document how to run on your preferred system!

Usage

Run ImpactX

In order to run a new simulation:

create a new directory, where the simulation will be run
make sure the ImpactX executable is either copied into this directory or in your PATH environment variable
add an inputs file and on HPC systems a submission script to the directory
run

1. Run Directory

On Linux/macOS, this is as easy as this

mkdir -p <run_directory>

Where <run_directory> by the actual path to the run directory.

2. Executable

If you installed ImpactX with a package manager, a impactx-prefixed executable will be available as a regular system command to you. Depending on the choosen build options, the name is suffixed with more details. Try it like this:

impactx<TAB>

Hitting the <TAB> key will suggest available ImpactX executables as found in your PATH environment variable.

If you compiled the code yourself, the ImpactX executable is stored in the source folder under build/bin. We also create a symbolic link that is just called impactx that points to the last executable you built, which can be copied, too. Copy the executable to this directory:

cp build/bin/<impactx_executable> <run_directory>/

where <impactx_executable> should be replaced by the actual name of the executable (see above) and <run_directory> by the actual path to the run directory.

3. Inputs

Add an input file in the directory (see examples and parameters). This file contains the numerical and physical parameters that define the situation to be simulated.

On HPC systems, also copy and adjust a submission script that allocated computing nodes for you. Please reach out to us if you need help setting up a template that runs with ideal performance.

4. Run

Run the executable, e.g. with MPI:

cd <run_directory>

# run with an inputs file:
mpirun -np <n_ranks> ./impactx <input_file>

or

# run with a PICMI input script:
mpirun -np <n_ranks> python <python_script>

Here, <n_ranks> is the number of MPI ranks used, and <input_file> is the name of the input file (<python_script> is the name of the PICMI script). Note that the actual executable might have a longer name, depending on build options.

We used the copied executable in the current directory (./); if you installed with a package manager, skip the ./ because ImpactX is in your PATH.

On an HPC system, you would instead submit the job script at this point, e.g. sbatch <submission_script> (SLURM on Perlmutter/NERSC) or bsub <submission_script> (LSF on Summit/OLCF).

Tip

In the next sections, we will explain parameters of the <input_file>. You can overwrite all parameters inside this file also from the command line, e.g.:

mpirun -np 4 ./impactx <input_file> max_step=10 amr.n_cell=64 64 128

Parameters: Python

This documents on how to use ImpactX as a Python script (python3 run_script.py).

General

class impactx.ImpactX

This is the central simulation class.

property particle_shape

Control the particle B-spline order.

The order of the shape factors (splines) for the macro-particles along all spatial directions: 1 for linear, 2 for quadratic, 3 for cubic. Low-order shape factors result in faster simulations, but may lead to more noisy results. High-order shape factors are computationally more expensive, but may increase the overall accuracy of the results. For production runs it is generally safer to use high-order shape factors, such as cubic order.

Parameters: order (int) – B-spline order 1, 2, or 3

property n_cell: The number of grid points along each direction on the coarsest level.

property max_level: The maximum mesh-refinement level for the simulation.

property finest_level: The currently finest level of mesh-refinement used. This is always less or equal to max_level.

property domain

The physical extent of the full simulation domain, relative to the reference particle position, in meters. When set, turns dynamic_size to False.

Note: particles that move outside the simulation domain are removed.

property prob_relative

This is a list with amr.max_level + 1 entries.

By default, we dynamically extract the minimum and maximum of the particle positions in the beam. The field mesh spans, per direction, multiple times the maximum physical extent of beam particles, as given by this factor. The beam minimum and maximum extent are symmetrically padded by the mesh. For instance, 1.2 means the mesh will span 10% above and 10% below the beam; 1.0 means the beam is exactly covered with the mesh.

Default: [3.0 1.0 1.0 ...]. When set, turns dynamic_size to True.

property dynamic_size: Use dynamic (True) resizing of the field mesh or static sizing (False).

property space_charge

Enable (True) or disable (False) space charge calculations (default: False).

Whether to calculate space charge effects.

property mlmg_relative_tolerance

Default: 1.e-7

The relative precision with which the electrostatic space-charge fields should be calculated. More specifically, the space-charge fields are computed with an iterative Multi-Level Multi-Grid (MLMG) solver. This solver can fail to reach the default precision within a reasonable time.

property mlmg_absolute_tolerance

Default: 0, which means: ignored

The absolute tolerance with which the space-charge fields should be calculated in units of $V/m^2$. More specifically, the acceptable residual with which the solution can be considered converged. In general this should be left as the default, but in cases where the simulation state changes very little between steps it can occur that the initial guess for the MLMG solver is so close to the converged value that it fails to improve that solution sufficiently to reach the mlmg_relative_tolerance value.

property mlmg_max_iters

Default: 100

Maximum number of iterations used for MLMG solver for space-charge fields calculation. In case if MLMG converges but fails to reach the desired self_fields_required_precision, this parameter may be increased.

property mlmg_verbosity

Default: 1

The verbosity used for MLMG solver for space-charge fields calculation. Currently MLMG solver looks for verbosity levels from 0-5. A higher number results in more verbose output.

property diagnostics: Enable (True) or disable (False) diagnostics generally (default: True). Disabling this is mostly used for benchmarking.

property slice_step_diagnostics

Enable (True) or disable (False) diagnostics every slice step in elements (default: True).

By default, diagnostics is performed at the beginning and end of the simulation. Enabling this flag will write diagnostics every step and slice step.

property diag_file_min_digits: The minimum number of digits (default: 6) used for the step number appended to the diagnostic file names.

property particle_lost_diagnostics_backend: Diagnostics for particles lost in apertures. See the BeamMonitor element for backend values.

init_grids()

Initialize AMReX blocks/grids for domain decomposition & space charge mesh.

This must come first, before particle beams and lattice elements are initialized.

add_particles(charge_C, distr, npart)

Generate and add n particles to the particle container. Note: Set the reference particle properties (charge, mass, energy) first.

Will also resize the geometry based on the updated particle distribution’s extent and then redistribute particles in according AMReX grid boxes.

Parameters

charge_C (float) – bunch charge (C)
distr – distribution function to draw from (object from impactx.distribution)
npart (int) – number of particles to draw

particle_container(): Access the beam particle container (impactx.ParticleContainer).

property lattice: Access the elements in the accelerator lattice. See impactx.elements for lattice elements.

property periods: The number of periods to repeat the lattice.

property abort_on_warning_threshold: (optional) Set to “low”, “medium” or “high”. Cause the code to abort if a warning is raised that exceeds the warning threshold.

property abort_on_unused_inputs: Set to 1 to cause the simulation to fail after its completion if there were unused parameters. (default: 0 for false) It is mainly intended for continuous integration and automated testing to check that all tests and inputs are adapted to API changes.

property always_warn_immediately: If set to 1, ImpactX immediately prints every warning message as soon as it is generated. (default: 0 for false) It is mainly intended for debug purposes, in case a simulation crashes before a global warning report can be printed.

property verbose: Controls how much information is printed to the terminal, when running ImpactX. 0 for silent, higher is more verbose. Default is 1.

evolve(): Run the main simulation loop for a number of steps.

resize_mesh(): Resize the mesh domain based on the dynamic_size and related parameters.

class impactx.Config

Configuration information on ImpactX that were set at compile-time.

property have_mpi: Indicates multi-process/multi-node support via the message-passing interface (MPI). Possible values: True/False

Note

Particle beam particles are not yet dynamically load balanced. Please see the progress in issue 198.

property have_gpu: Indicates GPU support. Possible values: True/False

property gpu_backend: Indicates the available GPU support. Possible values: None, "CUDA" (for Nvidia GPUs), "HIP" (for AMD GPUs) or "SYCL" (for Intel GPUs).

property have_omp

Indicates multi-threaded CPU support via OpenMP. Possible values: True/False`

Set the environment variable OMP_NUM_THREADS to control the number of threads.

Warning

By default, OpenMP spawns as many threads as there are available virtual cores on a host. When MPI and OpenMP support are used at the same time, it can easily happen that one over-subscribes the available physical CPU cores. This will lead to a severe slow-down of the simulation.

By setting appropriate environment variables for OpenMP, ensure that the number of MPI processes (ranks) per node multiplied with the number of OpenMP threads is equal to the number of physical (or virtual) CPU cores. Please see our examples in the high-performance computing (HPC) on how to run efficiently in parallel environments such as supercomputers.

Particles

class impactx.ParticleContainer

Beam Particles in ImpactX.

This class stores particles, distributed over MPI ranks.

add_n_particles(x, y, t, px, py, pt, qm, bchchg)

Add new particles to the container for fixed s.

Note: This can only be used after the initialization (grids) have: been created, meaning after the call to ImpactX.init_grids() has been made in the ImpactX class.

Parameters

x – positions in x
y – positions in y
t – positions as time-of-flight in c*t
px – momentum in x
py – momentum in y
pt – momentum in t
qm – charge over mass in 1/eV
bchchg – total charge within a bunch in C

ref_particle()

Access the reference particle (impactx.RefPart).

Returns: return a data reference to the reference particle
Return type: impactx.RefPart

set_ref_particle(refpart)

Set reference particle attributes.

Parameters: refpart (impactx.RefPart) – a reference particle to copy all attributes from

reduced_beam_characteristics()

Compute reduced beam characteristics like the position and momentum moments of the particle distribution, as well as emittance and Twiss parameters.

Returns: beam properties with string keywords
Return type: dict

min_and_max_positions()

Compute the min and max of the particle position in each dimension.

Returns: x_min, y_min, z_min, x_max, y_max, z_max
Return type: Tuple[float, float, float, float, float, float]

mean_and_std_positions()

Compute the mean and std of the particle position in each dimension.

Returns: x_mean, x_std, y_mean, y_std, z_mean, z_std
Return type: Tuple[float, float, float, float, float, float]

redistribute(): Redistribute particles in the current mesh in x, y, z.

class impactx.RefPart

This struct stores the reference particle attributes stored in impactx.ParticleContainer.

property s: integrated orbit path length, in meters

property x: horizontal position x, in meters

property y: vertical position y, in meters

property z: longitudinal position y, in meters

property t: clock time * c in meters

property px: momentum in x, normalized to mass*c, $p_x = \gamma \beta_x$

property py: momentum in y, normalized to mass*c, $p_x = \gamma \beta_x$

property pz: momentum in z, normalized to mass*c, $p_x = \gamma \beta_x$

property pt: energy, normalized by rest energy, $p_t = -\gamma$

property gamma: Read-only: Get reference particle relativistic gamma, $\gamma = 1/\sqrt{1-\beta^2}$

property beta: Read-only: Get reference particle relativistic beta, $\beta = v/c$

property beta_gamma: Read-only: Get reference particle $\beta \cdot \gamma$

property qm_qeeV: Read-only: Get reference particle charge to mass ratio (elementary charge/eV)

set_charge_qe(charge_qe): Write-only: Set reference particle charge in (positive) elementary charges.

set_mass_MeV(massE): Write-only: Set reference particle rest mass (MeV/c^2).

set_kin_energy_MeV(kin_energy_MeV): Write-only: Set reference particle kinetic energy (MeV)

load_file(madx_file)

Load reference particle information from a MAD-X file.

Parameters: madx_file – file name to MAD-X file with a BEAM entry

Initial Beam Distributions

This module provides particle beam distributions that can be used to initialize particle beams in an impactx.ParticleContainer.

class impactx.distribution.Gaussian(lambdax, lambday, lambdat, lambdapx, lambdapy, lambdapt, muxpx=0.0, muypy=0.0, mutpt=0.0)

A 6D Gaussian distribution.

Parameters

lambdax – phase space position axis intercept; for zero correlation, these are the related RMS sizes (in meters)
lambday – see lambdax
lambdat – see lambdax
lambdapx – phase space momentum axis intercept; for zero correlation, these are the related normalized RMS momenta (in radians)
lambdapy – see lambdapx
lambdapt – see lambdapx
muxpx – correlation length-momentum
muypy – see muxpx
mutpt – see muxpx

class impactx.distribution.Kurth4D(lambdax, lambday, lambdat, lambdapx, lambdapy, lambdapt, muxpx=0.0, muypy=0.0, mutpt=0.0): A 4D Kurth distribution transversely + a uniform distribution in t + a Gaussian distribution in pt.

class impactx.distribution.Kurth6D(lambdax, lambday, lambdat, lambdapx, lambdapy, lambdapt, muxpx=0.0, muypy=0.0, mutpt=0.0)

A 6D Kurth distribution.

R. Kurth, Quarterly of Applied Mathematics vol. 32, pp. 325-329 (1978) C. Mitchell, K. Hwang and R. D. Ryne, IPAC2021, WEPAB248 (2021)

class impactx.distribution.KVdist(lambdax, lambday, lambdat, lambdapx, lambdapy, lambdapt, muxpx=0.0, muypy=0.0, mutpt=0.0): A K-V distribution transversely + a uniform distribution in t + a Gaussian distribution in pt.

class impactx.distribution.None: This distribution sets all values to zero.

class impactx.distribution.Semigaussian(lambdax, lambday, lambdat, lambdapx, lambdapy, lambdapt, muxpx=0.0, muypy=0.0, mutpt=0.0): A 6D Semi-Gaussian distribution (uniform in position, Gaussian in momentum).

class impactx.distribution.Triangle(lambdax, lambday, lambdat, lambdapx, lambdapy, lambdapt, muxpx=0.0, muypy=0.0, mutpt=0.0)

A triangle distribution for laser-plasma acceleration related applications.

A ramped, triangular current profile with a Gaussian energy spread (possibly correlated). The transverse distribution is a 4D waterbag.

class impactx.distribution.Waterbag(lambdax, lambday, lambdat, lambdapx, lambdapy, lambdapt, muxpx=0.0, muypy=0.0, mutpt=0.0): A 6D Waterbag distribution.

class impactx.distribution.Thermal(k, kT, kT_halo, normalize, normalize_halo, halo): A 6D stationary thermal or bithermal distribution.

Lattice Elements

This module provides elements for the accelerator lattice.

class impactx.elements.KnownElementsList

An iterable, list-like type of elements.

clear(): Clear the list to become empty.

extend(list): Add a list of elements to the list.

append(element): Add a single element to the list.

load_file(madx_file, nslice=1)

Load and append an accelerator lattice description from a MAD-X file.

Parameters

madx_file – file name to MAD-X file with beamline elements
nslice – number of slices used for the application of space charge

class impactx.elements.CFbend(ds, rc, k, dx=0, dy=0, rotation=0, nslice=1)

A combined function bending magnet. This is an ideal Sbend with a normal quadrupole field component.

Parameters

ds – Segment length in m.
rc – Radius of curvature in m.
k – Quadrupole strength in m^(-2) (MADX convention) = (gradient in T/m) / (rigidity in T-m) k > 0 horizontal focusing k < 0 horizontal defocusing
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

class impactx.elements.ConstF(ds, kx, ky, kt, dx=0, dy=0, rotation=0, nslice=1)

A linear Constant Focusing element.

Parameters

ds – Segment length in m.
kx – Focusing strength for x in 1/m.
ky – Focusing strength for y in 1/m.
kt – Focusing strength for t in 1/m.
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

property kx: focusing x strength in 1/m

property ky: focusing y strength in 1/m

property kt: focusing t strength in 1/m

class impactx.elements.DipEdge(psi, rc, g, K2, dx=0, dy=0, rotation=0)

Edge focusing associated with bend entry or exit

This model assumes a first-order effect of nonzero gap. Here we use the linear fringe field map, given to first order in g/rc (gap / radius of curvature).

References:

1. 1. Brown, SLAC Report No. 75 (1982).
1. Hwang and S. Y. Lee, PRAB 18, 122401 (2015).

Parameters

psi – Pole face angle in rad
rc – Radius of curvature in m
g – Gap parameter in m
K2 – Fringe field integral (unitless)
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]

class impactx.elements.Drift(ds, dx=0, dy=0, rotation=0, nslice=1)

A drift.

Parameters

ds – Segment length in m
nslice – number of slices used for the application of space charge

class impactx.elements.ChrDrift(ds, dx=0, dy=0, rotation=0, nslice=1)

A drift with chromatic effects included. The Hamiltonian is expanded through second order in the transverse variables (x,px,y,py), with the exact pt dependence retained.

Parameters

ds – Segment length in m
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

class impactx.elements.ExactDrift(ds, dx=0, dy=0, rotation=0, nslice=1)

A drift using the exact nonlinear transfer map.

Parameters

ds – Segment length in m
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

class impactx.elements.Kicker(xkick, ykick, units, dx=0, dy=0, rotation=0)

A thin transverse kicker.

Parameters

xkick – horizontal kick strength (dimensionless OR T-m)
ykick – vertical kick strength (dimensionless OR T-m)
units – specification of units ("dimensionless" in units of the magnetic rigidity of the reference particle or "T-m")

class impactx.elements.Multipole(multipole, K_normal, K_skew, dx=0, dy=0, rotation=0)

A general thin multipole element.

Parameters

multipole – index m (m=1 dipole, m=2 quadrupole, m=3 sextupole etc.)
K_normal – Integrated normal multipole coefficient (1/meter^m)
K_skew – Integrated skew multipole coefficient (1/meter^m)
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]

class impactx.elements.NonlinearLens(knll, cnll, dx=0, dy=0, rotation=0)

Single short segment of the nonlinear magnetic insert element.

A thin lens associated with a single short segment of the nonlinear magnetic insert described by V. Danilov and S. Nagaitsev, PRSTAB 13, 084002 (2010), Sect. V.A. This element appears in MAD-X as type NLLENS.

Parameters

knll – integrated strength of the nonlinear lens (m)
cnll – distance of singularities from the origin (m)
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]

class impactx.elements.BeamMonitor(name, backend='default', encoding='g')

A beam monitor, writing all beam particles at fixed s to openPMD files.

If the same element name is used multiple times, then an output series is created with multiple outputs.

The I/O backend for openPMD data dumps. bp is the ADIOS2 I/O library, h5 is the HDF5 format, and json is a simple text format. json only works with serial/single-rank jobs. By default, the first available backend in the order given above is taken.

openPMD iteration encoding determines if multiple files are created for individual output steps or not. Variable based is an experimental feature with ADIOS2.

Parameters

name – name of the series
backend – I/O backend, e.g., bp, h5, json
encoding – openPMD iteration encoding: (v)ariable based, (f)ile based, (g)roup based (default)

property name: name of the series

property nonlinear_lens_invariants: Compute and output the invariants H and I within the nonlinear magnetic insert element

property alpha: Twiss alpha of the bare linear lattice at the location of output for the nonlinear IOTA invariants H and I. Horizontal and vertical values must be equal.

property beta: Twiss beta (in meters) of the bare linear lattice at the location of output for the nonlinear IOTA invariants H and I. Horizontal and vertical values must be equal.

property tn: Dimensionless strength of the IOTA nonlinear magnetic insert element used for computing H and I.

property cn: Scale factor (in meters^(1/2)) of the IOTA nonlinear magnetic insert element used for computing H and I.

class impactx.elements.Programmable

A programmable beam optics element.

This element can be programmed to receive callback hooks into Python functions.

property beam_particles

This is a function hook for pushing all beam particles. This accepts a function or lambda with the following arguments:

user_defined_function(pti: ImpactXParIter, refpart: RefPart): This function is called repeatedly for all particle tiles or boxes in the beam particle container. Particles can be pushed and are relative to the reference particle

property ref_particle

This is a function hook for pushing the reference particle. This accepts a function or lambda with the following argument:

another_user_defined_function(refpart: RefPart): This function is called for the reference particle as it passes through the element. The reference particle is updated before the beam particles are pushed.

class impactx.elements.Quad(ds, k, dx=0, dy=0, rotation=0, nslice=1)

A Quadrupole magnet.

Parameters

ds – Segment length in m.
k – Quadrupole strength in m^(-2) (MADX convention) = (gradient in T/m) / (rigidity in T-m) k > 0 horizontal focusing k < 0 horizontal defocusing
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

class impactx.elements.ChrQuad(ds, k, units, dx=0, dy=0, rotation=0, nslice=1)

A Quadrupole magnet, with chromatic effects included. The Hamiltonian is expanded through second order in the transverse variables (x,px,y,py), with the exact pt dependence retained.

Parameters

ds – Segment length in m.
k –

Quadrupole strength in m^(-2) (MADX convention, if units = 0)
= (gradient in T/m) / (rigidity in T-m)

OR Quadrupole strength in T/m (MaryLie convention, if units = 1)
k > 0 horizontal focusing k < 0 horizontal defocusing
units – specification of units for quadrupole field strength
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

property k: quadrupole strength in 1/m^2 (or T/m)

property units: unit specification for quad strength

class impactx.elements.ChrPlasmaLens(ds, g, dx=0, dy=0, rotation=0, nslice=1)

An active cylindrically symmetric plasma lens, with chromatic effects included. The Hamiltonian is expanded through second order in the transverse variables (x,px,y,py), with the exact pt dependence retained.

Parameters

ds – Segment length in m.
k –

focusing strength in m^(-2) (if units = 0)
= (azimuthal magnetic field gradient in T/m) / (rigidity in T-m)

OR azimuthal magnetic field gradient in T/m (if units = 1)
units – specification of units for plasma lens focusing strength
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

property k: plasma lens focusing strength in 1/m^2 (or T/m)

property units: unit specification for plasma lens focusing strength

class impactx.elements.ChrAcc(ds, ez, bz, dx=0, dy=0, rotation=0, nslice=1)

Acceleration in a uniform field Ez, with a uniform solenoidal field Bz.

The Hamiltonian is expanded through second order in the transverse variables (x,px,y,py), with the exact pt dependence retained.

Parameters

ds – Segment length in m
ez – electric field strength in m^(-1) = (charge * electric field Ez in V/m) / (m*c^2)
bz – magnetic field strength in m^(-1) = (charge * magnetic field Bz in T) / (m*c)
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

property ez: electric field strength in 1/m

property bz: magnetic field strength in 1/m

class impactx.elements.RFCavity(ds, escale, freq, phase, dx=0, dy=0, rotation=0, mapsteps=1, nslice=1)

A radiofrequency cavity.

Parameters

ds – Segment length in m.
escale – scaling factor for on-axis RF electric field in 1/m = (peak on-axis electric field Ez in MV/m) / (particle rest energy in MeV)
freq – RF frequency in Hz
phase – RF driven phase in degrees
cos_coefficients – array of float cosine coefficients in Fourier expansion of on-axis electric field Ez (optional); default is a 9-cell TESLA superconducting cavity model from DOI:10.1103/PhysRevSTAB.3.092001
cos_coefficients – array of float sine coefficients in Fourier expansion of on-axis electric field Ez (optional); default is a 9-cell TESLA superconducting cavity model from DOI:10.1103/PhysRevSTAB.3.092001
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
mapsteps – number of integration steps per slice used for map and reference particle push in applied fields
nslice – number of slices used for the application of space charge

class impactx.elements.Sbend(ds, rc, dx=0, dy=0, rotation=0, nslice=1)

An ideal sector bend.

Parameters

ds – Segment length in m.
rc – Radius of curvature in m.
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

class impactx.elements.ExactSbend(ds, phi, B, dx=0, dy=0, rotation=0, nslice=1)

An ideal sector bend using the exact nonlinear map. The model consists of a uniform bending field B_y with a hard edge. Pole faces are normal to the entry and exit velocity of the reference particle.

References:

1. 1. Bruhwiler et al, in Proc. of EPAC 98, pp. 1171-1173 (1998).
1. Forest et al, Part. Accel. 45, pp. 65-94 (1994).

Parameters

ds – Segment length in m.
phi – Bend angle in degrees.
B – Magnetic field in Tesla; when B = 0 (default), the reference bending radius is defined by r0 = length / (angle in rad), corresponding to a magnetic field of B = rigidity / r0; otherwise the reference bending radius is defined by r0 = rigidity / B.
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

class impactx.elements.Buncher(V, k, dx=0, dy=0, rotation=0)

A short RF cavity element at zero crossing for bunching (MaryLie model).

Parameters

V – Normalized RF voltage drop V = Emax*L/(c*Brho)
k – Wavenumber of RF in 1/m
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]

class impactx.elements.ShortRF(V, freq, phase, dx=0, dy=0, rotation=0)

A short RF cavity element (MAD-X model).

Parameters

V – Normalized RF voltage V = maximum energy gain/(m*c^2)
freq – RF frequency in Hz
phase – RF synchronous phase in degrees (phase = 0 corresponds to maximum energy gain, phase = -90 corresponds go zero energy gain for bunching)
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]

class impactx.elements.ChrUniformAcc(ds, k, dx=0, dy=0, rotation=0, nslice=1)

A region of constant Ez and Bz for uniform acceleration, with chromatic effects included. The Hamiltonian is expanded through second order in the transverse variables (x,px,y,py), with the exact pt dependence retained.

Parameters

ds – Segment length in m.
ez – Electric field strength in m^(-1) = (particle charge in C * field Ez in V/m) / (particle mass in kg * (speed of light in m/s)^2)
bz – Magnetic field strength in m^(-1) = (particle charge in C * field Bz in T) / (particle mass in kg * speed of light in m/s)
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

class impactx.elements.SoftSolenoid(ds, bscale, cos_coefficients, sin_coefficients, dx=0, dy=0, rotation=0, mapsteps=1, nslice=1)

A soft-edge solenoid.

Parameters

ds – Segment length in m.
bscale – Scaling factor for on-axis magnetic field Bz in inverse meters
cos_coefficients – array of float cosine coefficients in Fourier expansion of on-axis magnetic field Bz (optional); default is a thin-shell model from DOI:10.1016/J.NIMA.2022.166706
sin_coefficients – array of float sine coefficients in Fourier expansion of on-axis magnetic field Bz (optional); default is a thin-shell model from DOI:10.1016/J.NIMA.2022.166706
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
mapsteps – number of integration steps per slice used for map and reference particle push in applied fields
nslice – number of slices used for the application of space charge

class impactx.elements.Sol(ds, ks, dx=0, dy=0, rotation=0, nslice=1)

An ideal hard-edge Solenoid magnet.

Parameters

ds – Segment length in m.
ks – Solenoid strength in m^(-1) (MADX convention) in (magnetic field Bz in T) / (rigidity in T-m)
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
nslice – number of slices used for the application of space charge

class impactx.elements.PRot(phi_in, phi_out)

Exact map for a pole-face rotation in the x-z plane.

Parameters

phi_in – angle of the reference particle with respect to the longitudinal (z) axis in the original frame in degrees
phi_out – angle of the reference particle with respect to the longitudinal (z) axis in the rotated frame in degrees
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]

class impactx.elements.Aperture(xmax, ymax, shape='rectangular', dx=0, dy=0, rotation=0)

A thin collimator element, applying a transverse aperture boundary.

Parameters

xmax – maximum allowed value of the horizontal coordinate (meter)
ymax – maximum allowed value of the vertical coordinate (meter)
shape – aperture boundary shape: "rectangular" (default) or "elliptical"
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]

property shape: aperture type (rectangular, elliptical)

property xmax: maximum horizontal coordinate

property ymax: maximum vertical coordinate

class impactx.elements.SoftQuadrupole(ds, gscale, cos_coefficients, sin_coefficients, dx=0, dy=0, rotation=0, mapsteps=1, nslice=1)

A soft-edge quadrupole.

Parameters

ds – Segment length in m.
gscale – Scaling factor for on-axis field gradient in inverse meters
cos_coefficients – array of float cosine coefficients in Fourier expansion of on-axis field gradient (optional); default is a tanh fringe field model based on http://www.physics.umd.edu/dsat/docs/MaryLieMan.pdf
sin_coefficients – array of float sine coefficients in Fourier expansion of on-axis field gradient (optional); default is a tanh fringe field model based on http://www.physics.umd.edu/dsat/docs/MaryLieMan.pdf
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]
mapsteps – number of integration steps per slice used for map and reference particle push in applied fields
nslice – number of slices used for the application of space charge

class impactx.elements.ThinDipole(theta, rc, dx=0, dy=0, rotation=0)

A general thin dipole element.

Parameters

theta – Bend angle (degrees)
rc – Effective curvature radius (meters)
dx – horizontal translation error in m
dy – vertical translation error in m
rotation – rotation error in the transverse plane [degrees]

Reference:

1. Ripken and F. Schmidt, Thin-Lens Formalism for Tracking, CERN/SL/95-12 (AP), 1995.

Coordinate Transformation

class impactx.TransformationDirection

Enumerated type indicating whether to transform to fixed $s$ or fixed $t$ coordinate system when applying impactx.coordinate_transformation.

Parameters

to_fixed_t –
to_fixed_s –

impactx.coordinate_transformation(pc, direction)

Function to transform the coordinates of the particles in a particle container either to fixed $t$ or to fixed $s$.

Parameters

pc – impactx.particle_container whose particle coordinates are to be transformed.
direction – enumerated type impactx.TransformationDirection, indicates whether to transform to fixed $s$ or fixed $t$.

Parameters: Inputs File

This documents on how to use ImpactX with an inputs file (impactx input_file.in).

Note

The AMReX parser (see Math parser and user-defined constants) is used for the right-hand-side of all input parameters that consist of one or more integers or floats, so expressions like <species_name>.density_max = "2.+1." and/or using user-defined constants are accepted.

Overall simulation parameters

max_step (integer)
The number of PIC cycles to perform.
stop_time (float; in seconds)
The maximum physical time of the simulation. Can be provided instead of max_step. If both max_step and stop_time are provided, both criteria are used and the simulation stops when the first criterion is hit.
amrex.abort_on_out_of_gpu_memory (0 or 1; default is 1 for true)
When running on GPUs, memory that does not fit on the device will be automatically swapped to host memory when this option is set to 0. This will cause severe performance drops. Note that even with this set to 1 ImpactX will not catch all out-of-memory events yet when operating close to maximum device memory. Please also see the documentation in AMReX.
amrex.the_arena_is_managed (0 or 1; default is 0 for false)
When running on GPUs, device memory that is accessed from the host will automatically be transferred with managed memory. This is useful for convenience during development, but has sometimes severe performance and memory footprint implications if relied on (and sometimes vendor bugs). For all regular ImpactX operations, we therefore do explicit memory transfers without the need for managed memory and thus changed the AMReX default to false. Please also see the documentation in AMReX.
amrex.omp_threads (system, nosmt or positive integer; default is nosmt)
An integer number can be set in lieu of the OMP_NUM_THREADS environment variable to control the number of OpenMP threads to use for the OMP compute backend on CPUs. By default, we use the nosmt option, which overwrites the OpenMP default of spawning one thread per logical CPU core, and instead only spawns a number of threads equal to the number of physical CPU cores on the machine. If set, the environment variable OMP_NUM_THREADS takes precedence over system and nosmt, but not over integer numbers set in this option.
amrex.abort_on_unused_inputs (0 or 1; default is 0 for false)
When set to 1, this option causes the simulation to fail after its completion if there were unused parameters. It is mainly intended for continuous integration and automated testing to check that all tests and inputs are adapted to API changes.
impactx.always_warn_immediately (0 or 1; default is 0 for false)
If set to 1, ImpactX immediately prints every warning message as soon as it is generated. It is mainly intended for debug purposes, in case a simulation crashes before a global warning report can be printed.
impactx.abort_on_warning_threshold (string: low, medium or high) optional
Optional threshold to abort as soon as a warning is raised. If the threshold is set, warning messages with priority greater than or equal to the threshold trigger an immediate abort. It is mainly intended for debug purposes, and is best used with impactx.always_warn_immediately=1. For more information on the warning logger, see this section of the WarpX documentation.
impactx.verbose (int: 0 for silent, higher is more verbose; default is 1) optional
Controls how much information is printed to the terminal, when running ImpactX.

Setting up the field mesh

ImpactX uses an AMReX grid of boxes to organize and parallelize the simulation domain. These boxes also contain a field mesh, if space charge calculations are enabled.

amr.n_cell (3 integers) optional (default: 1 blocking_factor per MPI process)
The number of grid points along each direction (on the coarsest level)
amr.max_level (integer, default: 0)
When using mesh refinement, the number of refinement levels that will be used.

Use 0 in order to disable mesh refinement.
amr.ref_ratio (integer per refined level, default: 2)
When using mesh refinement, this is the refinement ratio per level. With this option, all directions are fined by the same ratio.
amr.ref_ratio_vect (3 integers for x,y,z per refined level)
When using mesh refinement, this can be used to set the refinement ratio per direction and level, relative to the previous level.

Example: for three levels, a value of 2 2 4 8 8 16 refines the first level by 2-fold in x and y and 4-fold in z compared to the coarsest level (level 0/mother grid); compared to the first level, the second level is refined 8-fold in x and y and 16-fold in z.

Note

Field boundaries for space charge calculation are located at the outer ends of the field mesh. We currently assume Dirichlet boundary conditions with zero potential (a mirror charge). Thus, to emulate open boundaries, consider adding enough vacuum padding to the beam. This will be improved in future versions.

Note

Particles that move outside the simulation domain are removed.

geometry.dynamic_size (boolean) optional (default: true for dynamic)
Use dynamic (true) resizing of the field mesh, via geometry.prob_relative, or static sizing (false), via geometry.prob_lo/geometry.prob_hi.
geometry.prob_relative (positive float array with amr.max_level entries, unitless) optional (default: 3.0 1.0 1.0 ...)
By default, we dynamically extract the minimum and maximum of the particle positions in the beam. The field mesh spans, per direction, multiple times the maximum physical extent of beam particles, as given by this factor. The beam minimum and maximum extent are symmetrically padded by the mesh. For instance, 1.2 means the mesh will span 10% above and 10% below the beam; 1.0 means the beam is exactly covered with the mesh.
geometry.prob_lo and geometry.prob_hi (3 floats, in meters) optional (required if geometry.dynamic_size is false)
The extent of the full simulation domain relative to the reference particle position. This can be used to explicitly size the simulation box and ignore geometry.prob_relative.

This box is rectangular, and thus its extent is given here by the coordinates of the lower corner (geometry.prob_lo) and upper corner (geometry.prob_hi). The first axis of the coordinates is x and the last is z.

Domain Boundary Conditions

Note

TODO :-)

Initial Beam Distributions

beam.npart (integer) number of weighted simulation particles
beam.units (string) currently, only static is supported.
beam.kin_energy (float, in MeV) beam kinetic energy
beam.charge (float, in C) bunch charge
beam.particle (string) particle type: currently either electron, positron or proton
beam.distribution (string)
Indicates the initial distribution type. For additional information, consult the documentation on Beam Distribution Input. For all except the thermal distribution we allow input in two forms:
- Parameters that describe the phase space ellipse and position-momentum correlations:
  beam.lambdaX (float, in meters) phase space ellipse intersection with X
  
  beam.lambdaY (float, in meters) phase space ellipse intersection with Y
  
  beam.lambdaT (float, in meters) phase space ellipse intersection with T, normalized by multiplying with the speed of light c
  
  beam.lambdaPx (float, in radians) phase space ellipse intersection with Px
  
  beam.lambdaPy (float, in radians) phase space ellipse intersection with Py
  
  beam.lambdaPt (float, in radians) phase space ellipse intersection with Pt
  
  beam.muxpx (float, dimensionless, default: 0) correlation X-Px
  
  beam.muypy (float, dimensionless, default: 0) correlation Y-Py
  
  beam.mutpt (float, dimensionless, default: 0) correlation T-Pt
- Courant-Snyder / Twiss parameters. To enable input via CS / Twiss parameters, add the suffix _from_cs or from_twiss to the name of the distribution. Use the following parameters to characterize it:
  beam.alphaX (float, dimensionless, default: 0) CS / Twiss $\alpha$ for X
  
  beam.alphaY (float, dimensionless, default: 0) CS / Twiss $\alpha$ for Y
  
  beam.alphaT (float, dimensionless, default: 0) CS / Twiss $\alpha$ for T
  
  beam.betaX (float, in meters) CS / Twiss $\beta$ for X
  
  beam.betaY (float, in meters) CS / Twiss $\beta$ for Y
  
  beam.betaT (float, in meters) CS / Twiss $\beta$ for T
  
  beam.emittX (float, in meters times radian) geometric (unnormalized) emittance $\epsilon$ in X
  
  beam.emittY (float, in meters times radian) geometric (unnormalized) emittance $\epsilon$ in Y
  
  beam.emittT (float, in meters times radian) geometric (unnormalized) emittance $\epsilon$ in T
The following distributions are available:
- waterbag or waterbag_from_cs/waterbag_from_twiss for initial Waterbag distribution.
- kurth6d or kurth6d_from_cs/kurth6d_from_twiss for initial 6D Kurth distribution.
- gaussian or gaussian_from_cs/gaussian_from_twiss for initial 6D Gaussian (normal) distribution.
- kvdist or kvdist_from_cs/kvdist_from_twiss for initial K-V distribution in the transverse plane. The distribution is uniform in t and Gaussian in pt.
- kurth4d or kurth4d_from_cs/kurth4d_from_twiss for initial 4D Kurth distribution in the transverse plane. The distribution is uniform in t and Gaussian in pt.
- semigaussian or semigaussian_from_cs/semigaussian_from_twiss for initial Semi-Gaussian distribution. The distribution is uniform within a cylinder in (x,y,z) and Gaussian in momenta (px,py,pt).
- triangle or triangle_from_cs/triangle_from_twiss a triangle distribution for laser-plasma acceleration related applications. A ramped, triangular current profile with a Gaussian energy spread (possibly correlated). The transverse distribution is a 4D waterbag.
- thermal for a 6D stationary thermal or bithermal distribution. This distribution type is described, for example in: R. D. Ryne et al, “A Test Suite of Space-Charge Problems for Code Benchmarking”, in Proc. EPAC2004, Lucerne, Switzerland. C. E. Mitchell et al, “ImpactX Modeling of Benchmark Tests for Space Charge Validation”, in Proc. HB2023, Geneva, Switzerland. With additional parameters:
  beam.k (float, in inverse meters) external focusing strength
  
  beam.kT (float, dimensionless) temperature of core population
  = < p_x^2 > = < p_y^2 >, where all momenta are normalized by the reference momentum
  
  beam.kT_halo (float, dimensionless, default kT) temperature of halo population
  
  beam.normalize (float, dimensionless) normalizing constant for core population
  
  beam.normalize_halo (float, dimensionless) normalizing constant for halo population
  
  beam.halo (float, dimensionless) fraction of charge in halo

Lattice Elements

lattice.elements (list of strings) optional (default: no elements)
A list of names (one name per lattice element), in the order that they appear in the lattice.
lattice.periods (integer) optional (default: 1)
The number of periods to repeat the lattice.
lattice.reverse (boolean) optional (default: false)
Reverse the list of elements in the lattice. If reverse and periods both appear, then reverse is applied before periods.
lattice.nslice (integer) optional (default: 1)
A positive integer specifying the number of slices used for the application of space charge in all elements; overwritten by element parameter “nslice”
<element_name>.type (string)
Indicates the element type for this lattice element. This should be one of:
cfbend for a combined function bending magnet. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.rc (float, in meters) the bend radius

<element_name>.k (float, in inverse meters squared) the quadrupole strength
= (magnetic field gradient in T/m) / (magnetic rigidity in T-m)

k > 0 horizontal focusing

k < 0 horizontal defocusing

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

drift for a free drift. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

drift_chromatic for a free drift, with chromatic effects included.
The Hamiltonian is expanded through second order in the transverse variables (x,px,y,py), with the exact pt dependence retained. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

drift_exact for a free drift, using the exact nonlinear map. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

quad for a quadrupole. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.k (float, in inverse meters squared) the quadrupole strength
= (magnetic field gradient in T/m) / (magnetic rigidity in T-m)

k > 0 horizontal focusing

k < 0 horizontal defocusing

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

quad_chromatic for A Quadrupole magnet, with chromatic effects included.
The Hamiltonian is expanded through second order in the transverse variables (x,px,y,py), with the exact pt dependence retained. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.k (float, in inverse meters squared OR in T/m) the quadrupole strength
= (magnetic field gradient in T/m) / (magnetic rigidity in T-m) - if units = 0

OR = magnetic field gradient in T/m - if units = 1

k > 0 horizontal focusing

k < 0 horizontal defocusing

<element_name>.units (integer) specification of units (default: 0)

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

quadrupole_softedge for a soft-edge quadrupole. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.gscale (float, in inverse meters) Scaling factor for on-axis magnetic field gradient

<element_name>.cos_coefficients (array of float) cos coefficients in Fourier expansion of the on-axis field gradient (optional); default is a tanh fringe field model from MaryLie 3.0

<element_name>.sin_coefficients (array of float) sin coefficients in Fourier expansion of the on-axis field gradient (optional); default is a tanh fringe field model from MaryLie 3.0

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.mapsteps (integer) number of integration steps per slice used for map and reference particle push in applied fields
(default: 1)

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

plasma_lens_chromatic for an active cylindrically-symmetric plasma lens, with chromatic effects included.
The Hamiltonian is expanded through second order in the transverse variables (x,px,y,py), with the exact pt dependence retained. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.k (float, in inverse meters squared OR in T/m) the plasma lens focusing strength
= (azimuthal magnetic field gradient in T/m) / (magnetic rigidity in T-m) - if units = 0

OR = azimuthal magnetic field gradient in T/m - if units = 1

<element_name>.units (integer) specification of units (default: 0)

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

sbend for a bending magnet. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.rc (float, in meters) the bend radius

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

sbend_exact for a bending magnet using the exact nonlinear map for the bend body. The map corresponds to the map described in:
D. L. Bruhwiler et al, in Proc. of EPAC 98, pp. 1171-1173 (1998), E. Forest et al, Part. Accel. 45, pp. 65-94 (1994). The model consists of a uniform bending field B_y with a hard edge. Pole faces are normal to the entry and exit velocity of the reference particle. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.phi (float, in degrees) the bend angle

<element_name>.B (float, in Tesla) the bend magnetic field; when B = 0 (default), the reference bending radius is defined by r0 = length / (angle in rad), corresponding to a magnetic field of B = rigidity / r0; otherwise the reference bending radius is defined by r0 = rigidity / B

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

solenoid for an ideal hard-edge solenoid magnet. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.ks (float, in meters) Solenoid strength in m^(-1) (MADX convention)
= (magnetic field Bz in T) / (rigidity in T-m)

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

solenoid_softedge for a soft-edge solenoid. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.bscale (float, in inverse meters) Scaling factor for on-axis magnetic field Bz

<element_name>.cos_coefficients (array of float) cos coefficients in Fourier expansion of the on-axis magnetic field Bz (optional); default is a thin-shell model from DOI:10.1016/J.NIMA.2022.166706

<element_name>.sin_coefficients (array of float) sin coefficients in Fourier expansion of the on-axis magnetic field Bz (optional); default is a thin-shell model from DOI:10.1016/J.NIMA.2022.166706

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.mapsteps (integer) number of integration steps per slice used for map and reference particle push in applied fields (default: 1)

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

dipedge for dipole edge focusing. This requires these additional parameters:

<element_name>.psi (float, in radians) the pole face rotation angle

<element_name>.rc (float, in meters) the bend radius

<element_name>.g (float, in meters) the gap size

<element_name>.K2 (float, dimensionless) normalized field integral for fringe field

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

constf for a constant focusing element. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.kx (float, in 1/meters) the horizontal focusing strength

<element_name>.ky (float, in 1/meters) the vertical focusing strength

<element_name>.kt (float, in 1/meters) the longitudinal focusing strength

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

rfcavity a radiofrequency cavity. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.escale (float, in 1/m) scaling factor for on-axis RF electric field
= (peak on-axis electric field Ez in MV/m) / (particle rest energy in MeV)

<element_name>.freq (float, in Hz) RF frequency

<element_name>.phase (float, in degrees) RF driven phase

<element_name>.cos_coefficients (array of float) cosine coefficients in Fourier expansion of on-axis electric field Ez (optional); default is a 9-cell TESLA superconducting cavity model from DOI:10.1103/PhysRevSTAB.3.092001

<element_name>.cos_coefficients (array of float) sine coefficients in Fourier expansion of on-axis electric field Ez (optional); default is a 9-cell TESLA superconducting cavity model from DOI:10.1103/PhysRevSTAB.3.092001

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.mapsteps (integer) number of integration steps per slice used for map and reference particle push in applied fields (default: 1)

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

buncher for a short RF cavity (linear) bunching element. This requires these additional parameters:

<element_name>.V (float, dimensionless) normalized voltage drop across the cavity
= (maximum voltage drop in Volts) / (speed of light in m/s * magnetic rigidity in T-m)

<element_name>.k (float, in 1/meters) the RF wavenumber
= 2*pi/(RF wavelength in m)

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

shortrf for a short RF cavity element. This requires these additional parameters:

<element_name>.V (float, dimensionless) normalized voltage drop across the cavity
= (maximum energy gain in MeV) / (particle rest energy in MeV)

<element_name>.freq (float, in Hz) the RF frequency

<element_name>.phase (float, in degrees) the synchronous RF phase

phase = 0: maximum energy gain (on-crest)

phase = -90 deg: zero energy gain for bunching

phase = 90 deg: zero energy gain for debunching

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

uniform_acc_chromatic for a region of uniform acceleration, with chromatic effects included.
The Hamiltonian is expanded through second order in the transverse variables (x,px,y,py), with the exact pt dependence retained. This requires these additional parameters:

<element_name>.ds (float, in meters) the segment length

<element_name>.ez (float, in inverse meters) the electric field strength
= (particle charge in C * electric field Ez in V/m) / (particle mass in kg * (speed of light in m/s)^2)

<element_name>.bz (float, in inverse meters) the magnetic field strength
= (particle charge in C * magnetic field Bz in T) / (particle mass in kg * speed of light in m/s)

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

<element_name>.nslice (integer) number of slices used for the application of space charge (default: 1)

multipole for a thin multipole element. This requires these additional parameters:

<element_name>.multipole (integer, dimensionless) order of multipole
(m = 1) dipole, (m = 2) quadrupole, (m = 3) sextupole, etc.

<element_name>.k_normal (float, in 1/meters^m) integrated normal multipole coefficient (MAD-X convention)
= 1/(magnetic rigidity in T-m) * (derivative of order m-1 of By with respect to x)

<element_name>.k_skew (float, in 1/meters^m) integrated skew multipole strength (MAD-X convention)

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

nonlinear_lens for a thin IOTA nonlinear lens element. This requires these additional parameters:

<element_name>.knll (float, in meters) integrated strength of the lens segment (MAD-X convention)
= dimensionless lens strength * c parameter**2 * length / Twiss beta

<element_name>.cnll (float, in meters) distance of the singularities from the origin (MAD-X convention)
= c parameter * sqrt(Twiss beta)

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

prot for an exact pole-face rotation in the x-z plane. This requires these additional parameters:

<element_name>.phi_in (float, in degrees) angle of the reference particle with respect to the longitudinal (z) axis in the original frame

<element_name>.phi_out (float, in degrees) angle of the reference particle with respect to the longitudinal (z) axis in the rotated frame

kicker for a thin transverse kicker. This requires these additional parameters:

<element_name>.xkick (float, dimensionless OR in T-m) the horizontal kick strength

<element_name>.ykick (float, dimensionless OR in T-m) the vertical kick strength

<element_name>.units (string) specification of units: dimensionless (default, in units of the magnetic rigidity of the reference particle) or T-m

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

thin_dipole for a thin dipole element. This requires these additional parameters:

<element_name>.theta (float, in degrees) dipole bend angle

<element_name>.rc (float, in meters) effective radius of curvature

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

aperture for a thin collimator element applying a transverse aperture boundary. This requires these additional parameters:

<element_name>.xmax (float, in meters) maximum value of the horizontal coordinate

<element_name>.ymax (float, in meters) maximum value of the vertical coordinate

<element_name>.shape (string) shape of the aperture boundary: rectangular (default) or elliptical

<element_name>.dx (float, in meters) horizontal translation error

<element_name>.dy (float, in meters) vertical translation error

<element_name>.rotation (float, in degrees) rotation error in the transverse plane

beam_monitor a beam monitor, writing all beam particles at fixed s to openPMD files. If the same element name is used multiple times, then an output series is created with multiple outputs.

<element_name>.name (string, default value: <element_name>)

The output series name to use. By default, output is created under diags/openPMD/<element_name>.<backend>.

<element_name>.backend (string, default value: default)

I/O backend for openPMD data dumps. bp is the ADIOS2 I/O library, h5 is the HDF5 format, and json is a simple text format. json only works with serial/single-rank jobs. By default, the first available backend in the order given above is taken.

<element_name>.encoding (string, default value: g)

openPMD iteration encoding: (v)ariable based, (f)ile based, (g)roup based (default) variable based is an experimental feature with ADIOS2.

<element_name>.nonlinear_lens_invariants (boolean, default value: false)

Compute and output the invariants H and I within the nonlinear magnetic insert element (see: nonlinear_lens). Invariants associated with the nonlinear magnetic insert described by V. Danilov and S. Nagaitsev, PRSTAB 13, 084002 (2010), Sect. V.A.

<element_name>.alpha (float, unitless) Twiss alpha of the bare linear lattice at the location of output for the nonlinear IOTA invariants H and I. Horizontal and vertical values must be equal.

<element_name>.beta (float, meters) Twiss beta of the bare linear lattice at the location of output for the nonlinear IOTA invariants H and I. Horizontal and vertical values must be equal.

<element_name>.tn (float, unitless) dimensionless strength of the IOTA nonlinear magnetic insert element used for computing H and I.

<element_name>.cn (float, meters^(1/2)) scale factor of the IOTA nonlinear magnetic insert element used for computing H and I.

line a sub-lattice (line) of elements to append to the lattice.

<element_name>.elements (list of strings) optional (default: no elements) A list of names (one name per lattice element), in the order that they appear in the lattice.

<element_name>.reverse (boolean) optional (default: false) Reverse the list of elements in the line before appending to the lattice.

<element_name>.repeat (integer) optional (default: 1) Repeat the line multiple times before appending to the lattice. Note: If reverse and repeat both appear, then reverse is applied before repeat.

Distribution across MPI ranks and parallelization

amr.max_grid_size (integer) optional (default: 128)
Maximum allowable size of each subdomain (expressed in number of grid points, in each direction). Each subdomain has its own ghost cells, and can be handled by a different MPI rank ; several OpenMP threads can work simultaneously on the same subdomain.

If max_grid_size is such that the total number of subdomains is larger that the number of MPI ranks used, than some MPI ranks will handle several subdomains, thereby providing additional flexibility for load balancing.

When using mesh refinement, this number applies to the subdomains of the coarsest level, but also to any of the finer level.

Math parser and user-defined constants

ImpactX uses AMReX’s math parser that reads expressions in the input file. It can be used in all input parameters that consist of one or more integers or floats. Integer input expecting boolean, 0 or 1, are not parsed. Note that when multiple values are expected, the expressions are space delimited. For integer input values, the expressions are evaluated as real numbers and the final result rounded to the nearest integer. See this section of the AMReX documentation for a complete list of functions supported by the math parser.

ImpactX constants

ImpactX will provide a few pre-defined constants, that can be used for any parameter that consists of one or more floats.

Note

Develop, such as:

q_e	elementary charge
m_e	electron mass
m_p	proton mass
m_u	unified atomic mass unit (Dalton)
epsilon0	vacuum permittivity
mu0	vacuum permeability
clight	speed of light
pi	math constant pi

See in WarpX the file Source/Utils/WarpXConst.H for the values.

User-defined constants

Users can define their own constants in the input file. These constants can be used for any parameter that consists of one or more integers or floats. User-defined constant names can contain only letters, numbers and the character _. The name of each constant has to begin with a letter. The following names are used by ImpactX, and cannot be used as user-defined constants: x, y, z, X, Y, t. The values of the constants can include the predefined ImpactX constants listed above as well as other user-defined constants. For example:

my_constants.a0 = 3.0
my_constants.z_plateau = 150.e-6
my_constants.n0 = 1.e22
my_constants.wp = sqrt(n0*q_e**2/(epsilon0*m_e))

Coordinates

Besides, for profiles that depend on spatial coordinates (the plasma momentum distribution or the laser field, see below Particle initialization and Laser initialization), the parser will interpret some variables as spatial coordinates. These are specified in the input parameter, i.e., density_function(x,y,z) and field_function(X,Y,t).

The parser reads python-style expressions between double quotes, for instance "a0*x**2 * (1-y*1.e2) * (x>0)" is a valid expression where a0 is a user-defined constant (see above) and x and y are spatial coordinates. The names are case sensitive. The factor (x>0) is 1 where x>0 and 0 where x<=0. It allows the user to define functions by intervals. Alternatively the expression above can be written as if(x>0, a0*x**2 * (1-y*1.e2), 0).

Numerics and algorithms

algo.particle_shape (integer; 1, 2, or 3)
The order of the shape factors (splines) for the macro-particles along all spatial directions: 1 for linear, 2 for quadratic, 3 for cubic. Low-order shape factors result in faster simulations, but may lead to more noisy results. High-order shape factors are computationally more expensive, but may increase the overall accuracy of the results. For production runs it is generally safer to use high-order shape factors, such as cubic order.
algo.space_charge (boolean, optional, default: false)
Whether to calculate space charge effects.
algo.mlmg_relative_tolerance (float, optional, default: 1.e-7)
The relative precision with which the electrostatic space-charge fields should be calculated. More specifically, the space-charge fields are computed with an iterative Multi-Level Multi-Grid (MLMG) solver. This solver can fail to reach the default precision within a reasonable time.
algo.mlmg_absolute_tolerance (float, optional, default: 0, which means: ignored)
The absolute tolerance with which the space-charge fields should be calculated in units of V/m^2. More specifically, the acceptable residual with which the solution can be considered converged. In general this should be left as the default, but in cases where the simulation state changes very little between steps it can occur that the initial guess for the MLMG solver is so close to the converged value that it fails to improve that solution sufficiently to reach the mlmg_relative_tolerance value.”
algo.mlmg_max_iters (integer, optional, default: 100)
Maximum number of iterations used for MLMG solver for space-charge fields calculation. In case if MLMG converges but fails to reach the desired self_fields_required_precision, this parameter may be increased.
algo.mlmg_verbosity (integer, optional, default: 1)
The verbosity used for MLMG solver for space-charge fields calculation. Currently MLMG solver looks for verbosity levels from 0-5. A higher number results in more verbose output.

Diagnostics and output

diag.enable (boolean, optional, default: true) Enable or disable diagnostics generally. Disabling this is mostly used for benchmarking.

This option is ignored for the openPMD output elements (remove them from the lattice to disable).
diag.slice_step_diagnostics (boolean, optional, default: false) By default, diagnostics is performed at the beginning and end of the simulation. Enabling this flag will write diagnostics every step and slice step
diag.file_min_digits (integer, optional, default: 6)
The minimum number of digits used for the step number appended to the diagnostic file names.
diag.backend (string, default value: default)

Diagnostics for particles lost in apertures, stored as diags/openPMD/particles_lost.* at the end of the simulation. See the beam_monitor element for backend values.

In-situ visualization

Note

TODO :-)

Note

TODO :-)

Checkpoints and restart

Note

ImpactX will support checkpoints/restart via AMReX. The checkpoint capability can be turned with regular diagnostics: <diag_name>.format = checkpoint.

amr.restart (string)
Name of the checkpoint file to restart from. Returns an error if the folder does not exist or if it is not properly formatted.

Intervals parser

Note

TODO :-)

ImpactX can parse time step interval expressions of the form start:stop:period, e.g. 1:2:3, 4::, 5:6, :, ::10. A comma is used as a separator between groups of intervals, which we call slices. The resulting time steps are the union set of all given slices. White spaces are ignored. A single slice can have 0, 1 or 2 colons :, just as numpy slices, but with inclusive upper bound for stop.

For 0 colon the given value is the period
For 1 colon the given string is of the type start:stop
For 2 colons the given string is of the type start:stop:period

Any value that is not given is set to default. Default is 0 for the start, std::numeric_limits<int>::max() for the stop and 1 for the period. For the 1 and 2 colon syntax, actually having values in the string is optional (this means that ::5, 100 ::10 and 100 : are all valid syntaxes).

All values can be expressions that will be parsed in the same way as other integer input parameters.

Examples

something_intervals = 50 -> do something at timesteps 0, 50, 100, 150, etc. (equivalent to something_intervals = ::50)
something_intervals = 300:600:100 -> do something at timesteps 300, 400, 500 and 600.
something_intervals = 300::50 -> do something at timesteps 300, 350, 400, 450, etc.
something_intervals = 105:108,205:208 -> do something at timesteps 105, 106, 107, 108, 205, 206, 207 and 208. (equivalent to something_intervals = 105 : 108 : , 205 : 208 :)
something_intervals = : or something_intervals = :: -> do something at every timestep.
something_intervals = 167:167,253:253,275:425:50 do something at timesteps 167, 253, 275, 325, 375 and 425.

This is essentially the python slicing syntax except that the stop is inclusive (0:100 contains 100) and that no colon means that the given value is the period.

Note that if a given period is zero or negative, the corresponding slice is disregarded. For example, something_intervals = -1 deactivates something and something_intervals = ::-1,100:1000:25 is equivalent to something_intervals = 100:1000:25.

Workflows

This section collects typical user workflows and best practices for ImpactX.

Note

TODO: Add more workflows as in https://warpx.readthedocs.io/en/latest/usage/workflows.html

Data Analysis

Beam Monitor

ImpactX provides a zero-sized beam monitor element that can be placed in lattices to output the particle beam at multiple positions in a lattice. Output is written in the standardized, open particle-mesh data schema (openPMD) and is compatible with many codes and data analysis frameworks.

For data analysis of openPMD data, see examples of many supported tools, Python libraries and frameworks. Exporting data to ASCII is possible, too.

Additional Beam Attributes

We add the following additional attributes on the openPMD beam species at the monitor position.

Reference particle:

beta_ref reference particle normalized velocity $\beta = v/c$
gamma_ref reference particle Lorentz factor $\gamma = 1/\sqrt{1-\beta^2}$
s_ref integrated orbit path length, in meters
x_ref horizontal position x, in meters
y_ref vertical position y, in meters
z_ref longitudinal position z, in meters
t_ref clock time * c in meters
px_ref momentum in x, normalized to mass*c, $p_x = \gamma \beta_x$
py_ref momentum in y, normalized to mass*c, $p_y = \gamma \beta_y$
pz_ref momentum in z, normalized to mass*c, $p_z = \gamma \beta_z$
pt_ref energy, normalized by rest energy, $p_t = -\gamma$
mass reference rest mass, in kg
charge reference charge, in C

Example to print the integrated orbit path length s at each beam monitor position:

import openpmd_api as io

series = io.Series("diags/openPMD/monitor.h5", io.Access.read_only)

for k_i, i in series.iterations.items():
    beam = i.particles["beam"]
    s_ref = beam.get_attribute("s_ref")
    print(f"step {k_i:>3}: s_ref={s_ref}")

Reduced Beam Characteristics

ImpactX calculates reduced beam characteristics like averaged positions, momenta, beam emittances and Courant-Snyder (Twiss) parameters during runtime. These quantities are calculated before, after, and during each step of the simulation. If diag.slice_step_diagnostics is enabled, they will also be calculated during each slice of each beamline element.

The code writes out the values in an ASCII file prefixed reduced_beam_characteristics containing the follow columns:

step
Iteration within the simulation
s, ref_beta_gamma
Reference particle coordinate s (unit: meter) and relativistic momentum normalized by the particle mass and the speed of light (unit: dimensionless)
x_mean/min/max, y_mean/min/max, t_mean/min/max
Average / minimum / maximum beam particle position in the dimensions of x, y (transverse coordinates, unit: meter), and t (normalized time difference $ct$, unit: meter)
sig_x, sig_y, sig_t
RMS of the average beam particle positions (unit: meter)
px_mean/min/max, py_mean/min/max, pt_mean/min/max
Average / minimum / maximum beam momenta normalized by reference particle momentum (unit: dimensionless, radians for transverse momenta)
sig_px, sig_py, sig_pt
RMS of the average beam momenta (energy difference for pt) (unit: dimensionless)
emittance_x, emittance_y, emittance_t
Normalized beam emittance (unit: meter)
alpha_x, alpha_y, alpha_t
Courant-Snyder (Twiss) alpha (unit: dimensionless)
beta_x, beta_y, beta_t
Courant-Snyder (Twiss) beta (unit: meter)
charge
Cumulated beam charge (unit: Coulomb)

Interactive Analysis

When steering ImpactX from Python, one can at any point visualize the beam phase space with:

import matplotlib.pyplot as plt

from impactx import ImpactX, RefPart, distribution, elements

sim = ImpactX()

# ... setup and simulate ...

pc = sim.particle_container()

fig = pc.plot_phasespace()

# note: figure data available on MPI rank zero
if fig is not None:
    fig.savefig("phase_space.png")
    plt.show()

In situ visualization of the beam phase space projections.

Theory

Introduction

Concepts

Reference Trajectory

ImpactX is an s-based beam dynamics code with space charge. Particles are tracked using the longitudinal accelerator lattice position $s$ as the independent dynamical variable. Particle phase space coordinates are specified relative to a nominal reference trajectory.

The reference trajectory plays an important role in ImpactX. In addition to specifying the nominal machine orbit, it specifies the relationship between the local beam coordinate system and the global lab coordinate system. A reference trajectory is favorable instead of, e.g., differences to the beam centroid, because:

the reference trajectory is the ideal single-particle orbit used as part of the optics design, so it is better that it can be computed independently of the beam distribution,
differences between the beam centroid and the reference particle are important when investigating the effects of misalignments and errors for beamline designs,
the fields are usually specified relative to the reference trajectory, and the dynamics becomes more nonlinear as one moves away from the reference trajectory, so knowing how far the beam particles are form the reference trajectory is important for accuracy,
we want to keep track of the reference trajectory in global coordinates, and that global information is not available in the particle data alone, so it would need to be specified in some other way.

The reference values of $z=ct$ and $s$ should be identical until reaching a bending element. (If the lattice contains no bending elements, then they should coincide.) Also, the reference value of $ct$ coincides with the value of $s$ in the ultrarelativistic limit. More generally, the derivative $ds/d(ct) = \beta$, where the relativistic $\beta = \sqrt{1-\frac{1}{p_t^2}}$.

Collective Effects

Collective effects from space charge of the particle beam are solved between steps in $s$. One can set the number of slice steps through each lattice element for the application of the space charge push.

Note

Currently, space charge kicks are calculated in 3D. A 2D space-charge solver for purely transversal effects will be added in the future.

Coordinates and Units

Each particle in the beam is described at fixed $s$ by a set of 6 canonical phase space variables (x [m], px, y [m], py, t [m], pt). Coordinates x and y denote the horizontal and vertical displacement from the reference particle, respectively, and describe motion in the plane transverse to the velocity of the reference particle. The longitudinal coordinate t denotes the difference between the arrival time of the particle and the arrival time of the reference particle, multiplied by the speed of light $c$.

The momenta conjugate to x, y, and t are denoted px, py, and pt, respectively. These variables are normalized by the magnitude of the momentum of the reference particle, and are therefore dimensionless. In a region of zero vector potential, for example, $p_x = \Delta(\beta_x\gamma)/(\beta_0\gamma_0)$, where $\beta_0$ and $\gamma_0$ denote the relativistic factors associated with the reference velocity. In a region of zero scalar potential, pt denotes the deviation from the reference energy normalized by the design momentum times the speed of light, so that $p_t = \Delta(\gamma)/(\beta_0\gamma_0)$.

Unlike particles within the beam, the reference particle is described by a set of 8 phase space variables (x [m], px, y [m], py, z [m], pz, t [m], pt) that are specified in a global laboratory coordinate system (x,y,z). The momenta of the reference particle are normalized by $mc$, so that $p_x=\beta_x\gamma$, etc. A parameteric plot of the reference trajectory variables (x,z) allows the user to view the global geometry of the accelerator structure (footprint).

Beam Distribution Input

Phase space ellipse in the coordinate plane of position $q$ (realized as $x$, $y$, and $t$) and associated conjugate momentum $q$ (realized as $p_x$, $p_y$ and $p_t$).

Particle beam user input in ImpactX can be done in two ways.

The first option is to characterize the distribution via the intersections of the phase space ellipse with the coordinate axes and the correlation terms of the canonical coordinate pairs.

\[\begin{split}\begin{align} \lambda_q &= \sqrt{\frac{\epsilon}{\gamma}} \\ \lambda_p &= \sqrt{\frac{\epsilon}{\beta}} \\ \mu_{qp} &= \frac{\alpha}{\sqrt{\beta \gamma}} \end{align}\end{split}\]

The units are $[\lambda_q] = \mathrm{m}$, $[\lambda_p] = \mathrm{rad}$, and $[\mu_{qp}] = 1$. To convert between normalized and unnormalized emittance, use the relation $\epsilon_\mathrm{n} = (\beta\gamma)_\mathrm{ref} \cdot \epsilon$ which uses the momentum of the reference particle. Attention: Here, $(\beta\gamma)_\mathrm{ref}$ are the Lorentz variables for the reference particle momentum and not the Courant-Snyder parameters.

The second option is to specify the distribution via the Courant-Snyder / Twiss parameters $\alpha$ and $\beta$ along with the unnormalized (geometric, 1-RMS) emittance $\epsilon$ for all the spatial coordinates. Recall the Courant-Snyder relation $\gamma\beta - \alpha^2 = 1$ for conversion from $\gamma$ values to our input conventions.

Distribution Sampling and the Covariance Matrix

In ImpactX, beam sampling is performed under the assumption that the initial beam distribution centroid (mean phase space vector) coincides with the phase space origin. The covariance matrix $\Sigma$ is defined by $\Sigma_{ij}=\langle{\zeta_i\zeta_j\rangle}$, where $\zeta$ denotes the vector of phase space coordinates, and indices $i,j$ specify the components of $\zeta$.

Let $P$ denote a phase space probability density with unit covariance matrix (i.e., equal to the identity matrix). To produce a phase space density with a target covariance matrix $\Sigma$, we write $\Sigma$ in terms of its (lower) Cholesky decomposition as:

\[\begin{equation} \Sigma = LL^T, \end{equation}\]

where $L$ is a lower triangular matrix.

Define a beam distribution function $f$ by:

\[\begin{equation} f(\zeta)=\kappa P(L^{-1}\zeta),\quad\text{where}\quad \kappa=|\det L|^{-1}. \end{equation}\]

Then $f$ has the desired covariance matrix $\Sigma$. Samples from $f$ are obtained by sampling from $P$ and performing the linear transformation $\zeta\mapsto L\zeta$.

Let $P$ above denote a 2D probability distribution that is radially symmetric, in the sense that:

\[\begin{equation} P(\zeta)=G(||\zeta||^2)=G(q^2+p^2),\quad\quad \zeta=(q,p) \end{equation}\]

Here $q$ denotes a position coordinate (e.g., $x$, $y$, or $t$) and $p$ denotes the corresponding conjugate momentum.

Then the resulting distribution $f$ has 2D elliptical symmetry, in the sense that:

\[\begin{equation} f(\zeta)\propto P(L^{-1}\zeta)=G(||L^{-1}\zeta||^2)=G(\zeta^TS\zeta),\quad\quad S=\Sigma^{-1}. \end{equation}\]

The argument of $G$ is a quadratic form in $(q,p)$, and it is convenient to express this quadratic form as:

\[\begin{equation} \zeta^TS\zeta = \frac{q^2}{\lambda_q^2} + 2\mu_{qp}\frac{qp}{\lambda_q\lambda_p}+\frac{p^2}{\lambda_p^2}=\frac{1}{\epsilon}\left(\gamma q^2+2\alpha qp + \beta p^2\right). \end{equation}\]

Here $\alpha$, $\beta$, and $\gamma$ denote the Courant-Snyder Twiss functions, and $\epsilon$ denotes the rms (unnormalized) emittance.

The associated covariance matrix may be written explicitly in terms of the above parameters as:

\[\begin{split}\begin{equation} \begin{pmatrix} \lambda_q & 0 \\ 0 & \lambda_p \end{pmatrix} \begin{pmatrix} 1 & \mu_{qp} \\ \mu_{qp} & 1 \end{pmatrix}^{-1} \begin{pmatrix} \lambda_q & 0 \\ 0 & \lambda_p \end{pmatrix} = \epsilon \begin{pmatrix} \beta & -\alpha \\ -\alpha & \gamma \end{pmatrix}. \end{equation}\end{split}\]

Note: In the special case that $\mu_{qp}=0$, we have $\lambda_q=\sigma_q$ and $\lambda_p=\sigma_p$, where $\sigma_q=\langle{q^2\rangle}^{1/2}$ and $\sigma_p=\langle{p^2\rangle}^{1/2}$.

Assumptions

This is an overview of physical assumptions implemented in the numerics of ImpactX.

Tracking and Lattice Optics

Tracking through lattice optics in ImpactX is performed by updating the canonical phase space variables (x,px,y,py,t,pt) using symplectic transport. The elements supported currently fall into one of the following categories:

zero-length (thin) elements, such as multipole kicks and coordinate transformations
ideal (thick) elements using a hard-edge fringe field approximation, such as drifts, quadrupoles, and dipoles
soft-edge elements described by $s$-dependent, user-provided field data, such as RF cavities
ML surrogate models using a trained neural network (not necessarily symplectic)

Transport may be performed using one of three possible levels of approximation to the underlying Hamiltonian:

linear transfer map (default): obtained by expanding the Hamiltonian through terms of degree 2 in the deviation of phase space variables from those of the reference particle
chromatic or paraxial approximation: obtained by expanding the Hamiltonian through terms of degree 2 in the transverse phase space variables, while retaining the nonlinear dependence on the energy variable pt
exact Hamiltonian: obtained using the exact nonlinear Hamiltonian

Space Charge (Poisson Solver)

velocity spread: when solving for space-charge effects, we assume that the relative spread of velocities of particles in the beam is negligible compared to the velocity of the reference particle, so that in the bunch frame (rest frame of the reference particle) particle velocities are nonrelativistic

electrostatic in the bunch frame: we assume there are no retardation effects and we solve the Poisson equation in the bunch frame

Development

Contribute to ImpactX

We welcome new contributors! Here is how to participate to the ImpactX development.

Git workflow

The ImpactX project uses git for version control. If you are new to git, you can follow one of these tutorials:

Configure your GitHub Account & Development Machine

First, let’s setup your Git environment and GitHub account.

Go to https://github.com/settings/profile and add your real name and affiliation
Go to https://github.com/settings/emails and add & verify the professional e-mails you want to be associated with.
Configure git on the machine you develop on to use the same spelling of your name and email:
- git config --global user.name "FIRSTNAME LASTNAME"
- git config --global user.email EMAIL@EXAMPLE.com
Go to https://github.com/settings/keys and add the SSH public key of the machine you develop on. (Check out the GitHub guide to generating SSH keys or troubleshoot common SSH problems. )

Make your own fork

First, fork the ImpactX “mainline” repo on GitHub by pressing the Fork button on the top right of the page. A fork is a copy of ImpactX on GitHub, which is under your full control.

Then, we create local copies, for development:

# Clone the mainline ImpactX source code to your local computer.
# You cannot write to this repository, but you can read from it.
git clone git@github.com:ECP-WarpX/impactx.git
cd impactx

# rename what we just cloned: call it "mainline"
git remote rename origin mainline

# Add your own fork. You can get this address on your fork's Github page.
# Here is where you will publish new developments, so that they can be
# reviewed and integrated into "mainline" later on.
# "myGithubUsername" needs to be replaced with your user name on GitHub.
git remote add myGithubUsername git@github.com:myGithubUsername/impactx.git

Now you are free to play with your fork (for additional information, you can visit the Github fork help page).

Note

We only need to do the above steps for the first time.

Let’s Develop

You are all set! Now, the basic ImpactX development workflow is:

Implement your changes and push them on a new branch branch_name on your fork.
Create a Pull Request from branch branch_name on your fork to branch development on the main ImpactX repo.

Create a branch branch_name (the branch name should reflect the piece of code you want to add, like fix-spectral-solver) with

# start from an up-to-date development branch
git checkout development
git pull mainline development

# create a fresh branch
git checkout -b branch_name

and do the coding you want.

It is probably a good time to look at the AMReX documentation and at the Doxygen reference pages:

ImpactX Doxygen: https://impactx.readthedocs.io/en/latest/_static/doxyhtml
AMReX Doxygen: https://amrex-codes.github.io/amrex/doxygen
WarpX Doxygen: https://warpx.readthedocs.io/en/latest/_static/doxyhtml

Once you are done developing, add the files you created and/or modified to the git staging area with

git add <file_I_created> <and_file_I_modified>

Build your changes

If you changed C++ files, then now is a good time to test those changes by compiling ImpactX locally. Follow the developer instructions in our manual to set up a local development environment, then compile and run ImpactX.

Commit & push your changes

Periodically commit your changes with

git commit

The commit message (between quotation marks) is super important in order to follow the developments during code-review and identify bugs. A typical format is:

This is a short, 40-character title

After a newline, you can write arbitray paragraphs. You
usually limit the lines to 70 characters, but if you don't, then
nothing bad will happen.

The most important part is really that you find a descriptive title
and add an empty newline after it.

For the moment, commits are on your local repo only. You can push them to your fork with

git push -u myGithubUsername branch_name

If you want to synchronize your branch with the development branch (this is useful when the development branch is being modified while you are working on branch_name), you can use

git pull mainline development

and fix any conflict that may occur.

Submit a Pull Request

A Pull Request (PR) is the way to efficiently visualize the changes you made and to propose your new feature/improvement/fix to the ImpactX project. Right after you push changes, a banner should appear on the Github page of your fork, with your branch_name.

Click on the compare & pull request button to prepare your PR.
It is time to communicate your changes: write a title and a description for your PR. People who review your PR are happy to know
- what feature/fix you propose, and why
- how you made it (added new/edited files, created a new class than inherits from…)
- how you tested it and what was the output you got
- and anything else relevant to your PR (attach images and scripts, link papers, etc.)
Press Create pull request. Now you can navigate through your PR, which highlights the changes you made.

Please DO NOT write large pull requests, as they are very difficult and time-consuming to review. As much as possible, split them into small, targeted PRs. For example, if find typos in the documentation open a pull request that only fixes typos. If you want to fix a bug, make a small pull request that only fixes a bug.

If you want to implement a feature and are not too sure how to split it, just open an issue about your plans and ping other ImpactX developers on it to chime in. Generally, write helper functionality first, test it and then write implementation code. Submit tests, documentation changes and implementation of a feature together for pull request review.

Even before your work is ready to merge, it can be convenient to create a PR (so you can use Github tools to visualize your changes). In this case, please put the [WIP] tag (for Work-In-Progress) at the beginning of the PR title. You can also use the GitHub project tab in your fork to organize the work into separate tasks/PRs and share it with the ImpactX community to get feedback.

Include a test to your PR

A new feature is great, a working new feature is even better! Please test your code and add your test to the automated test suite. It’s the way to protect your work from adventurous developers.

Note

TOOD: Write a workflow how to add a test.

Include documentation about your PR

Now, let users know about your new feature by describing its usage in the ImpactX documentation. Our documentation uses Sphinx, and it is located in docs/source/.

Note

TODO: For instance, if you introduce a new runtime parameter in the input file, you can add it to Docs/source/running_cpp/parameters.rst.

If Sphinx is installed on your computer, you should be able to generate the html documentation with

make html

in docs/. Then open docs/build/html/index.html with your favorite web browser and look for your changes.

Once your code is ready with documentation and automated test, congratulations! You can create the PR (or remove the [WIP] tag if you already created it). Reviewers will interact with you if they have comments/questions.

Style and conventions

For indentation, ImpactX uses four spaces (no tabs)
Some text editors automatically modify the files you open. We recommend to turn on to remove trailing spaces and replace Tabs with 4 spaces.
The number of characters per line should be <100
Exception: in documentation files (.rst/.md) use one sentence per line independent of its number of characters, which will allow easier edits.
Space before and after assignment operator (=)
To define a function , for e.g., myfunction() use a space between the name of the function and the paranthesis - myfunction (). To call the function, the space is not required, i.e., just use myfunction().
The reason this is beneficial is that when we do a git grep to search for myfunction (), we can clearly see the locations where myfunction () is defined and where myfunction() is called.
Also, using git grep "myfunction ()" searches for files only in the git repo, which is more efficient compared to the grep "myfunction ()" command that searches through all the files in a directory, including plotfiles for example.
It is recommended that style changes are not included in the PR where new code is added. This is to avoid any errors that may be introduced in a PR just to do style change.
ImpactX uses CamelCase convention for file names and class names, rather than snake_case.
The names of all member variables should be prefixed with m_. This is particularly useful to avoid capturing member variables by value in a lambda function, which causes the whole object to be copied to GPU when running on a GPU-accelerated architecture. This convention should be used for all new piece of code, and it should be applied progressively to old code.
#include directives in C++ have a distinct order to avoid bugs, see the ImpactX repo structure for details
For all new code, we should avoid relying on using namespace amrex; and all amrex types should be prefixed with amrex::. Inside limited scopes, AMReX type literals can be included with using namespace amrex::literals;. Ideally, old code should be modified accordingly.

Testing

Preparation

Prepare for running tests of ImpactX by building ImpactX from source.

In order to run our tests, you need to have a few Python packages installed:

python3 -m pip install -U pip
python3 -m pip install -U build packaging setuptools wheel pytest
python3 -m pip install -r examples/requirements.txt

Run

You can run all our tests with:

ctest --test-dir build --output-on-failure

Further Options

help: ctest --test-dir build --help
list all tests: ctest --test-dir build -N
only run tests that have “FODO” in their name: ctest --test-dir build -R FODO

Documentation

Doxygen documentation

WarpX uses a Doxygen documentation. Whenever you create a new class, please document it where it is declared (typically in the header file):

/** A brief title
 *
 * few-line description explaining the purpose of my_class.
 *
 * If you are kind enough, also quickly explain how things in my_class work.
 * (typically a few more lines)
 */
class my_class
{ ... }

Doxygen reads this docstring, so please be accurate with the syntax! See Doxygen manual for more information. Similarly, please document functions when you declare them (typically in a header file) like:

/** A brief title
 *
 * few-line description explaining the purpose of my_function.
 *
 * \param[in,out] my_int a pointer to an integer variable on which
 *                       my_function will operate.
 * \return what is the meaning and value range of the returned value
 */
int my_class::my_function(int* my_int);

An online version of this documentation is linked here.

Breathe documentation

Your Doxygen documentation is not only useful for people looking into the code, it is also part of the ImpactX online documentation based on Sphinx! This is done using the Python module Breathe, that allows you to read Doxygen documentation dorectly in the source and include it in your Sphinx documentation, by calling Breathe functions. For instance, the following line will get the Doxygen documentation for ImpactXParticleContainer in src/particles/ImpactXParticleContainer.H and include it to the html page generated by Sphinx:

class ImpactXParticleContainer : public amrex::ParticleContainer_impl<RealSoA::nattribs, IntSoA::nattribs>

Beam Particles in ImpactX

This class stores particles, distributed over MPI ranks.

Building the documentation

To build the documentation on your local computer, you will need to install Doxygen as well as the Python module breathe. First, change into docs/ and install the Python requirements:

cd docs/
pip install -U -r requirements.txt

You will also need Doxygen (macOS: brew install doxygen; Ubuntu: sudo apt install doxygen).

Then, to compile the documentation, use

make html
# This will first compile the Doxygen documentation (execute doxygen)
# and then build html pages from rst files using sphinx and breathe.

Open the created build/html/index.html file with your favorite browser. Rebuild and refresh as needed.

ImpactX Structure

Repo Organization

All the ImpactX source code is located in src/. All sub-directories have a pretty straightforward name.

Here is a visual representation of the repository structure.

Code organization

The main ImpactX class is ImpactX, implemented in src/ImpactX.cpp.

Build System

ImpactX uses the CMake build system generator. Each sub-folder contains a file CMakeLists.txt with the names of the source files (.cpp) that are added to the build. Do not list header files (.H) here.

C++ Includes

All ImpactX header files need to be specified relative to the src/ directory.

e.g. #include "Utils/ImpactXConst.H"
files in the same directory as the including header-file can be included with #include "FileName.H"

By default, in a MyName.cpp source file we do not include headers already included in MyName.H. Besides this exception, if a function or a class is used in a source file, the header file containing its declaration must be included, unless the inclusion of a facade header is more appropriate. This is sometimes the case for AMReX headers. For instance AMReX_GpuLaunch.H is a façade header for AMReX_GpuLaunchFunctsC.H and AMReX_GpuLaunchFunctsG.H, which contain respectively the CPU and the GPU implemetation of some methods, and which should not be included directly. Whenever possible, forward declarations headers are included instead of the actual headers, in order to save compilation time (see dedicated section below). In ImpactX forward declaration headers have the suffix *_fwd.H, while in AMReX they have the suffix *Fwd.H. The include order (see PR #874 and PR #1947) and proper quotation marks are:

In a MyName.cpp file:

#include "MyName.H" (its header) then
(further) ImpactX header files #include "..." then
ImpactX forward declaration header files #include "..._fwd.H"
AMReX header files #include <...> then
AMReX forward declaration header files #include <...Fwd.H> then
PICSAR header files #include <...> then
other third party includes #include <...> then
standard library includes, e.g. #include <vector>

In a MyName.H file:

#include "MyName_fwd.H" (the corresponding forward declaration header, if it exists) then
ImpactX header files #include "..." then
ImpactX forward declaration header files #include "..._fwd.H"
AMReX header files #include <...> then
AMReX forward declaration header files #include <...Fwd.H> then
PICSAR header files #include <...> then
other third party includes #include <...> then
standard library includes, e.g. #include <vector>

Each of these groups of header files should ideally be sorted alphabetically, and a blank line should be placed between the groups.

For details why this is needed, please see PR #874, PR #1947, the LLVM guidelines, and include-what-you-use.

Forward Declaration Headers

Forward declarations can be used when a header file needs only to know that a given class exists, without any further detail (e.g., when only a pointer to an instance of that class is used). Forward declaration headers are a convenient way to organize forward declarations. If a forward declaration is needed for a given class MyClass, declared in MyClass.H, the forward declaration should appear in a header file named MyClass_fwd.H, placed in the same folder containing MyClass.H. As for regular header files, forward declaration headers must have include guards. Below we provide a simple example:

MyClass_fwd.H:

#ifndef MY_CLASS_FWD_H
#define MY_CLASS_FWD_H

class MyClass;

#endif //MY_CLASS_FWD_H

MyClass.H:

#ifndef MY_CLASS_H
#define MY_CLASS_H

#include "MyClass_fwd.H"
#include "someHeader.H"
class MyClass{/* stuff */};

#endif //MY_CLASS_H

MyClass.cpp:

#include "MyClass.H"
class MyClass{/* stuff */};

Usage: in SimpleUsage.H

#include "MyClass_fwd.H"
#include <memory>

/* stuff */
std::unique_ptr<MyClass> p_my_class;
/* stuff */

Implementation Details

Note

TODO :-)

C++ Objects & Functions

We generate the documentation of C++ objects and functions from our C++ source code by adding Doxygen strings.

Our Doxygen C++ API documentation in classic formatting is located here.

Python interface

Note

TODO :-)

Debugging the code

Sometimes, the code does not give you the result that you are expecting. This can be due to a variety of reasons, from misunderstandings or changes in the input parameters, system specific quirks, or bugs. You might also want to debug your code as you implement new features in ImpactX during development.

This section gives a step-by-step guidance on how to systematically check what might be going wrong.

Debugging Workflow

Try the following steps to debug a simulation:

Check the output text file, usually called output.txt: are there warnings or errors present?
On an HPC system, look for the job output and error files, usually called ImpactX.e... and ImpactX.o.... Read long messages from the top and follow potential guidance.
If your simulation already created output data files: Check if they look reasonable before the problem occurred; are the initial conditions of the simulation as you expected? Do you spot numerical artifacts or instabilities that could point to missing resolution or unexpected/incompatible numerical parameters?
Did the job output files indicate a crash? Check the Backtrace.<mpirank> files for the location of the code that triggered the crash. Backtraces are read from bottom (high-level) to top (most specific line that crashed).
In case of a crash, Backtraces can be more detailed if you re-compile with debug flags: for example, try compiling with -DCMAKE_BUILD_TYPE=RelWithDebInfo (some slowdown) or even -DCMAKE_BUILD_TYPE=Debug (this will make the simulation way slower) and rerun.
If debug builds are too costly, try instead compiling with -DAMReX_ASSERTIONS=ON to activate more checks and rerun.
If the problem looks like a memory violation, this could be from an invalid field or particle index access. Try compiling with -DAMReX_BOUND_CHECK=ON (this will make the simulation very slow), and rerun.
If the problem looks like a random memory might be used, try initializing memory with signaling Not-a-Number (NaN) values through the runtime option fab.init_snan = 1. Further useful runtime options are amrex.fpe_trap_invalid, amrex.fpe_trap_zero and amrex.fpe_trap_overflow (see details in the AMReX link below).
On Nvidia GPUs, if you suspect the problem might be a race condition due to a missing host / device synchronization, set the environment variable export CUDA_LAUNCH_BLOCKING=1 and rerun.
Consider simplifying your input options and re-adding more options after having found a working baseline.

Fore more information, see also the AMReX Debugging Manual.

Last but not least: the community of ImpactX developers and users can help if you get stuck. Collect your above findings, describe where and what you are running and how you installed the code, describe the issue you are seeing with details and input files used and what you already tried. Can you reproduce the problem with a smaller setup (less parallelism and/or less resolution)? Report these details in a ImpactX GitHub issue.

Debuggers

See the AMReX debugger section on additional runtime parameters to

disable backtraces
rethrow exceptions
avoid AMReX-level signal handling

You will need to set those runtime options to work directly with debuggers.

Maintenance

Dependencies & Releases

Update ImpactX’ Core Dependencies

ImpactX has direct dependencies on AMReX and WarpX, which we periodically update.

The following scripts automate this workflow, in case one needs a newer commit of AMReX or WarpX between releases:

Note

./Tools/Release/updateAMReX.py
./Tools/Release/updateWarpX.py

Create a new ImpactX release

ImpactX has one release per month. The version number is set at the beginning of the month and follows the format YY.MM.

In order to create a GitHub release, you need to:

Create a new branch from development and update the version number in all source files. We usually wait for the AMReX release to be tagged first, then we also point to its tag.

There is a script for updating core dependencies of ImpactX and the ImpactX version:
./Tools/Release/updateAMReX.py
./Tools/Release/updateWarpX.py

./Tools/Release/newVersion.sh
For a ImpactX release, ideally a git tag of AMReX & WarpX shall be used instead of an unnamed commit.

Then open a PR, wait for tests to pass and then merge.
Local Commit (Optional): at the moment, @ax3l is managing releases and signs tags (naming: YY.MM) locally with his GPG key before uploading them to GitHub.

Publish: On the GitHub Release page, create a new release via Draft a new release. Either select the locally created tag or create one online (naming: YY.MM) on the merged commit of the PR from step 1.

In the release description, please specify the compatible versions of dependencies (see previous releases), and provide info on the content of the release. In order to get a list of PRs merged since last release, you may run
git log <last-release-tag>.. --format='- %s'
Optional/future: create a release-<version> branch, write a changelog, and backport bug-fixes for a few days.

Epilogue

Glossary

In daily communication, we tend to abbreviate a lot of terms. It is important to us to make it easy to interact with the ImpactX community and thus, this list shall help to clarify often used terms.

Please see: https://warpx.readthedocs.io/en/latest/glossary.html

Funding and Acknowledgements

This work was supported by the Laboratory Directed Research and Development Program of Lawrence Berkeley National Laboratory under U.S. Department of Energy Contract No. DE-AC02-05CH11231.

ImpactX is supported by the CAMPA collaboration, a project of the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research and Office of High Energy Physics, Scientific Discovery through Advanced Computing (SciDAC) program.

We acknowledge all the contributors and users of the ImpactX community who participate to the code quality with valuable code improvement and important feedback.

ImpactX

Contact us

Code of Conduct

Our Pledge

Our Standards

Our Responsibilities

Scope

Enforcement

Attribution

Acknowledge ImpactX

In presentations

In publications

Main ImpactX Reference

Further ImpactX References

Installation

Users

HPC Systems

Using the Conda Package

Using the Spack Package

Using the PyPI Package

Using the Brew Package

From Source with CMake

Tips for macOS Users

Developers

Dependencies

Dependencies

Conda (Linux/macOS/Windows)

Spack (Linux/macOS)

Brew (macOS/Linux)

APT (Debian/Ubuntu Linux)

Compile

Build Options

Configure Your Compiler

Run

HPC

impactx.profile

HPC Systems

Perlmutter (NERSC)

Introduction

Preparation

Compilation

Update ImpactX & Dependencies

Running

Post-Processing

LUMI (CSC)

Introduction

Preparation

Compilation

Update ImpactX & Dependencies

Running

MI250X GPUs (2x64 GB)

Post-Processing

Known System Issues

Usage

Run ImpactX

1. Run Directory

2. Executable

3. Inputs

4. Run

Examples

Single Particle Dynamics

FODO Cell

Run

Analyze

Visualize

Chicane

Run

Analyze

Visualize

Quadrupole with Alignment Errors

Run

Analyze

Drift using a Pole-Face Rotation

Run

Analyze

Acceleration by RF Cavities

Run

Analyze

FODO Cell with RF

Run