How to contribute to ExCALIBUR tests

You are welcome to contribute new application benchmarks and new systems part of the ExCALIBUR benchmarking effort. The easiest way to contribute is to open an issue or a pull request to the repository.

Adding new benchmarks

In particular, adding new benchmarks that are useful to the scientific community is welcome!

Spack package

Before adding a new benchmark, make sure the application is available in Spack. If it is not, you can read the Spack Package Creation Tutorial to contribute a new recipe to build the application.

While we encourage users to contribute all Spack recipes upstream, we have a custom repo for packages not yet ready to be contributed to the main Spack repository. This is in the spack/repo directory, create a subdirectory inside spack/repo/packages with the name of the package you want to add, and place into it the package.py Spack recipe. On supported HPC systems, this repo is automatically added to the provided Spack environments.

In Spack recipes, please avoid cloning the head of a branch. The state of the branch at the time it was cloned will not get recorded by Spack or ReFrame which leads to issues with reproducibility. It is strongly recommended to only build tagged versions of packages with Spack.

ReFrame benchmark

New benchmarks should be added in the apps/ directory, under the specific application subdirectory. Please, add also a README.md file explaining what the application does and how to run the benchmarks in the same directory. Then, link the README.md file under nav: Supported Benchmarks: in the mkdocs documentation config.

For writing ReFrame benchmarks you can read the documentation, in particular

but you can also have a look at the sombrero example.

For GPU benchmarks you need to

set valid_systems = ['+gpu']
set the num_gpus_per_node attribute,
and add the key gpu to the extra_resources dictionary to request the appropriate number of GPUs.

For an example of a GPU benchmark take a look at OpenMM.

Adding new systems

If you configure the framework on a HPC system that is not included in supported systems, please upen a pull request to upload the configuration so that other users can benefit from it. To add a new system, consider the following items