Skip to content

update test suite docs #832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
laraPPr merged 12 commits into from
Aug 19, 2026
Merged

update test suite docs #832

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b08d889
update reframe config for test suite
smoors Aug 5, 2026
46e4e46
link release notes to the repo and update usage
smoors Aug 7, 2026
f1e2fb8
Merge branch 'main' into test-suite-update
smoors Aug 8, 2026
ec2dcc4
update
smoors Aug 11, 2026
f70af5b
update usage
smoors Aug 15, 2026
8eaef21
Update docs/test-suite/usage.md
smoors Aug 19, 2026
d4c1af7
Update docs/test-suite/usage.md
smoors Aug 19, 2026
d8bac8e
Update docs/test-suite/usage.md
smoors Aug 19, 2026
3e5bfd4
Update docs/test-suite/usage.md
smoors Aug 19, 2026
9524b5c
use --list-detailed everywhere
smoors Aug 19, 2026
2f6ba92
fix typos
smoors Aug 19, 2026
4c38f37
typo
smoors Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 46 additions & 110 deletions docs/test-suite/ReFrame-configuration-file.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,93 +15,21 @@ This page is organized as follows:
## Available ReFrame configuration files

There are some available ReFrame configuration files for HPC systems and public cloud in the [config directory](https://github.com/EESSI/test-suite/tree/main/config/) for more inspiration.
Below is a simple ReFrame configuration file with minimal changes required for getting you started on using the test suite for a CPU partition. Please check that `stagedir` is set to a path on a (shared) scratch filesystem for storing (temporary) files related to the tests, and `access` is set to a list of arguments that you would normally pass to the scheduler when submitting to this partition (for example '-p cpu' for submitting to a Slurm partition called cpu).
In the config directory, [`settings_example.py`](https://github.com/EESSI/test-suite/blob/main/config/settings_example.py) is a simple ReFrame configuration file with minimal changes required for getting you started on using the test suite for a CPU partition. Please check that `stagedir` is set to a path on a (shared) scratch filesystem for storing (temporary) files related to the tests, and `access` is set to a list of arguments that you would normally pass to the scheduler when submitting to this partition (for example '-p cpu' for submitting to a Slurm partition called cpu).

To write a ReFrame configuration file for your system, check the section [How to write a ReFrame configuration file](#write-reframe-config).


```python
"""
simple ReFrame configuration file
"""
import os

from eessi.testsuite.common_config import common_logging_config, common_eessi_init, format_perfvars, perflog_format
from eessi.testsuite.constants import *

site_configuration = {
'systems': [
{
'name': 'cpu_partition',
'descr': 'CPU partition',
'modules_system': 'lmod',
'hostnames': ['*'],
# Note that the stagedir should be a shared directory available on all nodes running ReFrame tests
'stagedir': f'/some/shared/dir/{os.environ.get("USER")}/reframe_output/staging',
'partitions': [
{
'name': 'cpu_partition',
'descr': 'CPU partition',
'scheduler': 'slurm',
'launcher': 'mpirun',
'access': ['-p cpu', '--export=None'],
'prepare_cmds': ['source %s' % common_eessi_init()],
'environs': ['default'],
'max_jobs': 4,
'resources': [
{
'name': 'memory',
'options': ['--mem={size}'],
}
],
'features': [
FEATURES.CPU
] + list(SCALES.keys()),
}
]
},
],
'environments': [
{
'name': 'default',
'cc': 'cc',
'cxx': '',
'ftn': '',
},
],
'logging': common_logging_config(),
'general': [
{
# Enable automatic detection of CPU architecture for each partition
# See https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.processor
'remote_detect': True,
}
],
}

# optional logging to syslog
site_configuration['logging'][0]['handlers_perflog'].append({
'type': 'syslog',
'address': '/dev/log',
'level': 'info',
'format': f'reframe: {perflog_format}',
'format_perfvars': format_perfvars,
'append': True,
})
```


## Verifying your ReFrame configuration

To verify the ReFrame configuration, you can [query the configuration using `--show-config`](https://reframe-hpc.readthedocs.io/en/stable/configure.html#querying-configuration-options).
To verify the ReFrame configuration, you can [query the configuration](https://reframe-hpc.readthedocs.io/en/stable/configure.html#querying-configuration-options) using `--show-config`.

To see the full configuration, use:

```bash
reframe --show-config
```

To only show the configuration of a particular system partition, you can use the [`--system` option](https://reframe-hpc.readthedocs.io/en/stable/manpage.html#cmdoption-system).
To only show the configuration of a particular system partition, you can use the [`--system`](https://reframe-hpc.readthedocs.io/en/stable/manpage.html#cmdoption-system) option.
To query a specific setting, you can pass an argument to `--show-config`.

For example, to show the configuration of the `gpu` partition of the `example` system:
Expand Down Expand Up @@ -135,8 +63,9 @@ The EESSI test suite standardizes a few string-based values as constants, as wel
Every ReFrame configuration file used for running the EESSI test suite should therefore start with the following import statements:

```python
from eessi.testsuite.common_config import common_logging_config, common_eessi_init
from eessi.testsuite.constants import *
from eessi.testsuite.common_config import (common_eessi_init, common_general_config, common_logging_config,
set_common_required_config)
from eessi.testsuite.constants import EXTRAS, FEATURES, SCALES, DEVICE_TYPES, GPU_VENDORS
```

### High-level system info (`systems`)
Expand Down Expand Up @@ -209,10 +138,18 @@ site_configuration = {
'name': 'cpu_partition',
'descr': 'CPU partition'
'scheduler': 'slurm',
'prepare_cmds': ['source %s' % common_eessi_init()],
'prepare_cmds': [
# Pass job environment variables like $PATH, etc., into job steps
'export SLURM_EXPORT_ENV=ALL',
# If your system doesn't have an Lmod installation by default on the batch nodes,
# uncommenting the following two lines will use one from EESSI.
# 'source /cvmfs/software.eessi.io/2025.06/init/lmod/bash',
# 'module unload EESSI',
],
'launcher': 'mpirun',
'access': ['-p cpu'],
'environs': ['default'],
# Optionally specify additional (local) environments here
# 'environs': ['local_environ'],
'max_jobs': 4,
'features': [
FEATURES.CPU
Expand All @@ -222,17 +159,19 @@ site_configuration = {
'name': 'gpu_partition',
'descr': 'GPU partition'
'scheduler': 'slurm',
'prepare_cmds': ['source %s' % common_eessi_init()],
'prepare_cmds': [
# Pass job environment variables like $PATH, etc., into job steps
'export SLURM_EXPORT_ENV=ALL',
# If your system doesn't have an Lmod installation by default on the batch nodes,
# Uncommenting the following two lines will use one from EESSI.
# 'source /cvmfs/software.eessi.io/2025.06/init/lmod/bash',
# 'module unload EESSI',
],
'launcher': 'mpirun',
'access': ['-p gpu'],
'environs': ['default'],
# Optionally specify additional (local) environments here
# 'environs': ['local_environ'],
Comment thread
smoors marked this conversation as resolved.
'max_jobs': 4,
'resources': [
{
'name': '_rfm_gpu',
'options': ['--gpus-per-node={num_gpus_per_node}'],
}
],
Comment thread
smoors marked this conversation as resolved.
'devices': [
{
'type': DEVICE_TYPES.GPU,
Expand Down Expand Up @@ -272,30 +211,17 @@ The most common configuration items defined at this level are:
(for example by using '`--export=None`' with Slurm). This avoids test failures due to environment variables set
in the submission environment that are passed down to submitted jobs.
- [`prepare_cmds`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.prepare_cmds):
Commands to execute at the start of every job that runs a test. If your batch scheduler does not export
the environment of the submit host, this is typically where you can initialize the EESSI environment.
Commands to execute at the start of every job that runs a test, such as making the EESSI-installed version of Lmod available.
- [`environs`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.environs):
The names of the *programming environments* (to be defined later in the configuration file via [`environments`](#environments))
that may be used on this partition. A programming environment is required for tests that are compiled first,
before they can run. The EESSI test suite however only tests existing software installations, so no compilation
(or specific programming environment) is needed. Simply specify `'environs': ['default']`,
since ReFrame requires that *a* default environment is defined.
The names of optional additional environments (to be defined later in the configuration file via [`environments`](#environments))
ReFrame requires each test to run in a specific environment. The EESSI test suite automatically adds the available
EESSI modules as environments (e.g. `EESSI-2025.6`) to each partition. We can also add custom environments (e.g. a local
environment for testing a locally installed software stack).
- [`max_jobs`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.max_jobs):
The maximum amount of jobs ReFrame is allowed to submit in parallel. Some batch systems limit how many jobs users
are allowed to have in the queue. You can use this to make sure ReFrame doesn't exceed that limit.
- [`resources`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#custom-job-scheduler-resources):
This field defines how additional resources can be requested in a batch job. Specifically, on a GPU partition,
you have to define a resource with the name '`_rfm_gpu`'. The `options` field should then contain the argument to be
passed to the batch scheduler in order to request a certain number of GPUs _per node_, which could be different for
different batch schedulers. For example, when using Slurm you would specify:
```python
'resources': [
{
'name': '_rfm_gpu',
'options': ['--gpus-per-node={num_gpus_per_node}'],
},
],
```
This field should *NOT* be defined, as it is already defined by the `set_common_required_config` function of the EESSI test suite, see below.
- [`processor`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.processor):
We recommend to *NOT* define this field, unless [CPU autodetection](#cpu-auto-detection) is not working for you.
The EESSI test suite relies on information about your processor topology to run. Using CPU autodetection is the
Expand All @@ -309,7 +235,7 @@ The most common configuration items defined at this level are:
'num_sockets': 2, # Number of sockets in a node
'num_cpus_per_socket': 32, # Number of CPU cores per socket
'num_cpus_per_core': 1, # Number of hardware threads per CPU core
}
}
```
- [`features`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.features):
The `features` field is used by the EESSI test suite to run tests _only_ on a partition if it supports a certain
Expand Down Expand Up @@ -343,10 +269,11 @@ The most common configuration items defined at this level are:
}
```
- [`extras`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.extras): This field specifies extra information on the partition, such as the GPU vendor. Valid fields for `extras` are standardized as constants in [`eessi.testsuite.constants`](https://github.com/EESSI/test-suite/blob/main/eessi/testsuite/constants.py) (for example `EXTRAS.GPU_VENDOR`). This is used by the EESSI test suite to decide if a partition can run a test that _specifically_ requires a certain brand of GPU.
Typically, there is no need to define `extras` for CPU partitions.
`EXTRAS.MEM_PER_NODE` is a required field for `extras`. It specifies the maximum amount of memory per node that can be requested for any test job on the partition.
For GPU partitions, you typically want to specify the GPU vendor, for example:
```python
'extras': {
EXTRAS.MEM_PER_NODE: 229376 # in MiB
EXTRAS.GPU_VENDOR: GPU_VENDORS.NVIDIA
}
```
Expand All @@ -364,14 +291,14 @@ Note that as more tests are added to the EESSI test suite, the use of `features`

### Environments { #environments }

ReFrame needs a programming environment to be defined in its configuration file for tests that need to be compiled before they are run. While we don't have such tests in the EESSI test suite, ReFrame requires _some_ programming environment to be defined:
ReFrame needs a programming environment to be defined in its configuration file for tests that need to be compiled before they are run. While we currently don't have such tests in the EESSI test suite, ReFrame requires _some_ programming environment to be defined. The EESSI modules are already added by default as environments for testing the EESSI stack. If additional `environs` are added to any partition, they must be defined in the `environments` section, for example:

```python
site_configuration = {
...
'environments': [
{
'name': 'default', # Note: needs to match whatever we set for 'environs' in the partition
'name': 'local_environ', # Note: needs to be listed in the partition's `environs` section
'cc': 'cc',
'cxx': '',
'ftn': '',
Expand Down Expand Up @@ -405,6 +332,15 @@ Alternatively, a prefix can be passed as an argument like `common_logging_config
the regular ReFrame log ends up. Note that the performance logs do *not* respect this prefix: they will still end up
in the standard ReFrame prefix (by default the current directory, unless otherwise set with `$RFM_PREFIX` or `--prefix`).

### Common required configuration

After defining the `site_configuration` dictionary, we run the `set_common_required_config` function to update the
dictionary with required configuration options:

```python
set_common_required_config(site_configuration)
```

### Auto-detection of processor information { #cpu-auto-detection }

You can let ReFrame [auto-detect the processor information](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.processor) for your system.
Expand Down
3 changes: 3 additions & 0 deletions docs/test-suite/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ The [EESSI test suite](https://github.com/EESSI/test-suite) is a collection of t
It is used to check whether the software installations included in the [EESSI software layer](../software_layer.md)
are working and performing as expected.

In addition to the EESSI software stack, also it works with other (e.g.locally
installed) software stacks, provided they use the [EasyBuild module naming scheme](https://docs.easybuild.io/api/easybuild/tools/module_naming_scheme/easybuild_mns/).

To get started, you should look into the [installation and configuration guidelines](installation-configuration.md) first.

To write the ReFrame configuration file for your system, check [ReFrame configuration file](ReFrame-configuration-file.md).
Expand Down
32 changes: 9 additions & 23 deletions docs/test-suite/installation-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,29 @@ This page covers the requirements, installation and configuration of the [EESSI
The EESSI test suite requires

* Python >= 3.7
* [ReFrame](https://reframe-hpc.readthedocs.io) v4.3.3 (or newer)
* [ReFrame test library (`hpctestlib`)](https://reframe-hpc.readthedocs.io/en/stable/hpctestlib.html)
* [ReFrame](https://reframe-hpc.readthedocs.io) v4.3.3 or newer (ReFrame v4.10 requires Python >= 3.10)
* (optionally) [EasyBuild](https://easybuild.io/)

??? note "(If your system Python version is lower than the minimum required version, click here for some tips)"

* You can upgrade and manage python versions using `pyenv`. [Link](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation)
* You can upgrade and manage Python versions using [`pyenv`](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation).
* For installation follow the documentation in the repository using the link above.
* Then follow the steps to install a new Python version and further change local paths to the new Python:

pyenv install <python version number>
pyenv local <python version number>

* You can upgrade and manage Python versions using [`uv`](https://docs.astral.sh/uv/guides/install-python/).
* You can install a more recent version of Python on top of the GCC/GCCcore compiler.
* You can install a ReFrame module with EasyBuild and a [ReFrame easyconfig](https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs/r/ReFrame) containing a more recent Python version.
* Set RFM_PURGE_ENVIRONMENT=1 if you use Python from a module. The ReFrame easyconfigs automatically do that for you.

??? note EasyBuild is needed for certain tests (e.g. BLAS) that need to load multiple modules together. EasyBuilds functionality is used in these cases to finding matching-pairs of modules. If EasyBuild is not available, a warning will be printed and the tests requiring this functionality will be skipped.
??? note "(For more info about the optional EasyBuild requirement, click here)"

EasyBuild is needed for certain tests (e.g. BLAS) that need to load
multiple modules together. EasyBuild’s functionality is used in these cases
to find matching-pairs of modules. If EasyBuild is not available, a warning
will be printed and the tests requiring this functionality will be skipped.

#### Installing Reframe

Expand All @@ -52,25 +57,6 @@ reframe --version
(see [ReFrame issue #2914](https://github.com/reframe-hpc/reframe/issues/2914)).


#### Installing ReFrame test library (`hpctestlib`)

The EESSI test suite requires that the ReFrame test library (`hpctestlib`) is available, which is currently not included in a standard installation of ReFrame.

We recommend installing ReFrame using [EasyBuild](https://easybuild.io/) (version 4.8.1, or newer), or using a ReFrame installation that is available in the EESSI repository (version 2023.06, or newer).

For example (using EESSI):

```bash
source /cvmfs/software.eessi.io/versions/2023.06/init/bash
module load ReFrame/4.3.3
```

To check whether the ReFrame test library is available, try importing a submodule of the `hpctestlib` Python package:

```bash
python3 -c 'import hpctestlib.sciapps.gromacs'
```

## Installation { #installation }

To install the EESSI test suite, you can either use `pip` or clone the GitHub repository directly:
Expand Down
Loading
Loading