diff --git a/docs/test-suite/ReFrame-configuration-file.md b/docs/test-suite/ReFrame-configuration-file.md index 2c72ce14ec..7eb6c88ac0 100644 --- a/docs/test-suite/ReFrame-configuration-file.md +++ b/docs/test-suite/ReFrame-configuration-file.md @@ -15,85 +15,13 @@ This page is organized as follows: ## Available ReFrame configuration files There are some available ReFrame configuration files for HPC systems and public cloud in the [config directory](https://github.com/EESSI/test-suite/tree/main/config/) for more inspiration. -Below is a simple ReFrame configuration file with minimal changes required for getting you started on using the test suite for a CPU partition. Please check that `stagedir` is set to a path on a (shared) scratch filesystem for storing (temporary) files related to the tests, and `access` is set to a list of arguments that you would normally pass to the scheduler when submitting to this partition (for example '-p cpu' for submitting to a Slurm partition called cpu). +In the config directory, [`settings_example.py`](https://github.com/EESSI/test-suite/blob/main/config/settings_example.py) is a simple ReFrame configuration file with minimal changes required for getting you started on using the test suite for a CPU partition. Please check that `stagedir` is set to a path on a (shared) scratch filesystem for storing (temporary) files related to the tests, and `access` is set to a list of arguments that you would normally pass to the scheduler when submitting to this partition (for example '-p cpu' for submitting to a Slurm partition called cpu). To write a ReFrame configuration file for your system, check the section [How to write a ReFrame configuration file](#write-reframe-config). - -```python -""" -simple ReFrame configuration file -""" -import os - -from eessi.testsuite.common_config import common_logging_config, common_eessi_init, format_perfvars, perflog_format -from eessi.testsuite.constants import * - -site_configuration = { - 'systems': [ - { - 'name': 'cpu_partition', - 'descr': 'CPU partition', - 'modules_system': 'lmod', - 'hostnames': ['*'], - # Note that the stagedir should be a shared directory available on all nodes running ReFrame tests - 'stagedir': f'/some/shared/dir/{os.environ.get("USER")}/reframe_output/staging', - 'partitions': [ - { - 'name': 'cpu_partition', - 'descr': 'CPU partition', - 'scheduler': 'slurm', - 'launcher': 'mpirun', - 'access': ['-p cpu', '--export=None'], - 'prepare_cmds': ['source %s' % common_eessi_init()], - 'environs': ['default'], - 'max_jobs': 4, - 'resources': [ - { - 'name': 'memory', - 'options': ['--mem={size}'], - } - ], - 'features': [ - FEATURES.CPU - ] + list(SCALES.keys()), - } - ] - }, - ], - 'environments': [ - { - 'name': 'default', - 'cc': 'cc', - 'cxx': '', - 'ftn': '', - }, - ], - 'logging': common_logging_config(), - 'general': [ - { - # Enable automatic detection of CPU architecture for each partition - # See https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.processor - 'remote_detect': True, - } - ], -} - -# optional logging to syslog -site_configuration['logging'][0]['handlers_perflog'].append({ - 'type': 'syslog', - 'address': '/dev/log', - 'level': 'info', - 'format': f'reframe: {perflog_format}', - 'format_perfvars': format_perfvars, - 'append': True, -}) -``` - - ## Verifying your ReFrame configuration -To verify the ReFrame configuration, you can [query the configuration using `--show-config`](https://reframe-hpc.readthedocs.io/en/stable/configure.html#querying-configuration-options). +To verify the ReFrame configuration, you can [query the configuration](https://reframe-hpc.readthedocs.io/en/stable/configure.html#querying-configuration-options) using `--show-config`. To see the full configuration, use: @@ -101,7 +29,7 @@ To see the full configuration, use: reframe --show-config ``` -To only show the configuration of a particular system partition, you can use the [`--system` option](https://reframe-hpc.readthedocs.io/en/stable/manpage.html#cmdoption-system). +To only show the configuration of a particular system partition, you can use the [`--system`](https://reframe-hpc.readthedocs.io/en/stable/manpage.html#cmdoption-system) option. To query a specific setting, you can pass an argument to `--show-config`. For example, to show the configuration of the `gpu` partition of the `example` system: @@ -135,8 +63,9 @@ The EESSI test suite standardizes a few string-based values as constants, as wel Every ReFrame configuration file used for running the EESSI test suite should therefore start with the following import statements: ```python -from eessi.testsuite.common_config import common_logging_config, common_eessi_init -from eessi.testsuite.constants import * +from eessi.testsuite.common_config import (common_eessi_init, common_general_config, common_logging_config, + set_common_required_config) +from eessi.testsuite.constants import EXTRAS, FEATURES, SCALES, DEVICE_TYPES, GPU_VENDORS ``` ### High-level system info (`systems`) @@ -209,10 +138,18 @@ site_configuration = { 'name': 'cpu_partition', 'descr': 'CPU partition' 'scheduler': 'slurm', - 'prepare_cmds': ['source %s' % common_eessi_init()], + 'prepare_cmds': [ + # Pass job environment variables like $PATH, etc., into job steps + 'export SLURM_EXPORT_ENV=ALL', + # If your system doesn't have an Lmod installation by default on the batch nodes, + # uncommenting the following two lines will use one from EESSI. + # 'source /cvmfs/software.eessi.io/2025.06/init/lmod/bash', + # 'module unload EESSI', + ], 'launcher': 'mpirun', 'access': ['-p cpu'], - 'environs': ['default'], + # Optionally specify additional (local) environments here + # 'environs': ['local_environ'], 'max_jobs': 4, 'features': [ FEATURES.CPU @@ -222,17 +159,19 @@ site_configuration = { 'name': 'gpu_partition', 'descr': 'GPU partition' 'scheduler': 'slurm', - 'prepare_cmds': ['source %s' % common_eessi_init()], + 'prepare_cmds': [ + # Pass job environment variables like $PATH, etc., into job steps + 'export SLURM_EXPORT_ENV=ALL', + # If your system doesn't have an Lmod installation by default on the batch nodes, + # Uncommenting the following two lines will use one from EESSI. + # 'source /cvmfs/software.eessi.io/2025.06/init/lmod/bash', + # 'module unload EESSI', + ], 'launcher': 'mpirun', 'access': ['-p gpu'], - 'environs': ['default'], + # Optionally specify additional (local) environments here + # 'environs': ['local_environ'], 'max_jobs': 4, - 'resources': [ - { - 'name': '_rfm_gpu', - 'options': ['--gpus-per-node={num_gpus_per_node}'], - } - ], 'devices': [ { 'type': DEVICE_TYPES.GPU, @@ -272,30 +211,17 @@ The most common configuration items defined at this level are: (for example by using '`--export=None`' with Slurm). This avoids test failures due to environment variables set in the submission environment that are passed down to submitted jobs. - [`prepare_cmds`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.prepare_cmds): - Commands to execute at the start of every job that runs a test. If your batch scheduler does not export - the environment of the submit host, this is typically where you can initialize the EESSI environment. + Commands to execute at the start of every job that runs a test, such as making the EESSI-installed version of Lmod available. - [`environs`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.environs): - The names of the *programming environments* (to be defined later in the configuration file via [`environments`](#environments)) - that may be used on this partition. A programming environment is required for tests that are compiled first, - before they can run. The EESSI test suite however only tests existing software installations, so no compilation - (or specific programming environment) is needed. Simply specify `'environs': ['default']`, - since ReFrame requires that *a* default environment is defined. + The names of optional additional environments (to be defined later in the configuration file via [`environments`](#environments)) + ReFrame requires each test to run in a specific environment. The EESSI test suite automatically adds the available + EESSI modules as environments (e.g. `EESSI-2025.6`) to each partition. We can also add custom environments (e.g. a local + environment for testing a locally installed software stack). - [`max_jobs`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.max_jobs): The maximum amount of jobs ReFrame is allowed to submit in parallel. Some batch systems limit how many jobs users are allowed to have in the queue. You can use this to make sure ReFrame doesn't exceed that limit. - [`resources`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#custom-job-scheduler-resources): - This field defines how additional resources can be requested in a batch job. Specifically, on a GPU partition, - you have to define a resource with the name '`_rfm_gpu`'. The `options` field should then contain the argument to be - passed to the batch scheduler in order to request a certain number of GPUs _per node_, which could be different for - different batch schedulers. For example, when using Slurm you would specify: - ```python - 'resources': [ - { - 'name': '_rfm_gpu', - 'options': ['--gpus-per-node={num_gpus_per_node}'], - }, - ], - ``` + This field should *NOT* be defined, as it is already defined by the `set_common_required_config` function of the EESSI test suite, see below. - [`processor`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.processor): We recommend to *NOT* define this field, unless [CPU autodetection](#cpu-auto-detection) is not working for you. The EESSI test suite relies on information about your processor topology to run. Using CPU autodetection is the @@ -309,7 +235,7 @@ The most common configuration items defined at this level are: 'num_sockets': 2, # Number of sockets in a node 'num_cpus_per_socket': 32, # Number of CPU cores per socket 'num_cpus_per_core': 1, # Number of hardware threads per CPU core - } + } ``` - [`features`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.features): The `features` field is used by the EESSI test suite to run tests _only_ on a partition if it supports a certain @@ -343,10 +269,11 @@ The most common configuration items defined at this level are: } ``` - [`extras`](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.extras): This field specifies extra information on the partition, such as the GPU vendor. Valid fields for `extras` are standardized as constants in [`eessi.testsuite.constants`](https://github.com/EESSI/test-suite/blob/main/eessi/testsuite/constants.py) (for example `EXTRAS.GPU_VENDOR`). This is used by the EESSI test suite to decide if a partition can run a test that _specifically_ requires a certain brand of GPU. - Typically, there is no need to define `extras` for CPU partitions. + `EXTRAS.MEM_PER_NODE` is a required field for `extras`. It specifies the maximum amount of memory per node that can be requested for any test job on the partition. For GPU partitions, you typically want to specify the GPU vendor, for example: ```python 'extras': { + EXTRAS.MEM_PER_NODE: 229376 # in MiB EXTRAS.GPU_VENDOR: GPU_VENDORS.NVIDIA } ``` @@ -364,14 +291,14 @@ Note that as more tests are added to the EESSI test suite, the use of `features` ### Environments { #environments } -ReFrame needs a programming environment to be defined in its configuration file for tests that need to be compiled before they are run. While we don't have such tests in the EESSI test suite, ReFrame requires _some_ programming environment to be defined: +ReFrame needs a programming environment to be defined in its configuration file for tests that need to be compiled before they are run. While we currently don't have such tests in the EESSI test suite, ReFrame requires _some_ programming environment to be defined. The EESSI modules are already added by default as environments for testing the EESSI stack. If additional `environs` are added to any partition, they must be defined in the `environments` section, for example: ```python site_configuration = { ... 'environments': [ { - 'name': 'default', # Note: needs to match whatever we set for 'environs' in the partition + 'name': 'local_environ', # Note: needs to be listed in the partition's `environs` section 'cc': 'cc', 'cxx': '', 'ftn': '', @@ -405,6 +332,15 @@ Alternatively, a prefix can be passed as an argument like `common_logging_config the regular ReFrame log ends up. Note that the performance logs do *not* respect this prefix: they will still end up in the standard ReFrame prefix (by default the current directory, unless otherwise set with `$RFM_PREFIX` or `--prefix`). +### Common required configuration + +After defining the `site_configuration` dictionary, we run the `set_common_required_config` function to update the +dictionary with required configuration options: + +```python +set_common_required_config(site_configuration) +``` + ### Auto-detection of processor information { #cpu-auto-detection } You can let ReFrame [auto-detect the processor information](https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.systems.partitions.processor) for your system. diff --git a/docs/test-suite/index.md b/docs/test-suite/index.md index e22b340052..293f433655 100644 --- a/docs/test-suite/index.md +++ b/docs/test-suite/index.md @@ -5,6 +5,9 @@ The [EESSI test suite](https://github.com/EESSI/test-suite) is a collection of t It is used to check whether the software installations included in the [EESSI software layer](../software_layer.md) are working and performing as expected. +In addition to the EESSI software stack, also it works with other (e.g.locally +installed) software stacks, provided they use the [EasyBuild module naming scheme](https://docs.easybuild.io/api/easybuild/tools/module_naming_scheme/easybuild_mns/). + To get started, you should look into the [installation and configuration guidelines](installation-configuration.md) first. To write the ReFrame configuration file for your system, check [ReFrame configuration file](ReFrame-configuration-file.md). diff --git a/docs/test-suite/installation-configuration.md b/docs/test-suite/installation-configuration.md index 5bf811d56b..c5b17fa79c 100644 --- a/docs/test-suite/installation-configuration.md +++ b/docs/test-suite/installation-configuration.md @@ -9,24 +9,29 @@ This page covers the requirements, installation and configuration of the [EESSI The EESSI test suite requires * Python >= 3.7 -* [ReFrame](https://reframe-hpc.readthedocs.io) v4.3.3 (or newer) -* [ReFrame test library (`hpctestlib`)](https://reframe-hpc.readthedocs.io/en/stable/hpctestlib.html) +* [ReFrame](https://reframe-hpc.readthedocs.io) v4.3.3 or newer (ReFrame v4.10 requires Python >= 3.10) * (optionally) [EasyBuild](https://easybuild.io/) ??? note "(If your system Python version is lower than the minimum required version, click here for some tips)" - * You can upgrade and manage python versions using `pyenv`. [Link](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation) + * You can upgrade and manage Python versions using [`pyenv`](https://github.com/pyenv/pyenv?tab=readme-ov-file#installation). * For installation follow the documentation in the repository using the link above. * Then follow the steps to install a new Python version and further change local paths to the new Python: pyenv install pyenv local + * You can upgrade and manage Python versions using [`uv`](https://docs.astral.sh/uv/guides/install-python/). * You can install a more recent version of Python on top of the GCC/GCCcore compiler. * You can install a ReFrame module with EasyBuild and a [ReFrame easyconfig](https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs/r/ReFrame) containing a more recent Python version. * Set RFM_PURGE_ENVIRONMENT=1 if you use Python from a module. The ReFrame easyconfigs automatically do that for you. -??? note EasyBuild is needed for certain tests (e.g. BLAS) that need to load multiple modules together. EasyBuilds functionality is used in these cases to finding matching-pairs of modules. If EasyBuild is not available, a warning will be printed and the tests requiring this functionality will be skipped. +??? note "(For more info about the optional EasyBuild requirement, click here)" + + EasyBuild is needed for certain tests (e.g. BLAS) that need to load + multiple modules together. EasyBuild’s functionality is used in these cases + to find matching-pairs of modules. If EasyBuild is not available, a warning + will be printed and the tests requiring this functionality will be skipped. #### Installing Reframe @@ -52,25 +57,6 @@ reframe --version (see [ReFrame issue #2914](https://github.com/reframe-hpc/reframe/issues/2914)). -#### Installing ReFrame test library (`hpctestlib`) - -The EESSI test suite requires that the ReFrame test library (`hpctestlib`) is available, which is currently not included in a standard installation of ReFrame. - -We recommend installing ReFrame using [EasyBuild](https://easybuild.io/) (version 4.8.1, or newer), or using a ReFrame installation that is available in the EESSI repository (version 2023.06, or newer). - -For example (using EESSI): - -```bash -source /cvmfs/software.eessi.io/versions/2023.06/init/bash -module load ReFrame/4.3.3 -``` - -To check whether the ReFrame test library is available, try importing a submodule of the `hpctestlib` Python package: - -```bash -python3 -c 'import hpctestlib.sciapps.gromacs' -``` - ## Installation { #installation } To install the EESSI test suite, you can either use `pip` or clone the GitHub repository directly: diff --git a/docs/test-suite/release-notes.md b/docs/test-suite/release-notes.md index 8c8de5c94f..3cbbcde665 100644 --- a/docs/test-suite/release-notes.md +++ b/docs/test-suite/release-notes.md @@ -1,67 +1 @@ -# Release notes for EESSI test suite - -## 0.3.0 (27 june 2024) - -This is a minor release of the EESSI test-suite - -It includes: - -* Update config AWS MC cluster to use `software.eessi.io` (#126) -* Add test for QuantumESPRESSO (pw.x) (#128) -* Fix compact process binding for OpenMPI mpirun (#137) -* Use compact process binding for GROMACS (#139) -* Rename scale tags 1_cpn_2_nodes and 1_cpn_4_nodes (#140) -* Set SRUN_CPUS_PER_TASK for srun launcher (#141) -* Fix for "Failed to modify UD QP to INIT on mlx5_0" on Karolina CI runs (#142) -* Reduce the iteration count to make the OSU tests run faster, especially on slower interconnects (#143) -* Add test for ESPResSo (P3M) (#144) -* Use software.eessi.io repo in CI (#146) -* Add notes on release management to README (#148) -* Fix memory_per_node for Hortense (#151) -* Use MiB units for memory per node (#152) -* Added / updated memory for various systems in MiB units (#153) -* Add additional test for ESPRESSO (LJ) (#155) -* Bump default version used in CI (#157) - -## 0.2.0 (7 march 2024) - -This is a minor release of the EESSI test-suite - -It includes: - -* Implement the CI for regular runs on a system (#93) -* Add OSU tests and update the hooks and configs to make the tests portable (#54, #95, #96, #97, #110, #116, #117, #118, #121) -* Add extra scales to filter tests(#94) -* add new hook to filter out invalid scales based on features in the config (#111) -* unify test names (#108) -* updates to CI workflow ((#102, #103, #104, #105) -* Update common_config (#114) -* Add common config item to redirect the report file to the same directory as e.g. the perflog (#122) -* Fix code formatting + enforce it in CI workflow (#120) - -Bug fixes: - -* Fix hook _assign_num_tasks_per_node (#98) -* fix import common-config vsc_hortense (#99) -* fix typo in partition names in configuration file for vsc_hortense (#106) - - -## 0.1.0 (5 October 2023) - -Version 0.1.0 is the first release of the EESSI test suite. - -It includes: - -* A well-structured `eessi.testsuite` Python package that provides [constants](https://github.com/EESSI/test-suite/blob/main/eessi/testsuite/constants.py), - [utilities](https://github.com/EESSI/test-suite/blob/main/eessi/testsuite/utils.py), - [hooks](https://github.com/EESSI/test-suite/blob/main/eessi/testsuite/hooks.py), - and [tests](https://github.com/EESSI/test-suite/blob/main/eessi/testsuite/tests/), - which can be [installed with "`pip install`"](installation-configuration.md#pip-install). -* Tests for [GROMACS](available-tests.md#gromacs) and [TensorFlow](available-tests.md#tensorflow) in [`eessi.testsuite.tests.apps`](https://github.com/EESSI/test-suite/blob/main/eessi/testsuite/tests/apps) - that leverage the functionality provided by `eessi.testsuite.*`. -* Examples of [ReFrame configuration files](ReFrame-configuration-file.md#reframe-config-file) for various systems in - the [`config` subdirectory](https://github.com/EESSI/test-suite/tree/main/config). -* A [`common_logging_config()`](ReFrame-configuration-file.md#logging) function to facilitate the ReFrame logging configuration. -* A set of standard *device types* and *features* that can be used in the [`partitions` section of the ReFrame configuration file](ReFrame-configuration-file.md#partitions). -* A set of [*tags* (`CI` + `scale`) that can be used to filter checks](usage.md#filter-tag). -* [Scripts](https://github.com/EESSI/test-suite/tree/main/scripts) that show how to run the test suite. +EESSI test-suite [release notes](https://github.com/EESSI/test-suite/blob/main/RELEASE_NOTES.md) diff --git a/docs/test-suite/usage.md b/docs/test-suite/usage.md index b38fadfa51..85145a3f47 100644 --- a/docs/test-suite/usage.md +++ b/docs/test-suite/usage.md @@ -5,16 +5,23 @@ This page covers the usage of the [EESSI test suite](https://github.com/EESSI/te We assume you have already [installed and configured](installation-configuration.md) the EESSI test suite on your system. -## Listing available tests +## Prerequisites + +Before testing the EESSI software stack, you must make sure it is available by +[setting up your environment](../using_eessi/setting_up_environment.md). Specifically, the `EESSI` modules must +be in your `$MODULEPATH` so they can be loaded. You can check this by running `module --show-hidden avail EESSI`. +Loading an EESSI module before running `reframe` is not recommended; this is taken care of by the test suite. + +## Listing available tests { #listing-available-tests } To list the tests that are available in the EESSI test suite, -use `reframe --list` (or `reframe -L` for short). +use `reframe --list-detailed` (or `reframe -L` for short). If you have properly [configured ReFrame](installation-configuration.md), you should see a (potentially long) list of checks in the output: ``` -$ reframe --list +$ reframe --list-detailed ... [List of matched checks] - ... @@ -22,12 +29,25 @@ Found 123 check(s) ``` !!! note - When using `--list`, checks are only generated based on modules that are available in the system where the `reframe` command is invoked. + When using `reframe --list-detailed`, checks are only generated based on modules that are available in the system where the + `reframe` command is invoked. + + The systems, partitions, and environments specified in your ReFrame configuration file are *not* taken into account + when using `--list-detailed`. - The system partitions specified in your ReFrame configuration file are *not* taken into account when using `--list`. + So, if `--list-detailed` produces an overview of 50 checks, and you have 4 system partitions + and 2 environments in your ReFrame configuration, actually running the test suite may result in (up to) 800 + test cases being executed. - So, if `--list` produces an overview of 50 checks, and you have 4 system partitions in your configuration file, - actually running the test suite may result in (up to) 200 checks being executed. +To get a list of all "concrete" test cases, i.e. all valid combinations of checks, systems, partitions, and environments, use: + +``` +$ reframe --list-detailed C +... +[List of matched checks] +- ... +Concretized 1234 test case(s) +``` ## Performing a dry run { #dry-run } @@ -48,8 +68,10 @@ $ reframe --dry-run !!! note - When using `--dry-run`, the systems partitions listed in your ReFrame configuration file are also taken into - account when generating checks, next to available modules and test parameters, which is *not* the case when using `--list`. + Like `--list-detailed C`, `--dry-run` takes into account the system, partitions, and + environments listed in your ReFrame configuration, next to available + modules and test parameters when generating test cases. The difference is + that `--dry-run` also generates the job scripts (without launching them). ## Running the (full) test suite @@ -57,8 +79,9 @@ To actually run the (full) EESSI test suite and let ReFrame produce a performance report, use `reframe --run --performance-report`. We strongly recommend filtering the checks that will be run by using additional options -like `--system`, `--name`, `--tag` (see the ['Filtering tests' section below](#filtering-tests)), -and doing a [dry run](#dry-run) first to make sure that the generated checks correspond to what you have in mind. +like `--system`, `--name`, `--tag`, `--prgenv` (see the ['Filtering tests' section below](#filtering-tests)), +[listing the available tests](#listing-available-tests), and doing a [dry run](#dry-run) first to make sure +that the generated tests correspond to what you have in mind. ## ReFrame output and log files @@ -74,7 +97,7 @@ We strongly recommend controlling where these files go by using the [common logg is provided by the EESSI test suite in your ReFrame configuration file](ReFrame-configuration-file.md#logging) and setting [`$RFM_PREFIX`](installation-configuration.md#RFM_PREFIX) (avoid using the cmd line option `--prefix`). -If you do, and if you use [ReFrame v4.3.3 or more newer](installation-configuration.md#requirements), +If you do, and if you use [ReFrame v4.3.3 or newer](installation-configuration.md#requirements), you should find the output and log files at: * general ReFrame log file at `$RFM_PREFIX/logs/reframe__.log`; @@ -92,7 +115,7 @@ By default, ReFrame will automatically generate checks for each system partition based on the tests available in the EESSI test suite, available software modules, and tags defined in the EESSI test suite. -To avoid being overwhelmed by checks, it is recommend to [apply filters](https://reframe-hpc.readthedocs.io/en/stable/manpage.html#test-filtering) +To avoid being overwhelmed by checks, we recommend [applying filters](https://reframe-hpc.readthedocs.io/en/stable/manpage.html#test-filtering) so ReFrame only generates the checks you are interested in. ### Filtering by test name { #filter-name } @@ -132,13 +155,13 @@ reframe --name /d3adb33f The argument passed to `--name` is interpreted as a Python regular expression, so you can use wildcards like `.*`, character ranges like `[0-9]`, use `^` to specify that the pattern should match from the start of the test name, etc. -Use `--list` or `--dry-run` to check the impact of using the `--name` option. +Use `--list-detailed` or `--dry-run` to check the impact of using the `--name` option. ### Filtering by system (partition) { #filter-system-partition } By default, ReFrame will generate checks for each system partition that is listed in your configuration file. -To only let ReFrame checks for a particular system or system partition, +To only let ReFrame check for a particular system or system partition, you can use the [`--system` option](https://reframe-hpc.readthedocs.io/en/stable/manpage.html#cmdoption-system). For example: @@ -152,7 +175,7 @@ For example: reframe --system example:gpu ... ``` -Use `--dry-run` to check the impact of using the `--system` option. +Use `--list-detailed C` or `--dry-run` to check the impact of using the `--system` option. ### Filtering by tags { #filter-tag } @@ -161,7 +184,7 @@ To filter tests using one or more tags, you can use the [`--tag` option](https:/ Using `--list-tags` you can get a list of known tags. -To check the impact of this on generated checks by ReFrame, use `--list` or `--dry-run`. +To check the impact of this on generated checks by ReFrame, use `--list-detailed` or `--dry-run`. #### `CI` tag @@ -184,14 +207,14 @@ $ reframe --name GROMACS --tag CI #### `scale` tags The EESSI test suite defines a set of custom tags that control the *scale* of -checks, which specify many cores/GPUs/nodes should be used for running a check. +checks, which specify how many cores/GPUs/nodes should be used for running a check. The number of cores and GPUs serves as an upper limit; the actual count depends on the specific configuration of cores, GPUs, and sockets within the node, as well as the specific test being carried out. | tag name | description | |:--------:|-------------| -| `1_core` | using 1 CPU core 1 GPU | +| `1_core` | using 1 CPU core and 1 GPU | | `2_cores` | using 2 CPU cores and 1 GPU | | `4_cores` | using 4 CPU cores and 1 GPU | | `1cpn_2nodes` | using 1 CPU core per node, 1 GPU per node, and 2 nodes | @@ -212,31 +235,51 @@ To filter tests using multiple tags, you can: * use `|` as separator to indicate that *one* of the specified tags must match (logical OR, for example `--tag='1_core|2_cores'`); * use the `--tag` option multiple times to indicate that *all* specified tags must match (logical AND, for example `--tag CI --tag 1_core`); +### Filtering by environment { #filter-environment } + +By default, ReFrame will generate checks for each EESSI environment and each additional (e.g. local) environment listed in your configuration file. + +To only let ReFrame check for a particular environment, +you can use the [`--prgenv` option](https://reframe-hpc.readthedocs.io/en/stable/manpage.html#cmdoption-p). + +For example: + +* To let ReFrame only generate checks for the environment named `EESSI-2025.06`, use: + ``` + reframe --prgenv EESSI-2025.06 ... + ``` +* To let ReFrame only generate checks for a local environment named `local_env` (if defined in your configuration file), use: + ``` + reframe --prgenv local_env ... + ``` + +Use `--list-detailed C` or `--dry-run` to check the impact of using the `--prgenv` option. + ## Example commands -Running all GROMACS tests on 4 cores on the `cpu` partition +Running all GROMACS tests on 4 cores on the `cpu` partition: ``` reframe --run --system example:cpu --name GROMACS --tag 4_cores --performance-report ``` -List all checks for TensorFlow 2.11 using a single node +List all checks for TensorFlow 2.11 using a single node: ``` -reframe --list --name %module_name=TensorFlow/2.11 --tag 1_node +reframe --list-detailed --name %module_name=TensorFlow/2.11 --tag 1_node ``` -Dry run of TensorFlow CI checks on a quarter (1/4) of a node (on all system partitions) +Dry run of TensorFlow CI checks on a quarter (1/4) of a node (on all system partitions): ``` reframe --dry-run --name 'TensorFlow.*CUDA' --tag 1_4_node --tag CI ``` -## Overriding test parameters *(advanced)* +## Overriding test variables *(advanced)* -You can override test parameters using the [`--setvar` option (or `-S`)](https://reframe-hpc.readthedocs.io/en/stable/manpage.html#cmdoption-S). +You can override test variables using the [`--setvar` option (or `-S`)](https://reframe-hpc.readthedocs.io/en/stable/manpage.html#cmdoption-S). This can be done either globally (for all tests), or only for specific tests (which is recommended when using `--setvar`).