Using the EESSI test suite¶

This page covers the usage of the EESSI test suite.

We assume you have already installed and configured the EESSI test suite on your system.

Prerequisites¶

Before testing the EESSI software stack, you must make sure it is available by setting up your environment. Specifically, the EESSI modules must be in your $MODULEPATH so they can be loaded. You can check this by running module --show-hidden avail EESSI. Loading an EESSI module before running reframe is not recommended; this is taken care of by the test suite.

Listing available tests¶

To list the tests that are available in the EESSI test suite, use reframe --list-detailed (or reframe -L for short).

If you have properly configured ReFrame, you should see a (potentially long) list of checks in the output:

$ reframe --list-detailed
...
[List of matched checks]
- ...
Found 123 check(s)

Note

When using reframe --list-detailed, checks are only generated based on modules that are available in the system where the reframe command is invoked.

The systems, partitions, and environments specified in your ReFrame configuration file are not taken into account when using --list-detailed.

So, if --list-detailed produces an overview of 50 checks, and you have 4 system partitions and 2 environments in your ReFrame configuration, actually running the test suite may result in (up to) 800 test cases being executed.

To get a list of all "concrete" test cases, i.e. all valid combinations of checks, systems, partitions, and environments, use:

$ reframe --list-detailed C
...
[List of matched checks]
- ...
Concretized 1234 test case(s)

Performing a dry run¶

To perform a dry run of the EESSI test suite, use reframe --dry-run:

$ reframe --dry-run
...
[==========] Running 1234 check(s)

[----------] start processing checks
[ DRY      ] GROMACS_EESSI ...
...
[----------] all spawned checks have finished

[  PASSED  ] Ran 1234/1234 test case(s) from 1234 check(s) (0 failure(s), 0 skipped, 0 aborted)

Note

Like --list-detailed C, --dry-run takes into account the system, partitions, and environments listed in your ReFrame configuration, next to available modules and test parameters when generating test cases. The difference is that --dry-run also generates the job scripts (without launching them).

Running the (full) test suite¶

To actually run the (full) EESSI test suite and let ReFrame produce a performance report, use reframe --run --performance-report.

We strongly recommend filtering the checks that will be run by using additional options like --system, --name, --tag, --prgenv (see the 'Filtering tests' section below), listing the available tests, and doing a dry run first to make sure that the generated tests correspond to what you have in mind.

ReFrame output and log files¶

ReFrame will generate various output and log files:

a general ReFrame log file with debug logging on the ReFrame run (incl. selection of tests, generating checks, test results, etc.);
stage directories for each generated check, in which the checks are run;
output directories for each generated check, which include the test output;
performance log files for each test, which include performance results for the test runs;

We strongly recommend controlling where these files go by using the common logging configuration that is provided by the EESSI test suite in your ReFrame configuration file and setting $RFM_PREFIX (avoid using the cmd line option --prefix).

If you do, and if you use ReFrame v4.3.3 or newer, you should find the output and log files at:

general ReFrame log file at $RFM_PREFIX/logs/reframe_<datestamp>_<timestamp>.log;
stage directories in $RFM_PREFIX/stage/<system>/<partition>/<environment>/;
output directories in $RFM_PREFIX/output/<system>/<partition>/<environment>/;
performance log files in $RFM_PREFIX/perflogs/<system>/<partition>/<environment>/;

In the stage and output directories, there will be a subdirectory for each check that was run, which are tagged with a unique hash (like d3adb33f) that is determined based on the specific parameters for that check (see the ReFrame documentation for more details on the test naming scheme).

Filtering tests¶

By default, ReFrame will automatically generate checks for each system partition, based on the tests available in the EESSI test suite, available software modules, and tags defined in the EESSI test suite.

To avoid being overwhelmed by checks, we recommend applying filters so ReFrame only generates the checks you are interested in.

Filtering by test name¶

You can filter checks based on the full test name using the --name option (or -n), which includes the value for all test parameters.

Here's an example of a full test name:

GROMACS_EESSI %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_name=GROMACS/2023.1-foss-2022a /d3adb33f @example:gpu+default

To let ReFrame only generate checks for GROMACS, you can use:

reframe --name GROMACS

To only run GROMACS checks with a particular version of GROMACS, you can use --name to only retain specific GROMACS modules:

reframe --name %module_name=GROMACS/2023.1

Likewise, you can filter on any part of the test name.

You can also select one specific check using the corresponding test hash, which is also part of the full test name (see /d3adb33f in the example above): for example:

reframe --name /d3adb33f

The argument passed to --name is interpreted as a Python regular expression, so you can use wildcards like .*, character ranges like [0-9], use ^ to specify that the pattern should match from the start of the test name, etc.

Use --list-detailed or --dry-run to check the impact of using the --name option.

Filtering by system (partition)¶

By default, ReFrame will generate checks for each system partition that is listed in your configuration file.

To only let ReFrame check for a particular system or system partition, you can use the --system option.

For example:

To let ReFrame only generate checks for the system named example, use:
```
reframe --system example ...
```
To let ReFrame only generate checks for the gpu partition of the system named example, use:
```
reframe --system example:gpu ...
```

Use --list-detailed C or --dry-run to check the impact of using the --system option.

Filtering by tags¶

To filter tests using one or more tags, you can use the --tag option.

Using --list-tags you can get a list of known tags.

To check the impact of this on generated checks by ReFrame, use --list-detailed or --dry-run.

`CI` tag¶

For each software that is included in the EESSI test suite, a small test is tagged with CI to indicate it can be used in a Continuous Integration (CI) environment.

Hence, you can use this tag to let ReFrame only generate checks for small test cases:

reframe --tag CI

For example:

$ reframe --name GROMACS --tag CI
...

`scale` tags¶

The EESSI test suite defines a set of custom tags that control the scale of checks, which specify how many cores/GPUs/nodes should be used for running a check. The number of cores and GPUs serves as an upper limit; the actual count depends on the specific configuration of cores, GPUs, and sockets within the node, as well as the specific test being carried out.

tag name	description
`1_core`	using 1 CPU core and 1 GPU
`2_cores`	using 2 CPU cores and 1 GPU
`4_cores`	using 4 CPU cores and 1 GPU
`1cpn_2nodes`	using 1 CPU core per node, 1 GPU per node, and 2 nodes
`1cpn_4nodes`	using 1 CPU core per node, 1 GPU per node, and 4 nodes
`1_8_node`	using 1/8th of a node (12.5% of available cores/GPUs, 1 at minimum)
`1_4_node`	using a quarter of a node (25% of available cores/GPUs, 1 at minimum)
`1_2_node`	using half of a node (50% of available cores/GPUs, 1 at minimum)
`1_node`	using a full node (all available cores/GPUs)
`2_nodes`	using 2 full nodes
`4_nodes`	using 4 full nodes
`8_nodes`	using 8 full nodes
`16_nodes`	using 16 full nodes

Using multiple tags¶

To filter tests using multiple tags, you can:

use | as separator to indicate that one of the specified tags must match (logical OR, for example --tag='1_core|2_cores');
use the --tag option multiple times to indicate that all specified tags must match (logical AND, for example --tag CI --tag 1_core);

Filtering by environment¶

By default, ReFrame will generate checks for each EESSI environment and each additional (e.g. local) environment listed in your configuration file.

To only let ReFrame check for a particular environment, you can use the --prgenv option.

For example:

To let ReFrame only generate checks for the environment named EESSI-2025.06, use:
```
reframe --prgenv EESSI-2025.06 ...
```
To let ReFrame only generate checks for a local environment named local_env (if defined in your configuration file), use:
```
reframe --prgenv local_env ...
```

Use --list-detailed C or --dry-run to check the impact of using the --prgenv option.

Example commands¶

Running all GROMACS tests on 4 cores on the cpu partition:

reframe --run --system example:cpu --name GROMACS --tag 4_cores --performance-report

List all checks for TensorFlow 2.11 using a single node:

reframe --list-detailed --name %module_name=TensorFlow/2.11 --tag 1_node

Dry run of TensorFlow CI checks on a quarter (1/4) of a node (on all system partitions):

reframe --dry-run --name 'TensorFlow.*CUDA' --tag 1_4_node --tag CI

Overriding test variables (advanced)¶

You can override test variables using the --setvar option (or -S).

This can be done either globally (for all tests), or only for specific tests (which is recommended when using --setvar).

For example, to run all GROMACS checks with a specific GROMACS module, you can use:

reframe --setvar GROMACS_EESSI.modules=GROMACS/2023.1-foss-2022a ...

Warning

We do not recommend using --setvar, since it is quite easy to make unintended changes to test parameters this way that can result in broken checks.

You should try filtering tests using the --name or --tag options instead.

Using the EESSI test suite¶

Prerequisites¶

Listing available tests¶

Performing a dry run¶

Running the (full) test suite¶

ReFrame output and log files¶

Filtering tests¶

Filtering by test name¶

Filtering by system (partition)¶

Filtering by tags¶

CI tag¶

scale tags¶

Using multiple tags¶

Filtering by environment¶

Example commands¶

Overriding test variables (advanced)¶

`CI` tag¶

`scale` tags¶