Using the EESSI test suite¶
This page covers the usage of the EESSI test suite.
We assume you have already installed and configured the EESSI test suite on your system.
Listing available tests¶
To list the tests that are available in the EESSI test suite,
reframe --list (or
reframe -L for short).
If you have properly configured ReFrame, you should see a (potentially long) list of checks in the output:
--list, checks are only generated based on modules that are available in the system where the
reframe command is invoked.
The system partitions specified in your ReFrame configuration file are not taken into account when using
--list produces an overview of 50 checks, and you have 4 system partitions in your configuration file,
actually running the test suite may result in (up to) 200 checks being executed.
Performing a dry run¶
To perform a dry run of the EESSI test suite, use
--dry-run, the systems partitions listed in your ReFrame configuration file are also taken into
account when generating checks, next to available modules and test parameters, which is not the case when using
Running the (full) test suite¶
To actually run the (full) EESSI test suite and let ReFrame
produce a performance report, use
reframe --run --performance-report.
We strongly recommend filtering the checks that will be run by using additional options
--tag (see the 'Filtering tests' section below](#filtering-tests)),
and doing a dry run first to make sure that the generated checks correspond to what you have in mind.
ReFrame output and log files¶
ReFrame will generate various output and log files:
- a general ReFrame log file with debug logging on the ReFrame run (incl. selection of tests, generating checks, test results, etc.);
- stage directories for each generated check, in which the checks are run;
- output directories for each generated check, which include the test output;
- performance log files for each test, which include performance results for the test runs;
We strongly recommend controlling where these files go by using the common logging configuration that
is provided by the EESSI test suite in your ReFrame configuration file
$RFM_PREFIX (avoid using the cmd line option
If you do, and if you use ReFrame v4.3.3 or more newer, you should find the output and log files at:
- general ReFrame log file at
- stage directories in
- output directories in
- performance log files in
In the stage and output directories, there will be a subdirectory for each check that was run,
which are tagged with a unique hash (like
d3adb33f) that is determined based on the specific parameters for that check
(see the ReFrame documentation for more details on the test naming scheme).
By default, ReFrame will automatically generate checks for each system partition, based on the tests available in the EESSI test suite, available software modules, and tags defined in the EESSI test suite.
To avoid being overwhelmed by checks, it is recommend to apply filters so ReFrame only generates the checks you are interested in.
Filtering by test name¶
You can filter checks based on the full test name using the
--name option (or
which includes the value for all test parameters.
Here's an example of a full test name:
To let ReFrame only generate checks for GROMACS, you can use:
To only run GROMACS checks with a particular version of GROMACS, you can use
--name to only retain specific
Likewise, you can filter on any part of the test name.
You can also select one specific check using the corresponding test hash,
which is also part of the full test name (see
/d3adb33f in the example above):
The argument passed to
--name is interpreted as a Python regular expression, so you can use wildcards like
character ranges like
^ to specify that the pattern should match from the start of the test name, etc.
--dry-run to check the impact of using the
Filtering by system (partition)¶
By default, ReFrame will generate checks for each system partition that is listed in your configuration file.
To only let ReFrame checks for a particular system or system partition,
you can use the
- To let ReFrame only generate checks for the system named
- To let ReFrame only generate checks for the
gpupartition of the system named
--dry-run to check the impact of using the
Filtering by tags¶
To filter tests using one or more tags, you can use the
--list-tags you can get a list of known tags.
To check the impact of this on generated checks by ReFrame, use
For each software that is included in the EESSI test suite,
a small test is tagged with
CI to indicate it can be used in a Continuous Integration (CI) environment.
Hence, you can use this tag to let ReFrame only generate checks for small test cases:
The EESSI test suite defines a set of custom tags that control the scale of checks, which specify many cores/nodes should be used for running a check.
||using 1 CPU core and 1 GPU (if running a GPU test)|
||using 2 CPU cores and 1 GPU (if running a GPU test)|
||using 4 CPU cores and 1 GPU (if running a GPU test)|
||using 1/8th of a node (12.5% of available cores/GPUs, 1 at minimum)|
||using a quarter of a node (25% of available cores/GPUs, 1 at minimum)|
||using half of a node (50% of available cores/GPUs, 1 at minimum)|
||using a full node (all available cores/GPUs)|
||using 2 full nodes|
||using 4 full nodes|
||using 8 full nodes|
||using 16 full nodes|
Using multiple tags¶
To filter tests using multiple tags, you can:
|as separator to indicate that one of the specified tags must match (logical OR, for example
- use the
--tagoption multiple times to indicate that all specified tags must match (logical AND, for example
--tag CI --tag 1_core);
Overriding test parameters (advanced)¶
You can override test parameters using the
--setvar option (or
This can be done either globally (for all tests), or only for specific tests (which is recommended when using
For example, to run all GROMACS checks with a specific GROMACS module, you can use:
We do not recommend using
--setvar, since it is quite easy to make unintended changes to test parameters
this way that can result in broken checks.
Running all GROMACS tests on 4 cores on the
List all checks for TensorFlow 2.11 using a single node¶
Dry run of TensorFlow CI checks on a quarter (1/4) of a node (on all system partitions)¶
The EESSI test suite currently includes tests for:
For a complete overview of all available tests in the EESSI test suite, see the
eessi/testsuite/tests subdirectory in the
EESSI/test-suite GitHub repository.
Crambin(20K atom system)
Glutamine-Binding-Protein(61K atom system)
hEGFRDimer(465K atom system)
hEGFRDimerSmallerPL(465K atom system, only 10k steps)
hEGFRDimerPair(1.4M atom system)
hEGFRtetramerPair(3M atom system)
To run this GROMACS test with all HECBioSim systems, use:
To run this GROMACS test only for a specific HECBioSim system, use for example:
To run this GROMACS test with the smallest HECBioSim system (
Crambin), you can use the
A test for TensorFlow, a machine learning framework, is included, which is based on the "Multi-worker training with Keras" TensorFlow tutorial.
It is implemented in
This test requires TensorFlow v2.11 or newer, using an older TensorFlow version will not work!
To run this TensorFlow test, use: