Skip to content

Test launcher based on pytest#366

Open
svalat wants to merge 11 commits intoidefix-code:developfrom
svalat:feature/test-with-pytest
Open

Test launcher based on pytest#366
svalat wants to merge 11 commits intoidefix-code:developfrom
svalat:feature/test-with-pytest

Conversation

@svalat
Copy link
Copy Markdown
Collaborator

@svalat svalat commented Mar 19, 2026

The problem

When running locally the full test suite (tests in /test of Idefix) we might want to avoid getting a long log with no summary of what passed and what failed.

What it does

In order a ease running the suite, I implemented a way to call the tests via the pytest library which is used to implement python unit tests. It offers by default :

  • Summary logging (just SUCCESS if the test is valid)
  • Filtering test to select what to run (eg. only MPI => -k mpi)
  • Get a detailed output only for failed tests.
  • Can get verbose mode (-v) if desired
  • Get report in junit format which can be taken by github or we can generate an html page from it (via junit2html).

testme.json

I provided a way to describe the test running option combinations via a testme.json file which might be more readable
and less programmatic than the current testme.py.

Example:

{
     "variants": [
        {
            "dumpname": "dump.0001.dmp",
            "ini": ["idefix.ini","idefix-rkl.ini"],
            "noplot": true,
            "single": [false, true],
            "reconstruction": 2,
            "mpi": [false, true],
            "tolerance": 0
        }
    ]
}

This PR provides the testme.json equivalent to each testme.py in the test directory.

Documentation

I updated the documentation by adding a new dedicated page in doc/source/testing/testLauncher.rst

Missing

Make a default configuration for re-generate the reference results.

Legacy

I currently let untouched the current testme.py files to let the time to validate and complete the new process.

@svalat svalat added documentation Improvements or additions to documentation tests labels Mar 19, 2026
@svalat svalat changed the title Feature/test with pytest Test launcher based on pytest Mar 19, 2026
@svalat svalat force-pushed the feature/test-with-pytest branch from 7eaebeb to f39d6a6 Compare March 19, 2026 12:23
@glesur glesur self-requested a review March 26, 2026 22:54
Copy link
Copy Markdown
Contributor

@glesur glesur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few typos, but otherwise ok

Overview
--------

The class :doc:`idfxTest <idfxTest>` provides the basement to implement an *Idefix* integration test for validation.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The class :doc:`idfxTest <idfxTest>` provides the basement to implement an *Idefix* integration test for validation.
The class :doc:`idfxTest <idfxTest>` provides the toolbox to implement an *Idefix* integration test for validation.

Depencencies
------------

Before using :doc:`idfxTest <idfxTest>` you need to install some Python Depencencies (possibly in a ``virtual env``):
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Before using :doc:`idfxTest <idfxTest>` you need to install some Python Depencencies (possibly in a ``virtual env``):
Before using :doc:`idfxTest <idfxTest>` you need to install some Python depencencies (possibly in a ``virtual env``):

---------- generated xml file: idefix-tests.junit.xml ---------------------------------------------------------
=============================== 8 passed, 44 deselected in 73.03s (0:01:13) ===================================

When seeing an error, the output of the command will be printed at the end.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When seeing an error, the output of the command will be printed at the end.
When an error is detected, the output of the command will be printed at the end.

Definition of the tests
-----------------------

The script is simply searching all the files names ``testme.json`` into the ``test`` directory.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The script is simply searching all the files names ``testme.json`` into the ``test`` directory.
The script looks for all the files named ``testme.json`` in the ``test`` directory and all its sub-directories.

-----------------------

The script is simply searching all the files names ``testme.json`` into the ``test`` directory.
This file describe the combination of parameters to use to produce the list of *Idefix* run to perform.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This file describe the combination of parameters to use to produce the list of *Idefix* run to perform.
This file describes the combination of parameters used to produce the list of *Idefix* runs to check.

}


By default, the alphabetical order will be used.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
By default, the alphabetical order will be used.
By default, the alphabetical order is used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants