Proposal for the introduction of extensions (#537)

* extract all changes from previous

* fixup

* allow hyphens in extension names

* fixup hyphens

* only require one toolbox that implements extension

* specify how to work with multiple PEtab problems

* specify we do not require a quorum number of votes

* allow test cases to be provided by the extension library

* Apply suggestions from code review

Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com>

Co-authored-by: Daniel Weindl <dweindl@users.noreply.github.com>

Author: FFroehlich

.github/ISSUE_TEMPLATE/petab-extensions.md

@@ -0,0 +1,27 @@
+---
+
+name: PEtab Extension
+about: Suggest a new extension for PEtab core
+title: ''
+labels: file format
+assignees: ''
+
+---
+
+**Name of the Extension**
+Please make sure that the extension name matches the regular expression `^[a-zA-Z_]\w*$`.
+
+**Which problem would you like to address?**
+A clear and concise description of which use case you want to address and, if applicable, why the current specifications do not fulfill your requirements.
+
+**Describe the solution you would like**
+A clear and concise description of the changes you want to propose. Please describe any additional fields / files you would want to add, including allowed inputs and implications.
+
+**Describe why this should not be implemented by changes to PEtab core**
+A clear and concise description in what way the proposed changes introduce features that are orthogonal to the PEtab core specification.
+
+**List the extension library that implements validation checks**
+A link to the website or github repository that accompanies the proposed extension.
+
+**List the toolboxes that support the proposed standard**
+A link to the website or github repository that contains the software that implements support for the standard.

CHANGELOG.md

@@ -14,6 +14,13 @@ available at https://github.com/PEtab-dev/libpetab-python/.
 * Update tutorial.rst (#512)
 * Update how-to-cite (Closes #432) (#509)
 
+## 0.2 series
+
+### 0.2.0
+
+* Specify how PEtab functionality can be expanded through extensions.
+* YAML files are now required for the specification of PEtab problems
+
 ## 0.1 series
 
 ### 0.1.14

README.md

@@ -136,6 +136,8 @@ will have to:
 
 1. Create a parameter table.
 
+1. Create a yaml file that lists the model and all of the tables above.
+
 If you are using Python, some handy functions of the
 [PEtab library](https://github.com/PEtab-dev/libpetab-python/) can help
 you with that. This includes also a PEtab validator called `petablint` which

doc/_static/petab_schema.yaml

@@ -6,8 +6,13 @@ description: PEtab parameter estimation problem config file schema
 properties:
 
   format_version:
-    type: integer
-    description: Version of the PEtab format (e.g. 1).
+    anyof:
+      - type: string
+        #  (corresponding to PEP 440).
+        pattern: ^([1-9][0-9]*!)?(0|[1-9][0-9]*)(\.(0|[1-9][0-9]*))*((a|b|rc)(0|[1-9][0-9]*))?(\.post(0|[1-9][0-9]*))?(\.dev(0|[1-9][0-9]*))?$
+      - type: integer
+
+    description: Version of the PEtab format
 
   parameter_file:
     oneOf:
@@ -17,7 +22,6 @@ properties:
       File name (absolute or relative) or URL to PEtab parameter table
       containing parameters of all models listed in `problems`. A single
       table may be split into multiple files and described as an array here.
-
   problems:
     type: array
     description: |
@@ -80,7 +84,28 @@ properties:
         - measurement_files
         - condition_files
 
+  extensions:
+    type: object
+    description: |
+      PEtab extensions being used.
+    patternProperties:
+      "^[a-zA-Z][\\-\\w]*$":
+
+        type: object
+        description: |
+          Information on a specific extension
+        properties:
+          version:
+            type: string
+            pattern: ^([1-9][0-9]*!)?(0|[1-9][0-9]*)(\.(0|[1-9][0-9]*))*((a|b|rc)(0|[1-9][0-9]*))?(\.post(0|[1-9][0-9]*))?(\.dev(0|[1-9][0-9]*))?$
+
+        required:
+          - version
+      additionalProperties: true
+
+    additionalProperties: false
+
 required:
   - format_version
   - parameter_file
-  - problems
+  - problems

doc/development.rst

@@ -192,6 +192,56 @@ Upon a new release, the PEtab editors ensure that
 * the new version of the specifications is deposited at Zenodo
 * the new release is announced on the PEtab mailing list
 
+PEtab Extensions
+----------------
+
+An elaborate, monolithic format would make it difficult to understand and
+implement support for PEtab, leading to a steep learning curve and discouraging
+support in new toolboxes. To address this issue, the PEtab format is modular and
+permits modifications through extensions that complement the core standard.
+This modular specification evens the learning curve and provides toolbox
+developers with more guidance on which features to implement to maximize
+support for real world applications. Moreover, such modular extensions
+facilitate and promote the use of specialized tools for specific, non-parameter
+estimation tasks such as visualization.
+
+Requirements for new extensions:
+
+* Specifications in PEtab extensions take precedence over PEtab core, i.e., they
+can ease or refine format restrictions imposed by PEtab core.
+* PEtab extensions should extend PEtab core with new orthogonal features or
+tasks, i.e., they should not make trivial changes to PEtab core.
+* PEtab extensions must be named according to ^[a-zA-Z][\w\-]*$
+* PEtab extensions must be versioned using semantic versioning.
+* PEtab extensions required for interpretation of a problem specification must
+be specified in the PEtab-YAML files
+* There is at least one tool that supports the proposed extension
+* The authors provide a library that provides test cases and implements
+validation checks for the proposed format.
+
+Developers are free to develop any PEtab extension. To become an official
+PEtab extension, it needs to go through the following process.
+
+1. The developers write a proposal describing the motivation and specification
+of the extension, following the respective issue template provided in this
+repository.
+1. The proposal is submitted as an issue in this repository.
+1. The technical specification and documentation of the extension is submitted
+as a pull request in this repository that references the respective issue.
+
+The PEtab editors jointly decide whether an extension meets the requirements
+described here. In case of a positive evaluation, they announce a poll for the
+acceptance as official extension to the PEtab forum. All members of the PEtab
+community are eligible to vote. If at least 50% of the votes are in favor,
+the extension is accepted and the respective pull requests with specifications,
+documentation and test cases are merged. There is no quorum number of votes
+for acceptance.
+
+It is encouraged that extensions are informally discussed with the community
+before initiating the process of becoming an official extension. Such
+discussions can be conducted through the communication channels mentioned
+above.
+
 Versioning of the PEtab format
 ------------------------------
 
@@ -219,4 +269,4 @@ Changes to these processes
 Changes to the processes specified above require a public vote with
 agreement of the majority of voters. Any other changes not directly
 affecting those processes, such as changes to structure, orthography,
-grammar, formatting, the preamble can be made by the editors any time.
+grammar, formatting, the preamble can be made by the editors any time.

doc/documentation_data_format.rst

@@ -58,6 +58,9 @@ and
 - A parameter file specifying estimateable parameters and related information
   [TSV]
 
+- A grouping file that lists all of the files and provides additional information
+  including employed extensions [YAML]
+
 - (optional) A simulation file, which has the same format as the measurement
   file, but contains model simulations [TSV]
 
@@ -680,8 +683,11 @@ Detailed field description
 Extensions
 ~~~~~~~~~~
 
-Additional columns, such as ``Color``, etc. may be specified.
-
+Additional columns, such as ``Color``, etc. may be specified. Extensions
+that define operations on multiple PEtab problems need to employ a single
+PEtab YAML file as entrypoint to the analysis. This PEtab file may leave all
+fields specifying files empty and reference the other PEtab problems in the
+extension specific fields.
 
 Examples
 ~~~~~~~~
@@ -698,7 +704,7 @@ To link the SBML model, measurement table, condition table, etc. in an
 unambiguous way, we use a `YAML <https://yaml.org/>`_ file.
 
 This file also allows specifying a PEtab version (as the format is not unlikely
-to change in the future).
+to change in the future) and employed PEtab extensions.
 
 Furthermore, this can be used to describe parameter estimation problems
 comprising multiple models (more details below).

doc/tutorial.rst

@@ -20,8 +20,8 @@ For more details, we refer to the original publication.
 
 A PEtab problem consists of 1) an SBML model of a biological system, 2)
 condition, observable and measurement definitions, and 3) the
-specification of the parameters. We will show how to generate the
-respective files in the following.
+specification of the parameters and 4) a configuration file that lists all of
+these files. We will show how to generate the respective files in the following.
 
 1. The model
 ++++++++++++
@@ -324,11 +324,10 @@ https://petab.readthedocs.io/en/latest/documentation_data_format.html#visualizat
 5. YAML file
 ++++++++++++
 
-To group the previously mentioned PEtab files, a YAML file can be used,
-defining which files constitute a PEtab problem. While being optional,
-this makes it easier to import a PEtab problem into tools, and allows
-reusing files for different PEtab problems. This file has the following
-format (``Boehm_JProteomeRes2014.yaml``):
+To group the previously mentioned PEtab files, a YAML file must be used,
+defining which files constitute a PEtab problem. This makes it easier to import
+a PEtab problem into tools, and allows reusing files for different PEtab
+problems. This file has the following format (``Boehm_JProteomeRes2014.yaml``):
 
 .. code-block:: yaml