Initialize CI for PullRequest Testing

[MatthiasBSchaefer]

This PR adds the yml file to .github/workflows in order to start the CI for testing PullRequests with the ltx-resim tool.
infrastructure to run the CI-tests is currently hosted and maintained by LTX

see PRs #4725 #4763 #4764 #4765 , which are tests of the CI on a separate branch

[beutlich]

The log is pretty verbose and it's not obvious what might be the reason for failure at first sight.

I have saome ideas:

1. Can we reduce the log verbosity. Can we have a configurable log verbosity.
2. Can we add problem matchers? See for example https://github.com/modelica/ModelicaStandardLibrary/blob/master/.github/moparser.json or https://oneuptime.com/blog/post/2026-01-30-github-actions-problem-matchers/view.
3. Can we upload the comparison reports using actions/upload-artifact?

[beutlich]

Suggested change

	name: PR_testing
	on:
	# Triggers the workflow on push or pull request events but only for the "main" branch
	pull_request:
	# Allows you to run this workflow manually from the Actions tab
	workflow_dispatch:
	# A workflow run is made up of one or more jobs that can run sequentially or in parallel
	name: CI ReSim
	on:
	pull_request:
	workflow_dispatch:

[beutlich]

Suggested change

	runs-on: self-hosted
	runs-on: [ self-hosted, linux, regression-testing ]

[beutlich]

Suggested change

	- name: checkout_pr_repo
	- name: Checkout code

[beutlich]

Suggested change

	runs-on: self-hosted
	runs-on: [ self-hosted, linux, regression-testing ]

[beutlich]

Suggested change

	name: link_modelica
	name: Build libraries

Not sure, what is meant by "link".

[beutlich]

Suggested change

	- name: run modelica
	- name: Run tests

[beutlich]

Suggested change

	name: link_modelicatest
	name: Build libraries

[MatthiasBSchaefer]

The enviroment is meant to create\show the links to the reports.
So "Report modelicatest" might be a good name

[beutlich]

Suggested change

	- name: run modelicatest
	- name: Run tests

[beutlich]

Do not use hard-coded URIs for https://serv.ltx.de (here and in other places), instead prefer variables via https://github.com/modelica/ModelicaStandardLibrary/settings/variables/actions. This way LTX can move its server while workflow keeps untouched.

[beutlich]

For the sake of transparancy, can we add all the CI bash scripts to a new folder ReSim in .CI? Otherwise it will be pure magic for library officers. Usually and as best practice the CI scripts are maintained along with the repo under test.

[MatthiasBSchaefer]

@beutlich
Thank you for your detailed review !

1.) I know, log is pretty verbose. But it is not really meant to be analyzed. Finding errors (like example Modelica.Blocks.Examples.xxx is not running anymore due to the changes in the PR) can be done in the published reports. The log is more or less to analyze possible errors in the workflow itself by us (LTX). We can reduce the logging once we are sure the workflow is stable. (without any changes inside this repository)

2.) probably we could introduce problem matchers, but may not be necessary due to 1.)

3.) We discussed this in the MAP-LIB meetings. Upload artifacts would induce the need to download the reports (which are not that small) , unzip and then open in a browser. These steps, we want to avoid, making the reports directly clickable by publishing on our server.

4.) Thank you for the very good hint with the variables instead of hard links!

5.)We can publish the main shell scripts called by the CI. But these are mainly running the Modelica tools inside docker containers. And we wont publish our regression testing tool itself, nor the docker files, due to intellectual property. So publishing the scripts is only partially helpful concerning transparency, but would induce a change of these scripts (and such a change in the MSL) in order to add a further Modelica tool (e.g. Simulation X) to the regression testing.

6.) Your proposed naming of the jobs is fine for me.

[MatthiasBSchaefer]

As requested, i am now:

• using the "secrets and variables" feature of github
• reduced the logging in order to have a quick overview there
• published the principal scripts

Please have a look at
https://github.com/MatthiasBSchaefer/ModelicaStandardLibrary/actions/runs/25574891439
to see an example run.
(In this case i only added some whitespaces, so not differences in simulation success and trajectory comparison is expected)

[AnHeuermann]

I did run CodeRabbit on this PR and it pointed out multiple critical issues.

I would suggest to:

1. See my mail.
2. Replace shared volumes
   • Use artifacts (+ docker save / docker load) to pass data from one job to the next.
3. Use GitHub runners instead of self-hosted runners wherever possible to minimize attack surface.
   • Split all tools into separate jobs to enable parallelization (some use GitHub runners, some self-hosted runners)
   • Make Docker images available to GitHub runners (use GHCR or a private registry and authenticate with secrets)
4. Use artifacts to publish results
   • Use artifacts to upload to a self-hosted web-server or use something like GitHub pages

[AnHeuermann]

Test job is missing a checkout step.
This would only work on self-hosted runners where workspace files persist (generally a bad thing). There seems to be no sanitation of the VM / sandbox in between jobs. On the GitHub runners you would need to use artifacts between jobs.

[AnHeuermann]

Test job is also missing a checkout step, see comment above.

[AnHeuermann]

workflow_dispatch will probably fail due to missing context.

When triggered via workflow_dispatch, github.event.number and github.event.pull_request.base.sha are undefined, causing EVENT_NUMBER and PR_SHA to be empty.

on:
   pull_request_target:
   workflow_dispatch:
+    inputs:
+      pr_number:
+        description: 'PR number'
+        required: true
+        type: number
+      base_sha:
+        description: 'Base commit SHA'
+        required: true
+        type: string

This also needs an update on the next lines:

 env:
     PR_SERVER: ${{ vars.LTX_PR_SERVER }}
     TESTING_TOOLS: ${{ vars.LTX_TESTING_TOOLS }}
-    EVENT_NUMBER:  ${{ github.event.number }}
-    PR_SHA: ${{ github.event.pull_request.base.sha }}
+    EVENT_NUMBER:  ${{ github.event.number || inputs.pr_number }}
+    PR_SHA: ${{ github.event.pull_request.base.sha || inputs.base_sha }}

[AnHeuermann]

It's a common practice to add a shebang and to validate arguments in all shell scripts. Also using set -euo pipefail is a good way to ensure that your script fails loud, see https://gist.github.com/imrekoszo/eee95e94389e168a2dffe42bbc058a47.

#!/bin/bash
set -euo pipefail

if [ -z "${1:-}" ]; then
  echo "Error: event_number argument is required" >&2
  exit 1
fi

Personally I would also add set -x to echo the commands. Makes debugging later on way easier.
And I'm a fan of self documenting scripts, so I would suggest to add a help functions that also severs as the documentation for this script.

[AnHeuermann]

From @coderabbitai:

Add error handling and validate exit status.

The script assumes status.txt exists and contains a valid numeric exit code. If the file is missing, empty, or contains non-numeric content, the exit command will fail or produce incorrect results. Additionally, if docker cp or docker rm fails, the script continues without reporting the error.

Suggested change

	docker cp create_overview_$event_number\_$pkg_name:/shared_data/resim_output/PRs/PR_$event_number/$pkg_name/report /var/www/html/prs/$event_number/$pkg_name
	docker rm create_overview_$event_number\_$pkg_name
	exit $(head -n 1 /var/www/html/prs/$event_number/$pkg_name/report/status.txt)
	docker cp create_overview_${event_number}_${pkg_name}:/shared_data/resim_output/PRs/PR_${event_number}/${pkg_name}/report "/var/www/html/prs/${event_number}/${pkg_name}" || {
	echo "Error: Failed to copy report from container" >&2
	exit 1
	}
	docker rm create_overview_${event_number}_${pkg_name}
	status_file="/var/www/html/prs/${event_number}/${pkg_name}/report/status.txt"
	if [ -f "$status_file" ]; then
	status=$(head -n 1 "$status_file")
	if [[ "$status" =~ ^[0-9]+$ ]]; then
	exit "$status"
	else
	echo "Error: Invalid status in $status_file: $status" >&2
	exit 1
	fi
	else
	echo "Error: Status file not found: $status_file" >&2
	exit 1
	fi
	mkdir /var/www/html/prs/$event_number
	fi
	mkdir /var/www/html/prs/$event_number/$pkg_name

[AnHeuermann]

See previous comment.

Suggested change

	event_number=$1
	#!/bin/bash
	set -euo pipefail
	if [ $# -lt 4 ]; then
	echo "Error: requires 4 arguments: event_number master_hash pkg_name tools" >&2
	exit 1
	fi
	event_number=$1