Skip to content

Initialize CI for PullRequest Testing #4767

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MatthiasBSchaefer wants to merge 24 commits into from
Closed

Initialize CI for PullRequest Testing #4767

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f509569
added yml file for testing PRs on LTX runner
MatthiasBSchaefer Jan 20, 2026
2bab916
initialize PR_testing on LTX devices
MatthiasBSchaefer Jan 20, 2026
45b4fbc
DUMMY: Only4PRTesting: add whitespaces to CombiTimeTable
MatthiasBSchaefer Jan 20, 2026
daf4de9
Update PRTesting.yml
MatthiasBSchaefer Jan 23, 2026
ea52fe5
Merge branch 'modelica:MatthiasBSchaefer_PRTesting' into MatthiasBSch…
MatthiasBSchaefer Jan 23, 2026
3a4586b
Adapt yml file for pr testing on linux server
MatthiasBSchaefer Mar 5, 2026
854138f
Merge branch 'modelica:MatthiasBSchaefer_PRTesting' into MatthiasBSch…
MatthiasBSchaefer Mar 5, 2026
ade1b0c
Update PRTesting.yml : correct paths
MatthiasBSchaefer Mar 5, 2026
3787f03
adding spaces in Sources.mo
MatthiasBSchaefer Mar 5, 2026
5d0440c
Update PRTesting.yml
MatthiasBSchaefer Mar 7, 2026
66e8dc4
Update PRTesting.yml
MatthiasBSchaefer Mar 7, 2026
5062534
Merge branch 'modelica:MatthiasBSchaefer_PRTesting' into MatthiasBSch…
MatthiasBSchaefer Mar 9, 2026
daf479f
Update Continuous.mo
MatthiasBSchaefer Mar 12, 2026
fbae83c
Initialize CI for PullRequest Testing
MatthiasBSchaefer Mar 19, 2026
72a7897
adapt names
MatthiasBSchaefer Mar 26, 2026
48a0f19
Add trigger workflow for PR testing to get access to variables and se…
MatthiasBSchaefer Mar 31, 2026
5cdacb9
Add main scripts for CI PRTesting and adapt yml
MatthiasBSchaefer Apr 2, 2026
b9afba6
bugifx in CI_ReSim_trigger.yml
MatthiasBSchaefer Apr 2, 2026
c7922ae
Bugfix in CI_ReSim_trigger.yml
MatthiasBSchaefer Apr 2, 2026
27b6eea
remove print step in PRTesting.yml
MatthiasBSchaefer Apr 2, 2026
7b2861a
PRTesting.yml without target
MatthiasBSchaefer Apr 2, 2026
826763a
PRTesting.yml with labeled target
MatthiasBSchaefer Apr 2, 2026
5063ec5
Update yml script using "secrets and variables"; publish level 1 shel…
MatthiasBSchaefer May 12, 2026
1b23d6f
give execution rights to ReSim scripts
MatthiasBSchaefer May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .CI/ReSim/prepare_pr.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
event_number=$1
Copy link
Copy Markdown
Contributor

@AnHeuermann AnHeuermann May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a common practice to add a shebang and to validate arguments in all shell scripts. Also using set -euo pipefail is a good way to ensure that your script fails loud, see https://gist.github.com/imrekoszo/eee95e94389e168a2dffe42bbc058a47.

#!/bin/bash
set -euo pipefail

if [ -z "${1:-}" ]; then
  echo "Error: event_number argument is required" >&2
  exit 1
fi

Personally I would also add set -x to echo the commands. Makes debugging later on way easier.
And I'm a fan of self documenting scripts, so I would suggest to add a help functions that also severs as the documentation for this script.

# checkout the pull request
docker run --name compile_msl --rm --volume shared_data:/shared_data wsm_image /shared_data/run_scripts/git_pr_pull.sh $event_number
# compile msl binaries
docker run --name compile_msl --rm --volume shared_data:/shared_data wsm_image /shared_data/run_scripts/compile_msl.sh /shared_data/repos/pr_testing/ModelicaStandardLibrary
# delete old PRs with same number (in case of a new commit on a PR)
docker run --name compile_msl --rm --volume shared_data:/shared_data dymola_image /shared_data/run_scripts/delete_old_PRs.sh $event_number
33 changes: 33 additions & 0 deletions .CI/ReSim/run_pr.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
event_number=$1
Copy link
Copy Markdown
Contributor

@AnHeuermann AnHeuermann May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment.

Suggested change
event_number=$1
#!/bin/bash
set -euo pipefail
if [ $# -lt 4 ]; then
echo "Error: requires 4 arguments: event_number master_hash pkg_name tools" >&2
exit 1
fi
event_number=$1

master_hash=$2
pkg_name=$3

# Run Modelica in all tools
tools=$4

for tool in $tools
do
uppertool=$( echo $tool | tr 'a-z' 'A-Z')
docker run --rm --name $tool\_pr_compare_$event_number\_$pkg_name --volume shared_data:/shared_data $tool\_image python /shared_data/run_scripts/$uppertool\_PR_compare.py $master_hash $event_number $pkg_name
done

#docker run --rm --name dymola_pr_compare_$event_number\_$pkg_name --volume shared_data:/shared_data dymola_image python /shared_data/run_scripts/Dymola_PR_compare.py $master_hash $event_number $pkg_name
#docker run --rm --name om_pr_compare_$event_number\_$pkg_name --volume shared_data:/shared_data om_image python /shared_data/run_scripts/OM_PR_compare.py $master_hash $event_number $pkg_name
#docker run --rm --name wsm_pr_compare_$event_number\_$pkg_name --volume shared_data:/shared_data wsm_image python /shared_data/run_scripts/WSM_PR_compare.py $master_hash $event_number $pkg_name

# create an overview over all tools
docker run --name create_overview_$event_number\_$pkg_name --volume shared_data:/shared_data om_image python /shared_data/run_scripts/overview_report.py $event_number $pkg_name

# clear published directory (web server)
if [ -d "/var/www/html/prs/$event_number/$pkg_name" ]; then
rm -rf "/var/www/html/prs/$event_number/$pkg_name"
fi
if [ ! -d "/var/www/html/prs/$event_number" ]; then
mkdir /var/www/html/prs/$event_number
fi
mkdir /var/www/html/prs/$event_number/$pkg_name

# copy data to webserver directory
docker cp create_overview_$event_number\_$pkg_name:/shared_data/resim_output/PRs/PR_$event_number/$pkg_name/report /var/www/html/prs/$event_number/$pkg_name
docker rm create_overview_$event_number\_$pkg_name
exit $(head -n 1 /var/www/html/prs/$event_number/$pkg_name/report/status.txt)
Comment on lines +31 to +33
Copy link
Copy Markdown
Contributor

@AnHeuermann AnHeuermann May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From @coderabbitai:

Add error handling and validate exit status.

The script assumes status.txt exists and contains a valid numeric exit code. If the file is missing, empty, or contains non-numeric content, the exit command will fail or produce incorrect results. Additionally, if docker cp or docker rm fails, the script continues without reporting the error.

Suggested change
docker cp create_overview_$event_number\_$pkg_name:/shared_data/resim_output/PRs/PR_$event_number/$pkg_name/report /var/www/html/prs/$event_number/$pkg_name
docker rm create_overview_$event_number\_$pkg_name
exit $(head -n 1 /var/www/html/prs/$event_number/$pkg_name/report/status.txt)
docker cp create_overview_${event_number}_${pkg_name}:/shared_data/resim_output/PRs/PR_${event_number}/${pkg_name}/report "/var/www/html/prs/${event_number}/${pkg_name}" || {
echo "Error: Failed to copy report from container" >&2
exit 1
}
docker rm create_overview_${event_number}_${pkg_name}
status_file="/var/www/html/prs/${event_number}/${pkg_name}/report/status.txt"
if [ -f "$status_file" ]; then
status=$(head -n 1 "$status_file")
if [[ "$status" =~ ^[0-9]+$ ]]; then
exit "$status"
else
echo "Error: Invalid status in $status_file: $status" >&2
exit 1
fi
else
echo "Error: Status file not found: $status_file" >&2
exit 1
fi
mkdir /var/www/html/prs/$event_number
fi
mkdir /var/www/html/prs/$event_number/$pkg_name

43 changes: 43 additions & 0 deletions .github/workflows/ReSim_CI.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
name: CI ReSim

on:
pull_request_target:
workflow_dispatch:
Copy link
Copy Markdown
Contributor

@AnHeuermann AnHeuermann May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

workflow_dispatch will probably fail due to missing context.

When triggered via workflow_dispatch, github.event.number and github.event.pull_request.base.sha are undefined, causing EVENT_NUMBER and PR_SHA to be empty.

on:
   pull_request_target:
   workflow_dispatch:
+    inputs:
+      pr_number:
+        description: 'PR number'
+        required: true
+        type: number
+      base_sha:
+        description: 'Base commit SHA'
+        required: true
+        type: string

This also needs an update on the next lines:

 env:
     PR_SERVER: ${{ vars.LTX_PR_SERVER }}
     TESTING_TOOLS: ${{ vars.LTX_TESTING_TOOLS }}
-    EVENT_NUMBER:  ${{ github.event.number }}
-    PR_SHA: ${{ github.event.pull_request.base.sha }}
+    EVENT_NUMBER:  ${{ github.event.number || inputs.pr_number }}
+    PR_SHA: ${{ github.event.pull_request.base.sha || inputs.base_sha }}


env:
PR_SERVER: ${{ vars.LTX_PR_SERVER }}
TESTING_TOOLS: ${{ vars.LTX_TESTING_TOOLS }}
EVENT_NUMBER: ${{ github.event.number }}
PR_SHA: ${{ github.event.pull_request.base.sha }}

jobs:
prepare:
runs-on: [ self-hosted, Linux, regression_testing ]
steps:
- uses: actions/checkout@v6
with:
sparse-checkout: './.CI/ReSim'

- name: Checkout code in docker
run: ./.CI/ReSim/prepare_pr.sh ${{ env.EVENT_NUMBER }}

testrun_modelica:
needs: prepare
runs-on: [ self-hosted, Linux, regression_testing ]
environment:
name: Report modelica
url: ${{ env.PR_SERVER }}/${{ env.EVENT_NUMBER }}/Modelica/report/PR_comparison_report.html
steps:
- name: Run tests modelica
run: ./.CI/ReSim/run_pr.sh ${{ env.EVENT_NUMBER }} ${{ env.PR_SHA }} Modelica ${{ env.TESTING_TOOLS }}
Comment on lines +24 to +32
Copy link
Copy Markdown
Contributor

@AnHeuermann AnHeuermann May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test job is missing a checkout step.
This would only work on self-hosted runners where workspace files persist (generally a bad thing). There seems to be no sanitation of the VM / sandbox in between jobs. On the GitHub runners you would need to use artifacts between jobs.



testrun_modelicatest:
needs: prepare
runs-on: [ self-hosted, Linux, regression_testing ]
environment:
name: Report modelicatest
url: ${{ env.PR_SERVER }}/${{ env.EVENT_NUMBER }}/ModelicaTest/report/PR_comparison_report.html
steps:
Copy link
Copy Markdown
Contributor

@AnHeuermann AnHeuermann May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test job is also missing a checkout step, see comment above.

- name: Run tests modelicatest
run: ./.CI/ReSim/run_pr.sh ${{ env.EVENT_NUMBER }} ${{ env.PR_SHA }} ModelicaTest ${{ env.TESTING_TOOLS }}
Loading