Add maui desktop testing#5161
Open
LoopedBard3 wants to merge 1 commit intodotnet:mainfrom
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new MAUI Desktop BenchmarkDotNet (BDN) scenario to the repo, including a reusable shared helper to patch/build/run external desktop benchmark suites and (optionally) collect/upload PerfLabExporter results.
Changes:
- Introduces
BDNDesktopHelper(shared/bdndesktop.py) to patch TFMs, inject PerfLabExporter, build, run, and collect BDN results from external repos. - Adds a new
mauiDesktopBenchmarksscenario (pre/test/post scripts) plus a Helix project file to run MAUI desktop BDN suites. - Adds (currently disabled) pipeline job stubs for the new run kind in
sdk-perf-jobs.yml.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| src/scenarios/shared/runner.py | Whitespace-only change in argument parsing. |
| src/scenarios/shared/bdndesktop.py | New shared helper for patching external repos and running/collecting BDN desktop benchmarks. |
| src/scenarios/mauiDesktopBenchmarks/test.py | Clones dotnet/maui, patches props, builds dependencies, runs BDN suites via shared helper. |
| src/scenarios/mauiDesktopBenchmarks/pre.py | Minimal pre-command script (logging only). |
| src/scenarios/mauiDesktopBenchmarks/post.py | Cleans up MAUI repo checkout and combined report artifact. |
| eng/pipelines/sdk-perf-jobs.yml | Adds new MAUI desktop BDN job entries, currently gated by if false. |
| eng/performance/maui_desktop_benchmarks.proj | New Helix project definition for MAUI desktop BDN work item. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+162
to
+169
| # Add ProjectReference to BDN.Extensions | ||
| item_group = ET.SubElement(root, f'{ns}ItemGroup') | ||
| item_group.set('Label', 'PerfLabInjected') | ||
| proj_ref = ET.SubElement(item_group, f'{ns}ProjectReference') | ||
| proj_ref.set('Include', bdn_ext_rel) | ||
|
|
||
| tree.write(csproj_path, xml_declaration=True, encoding='utf-8') | ||
| log.info(f' Added ProjectReference: {bdn_ext_rel}') |
LoopedBard3
requested review from
simonrozsival
and removed request for
simonrozsival
Member
Author
|
/azp run |
|
Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command. |
Member
Author
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new MAUI “desktop BenchmarkDotNet” scenario to the performance scenarios infrastructure, including a shared helper to patch/build/run external-repo BDN projects and collect/upload PerfLab results.
Changes:
- Introduces
BDNDesktopHelperto patch external benchmark repos (TFM disabling, injectBenchmarkDotNet.Extensions, patchProgram.csto usePerfLabExporter+ exclusion filter), then build/run and collect results. - Adds a new
mauiDesktopBenchmarksscenario payload and a new Helix project (maui_desktop_benchmarks.proj) to run MAUI desktop BDN suites. - Updates pipeline jobs and makes
ExclusionFilterpublic for external consumption.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/scenarios/shared/bdndesktop.py | New shared helper to patch/build/run external desktop BDN benchmarks and collect/upload PerfLab reports. |
| src/scenarios/mauiDesktopBenchmarks/test.py | MAUI-specific orchestration (clone/sparse checkout, patch props, build deps, run suites). |
| src/scenarios/mauiDesktopBenchmarks/pre.py | Minimal pre-command stub for the scenario payload. |
| src/scenarios/mauiDesktopBenchmarks/post.py | Cleanup of cloned MAUI repo and local combined report artifact. |
| src/harness/BenchmarkDotNet.Extensions/ExclusionFilter.cs | Makes ExclusionFilter public for use by patched external Program.cs. |
| eng/pipelines/sdk-perf-jobs.yml | Adds MAUI desktop BDN jobs, but also comments out most existing public jobs. |
| eng/performance/maui_desktop_benchmarks.proj | New Helix project definition for the MAUI desktop BDN scenario work item. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
22
to
72
| # # Scenario benchmarks | ||
| # - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| # parameters: | ||
| # jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| # buildMachines: | ||
| # - win-x64 | ||
| # - ubuntu-x64 | ||
| # isPublic: true | ||
| # jobParameters: | ||
| # runKind: scenarios | ||
| # projectFileName: scenarios.proj | ||
| # channels: # for public jobs we want to make sure that the PRs don't break any of the supported frameworks | ||
| # - main | ||
| # - 9.0 | ||
| # - 8.0 | ||
| # ${{ each parameter in parameters.jobParameters }}: | ||
| # ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # # MAUI scenario benchmarks | ||
| # - ${{ if false }}: | ||
| # - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| # parameters: | ||
| # jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| # buildMachines: | ||
| # - win-x64 | ||
| # - ubuntu-x64 | ||
| # isPublic: true | ||
| # jobParameters: | ||
| # runKind: maui_scenarios | ||
| # projectFileName: maui_scenarios.proj | ||
| # channels: # for public jobs we want to make sure that the PRs don't break any of the supported frameworks | ||
| # - main | ||
| # - 9.0 | ||
| # - 8.0 | ||
| # ${{ each parameter in parameters.jobParameters }}: | ||
| # ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # MAUI Desktop BenchmarkDotNet benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: scenarios | ||
| projectFileName: scenarios.proj | ||
| channels: # for public jobs we want to make sure that the PRs don't break any of the supported frameworks | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # MAUI scenario benchmarks | ||
| - ${{ if false }}: | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: maui_scenarios | ||
| projectFileName: maui_scenarios.proj | ||
| channels: # for public jobs we want to make sure that the PRs don't break any of the supported frameworks | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # Blazor scenario benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: blazor_scenarios | ||
| projectFileName: blazor_scenarios.proj | ||
| channels: # for public jobs we want to make sure that the PRs don't break any of the supported frameworks | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # SDK scenario benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| #- win-x86 | ||
| #- ubuntu-x64-1804 reenable under new machine on new ubuntu once lttng/events are available | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: sdk_scenarios | ||
| projectFileName: sdk_scenarios.proj | ||
| channels: # for public jobs we want to make sure that the PRs don't break any of the supported frameworks | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # micro benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-performance-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - win-x86 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: micro | ||
| targetCsproj: src\benchmarks\micro\MicroBenchmarks.csproj | ||
| runCategories: 'runtime libraries' | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
|
|
||
| # Ubuntux64 Default and NativeAOT micro benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-performance-job.yml | ||
| buildMachines: | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: micro | ||
| targetCsproj: src\benchmarks\micro\MicroBenchmarks.csproj | ||
| runCategories: 'runtime libraries' | ||
| channels: | ||
| - main | ||
| # - nativeaot9.0 # Disable until I have time to properly fix the issues and can merge https://github.com/dotnet/performance/pull/4741 | ||
| # - nativeaot8.0 | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # ML.NET benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-performance-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: mlnet | ||
| targetCsproj: src\benchmarks\real-world\Microsoft.ML.Benchmarks\Microsoft.ML.Benchmarks.csproj | ||
| runCategories: 'mldotnet' | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # F# benchmarks | ||
| - ${{ if false }}: # skipping, no useful benchmarks there currently | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-performance-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: fsharp | ||
| targetCsproj: src\benchmarks\real-world\FSharp\FSharp.fsproj | ||
| runCategories: 'fsharp' | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-performance-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: fsharpmicro | ||
| targetCsproj: src\benchmarks\micro-fsharp\MicrobenchmarksFSharp.fsproj | ||
| runCategories: 'FSharpMicro' | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # bepuphysics benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-performance-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: bepuphysics | ||
| targetCsproj: src\benchmarks\real-world\bepuphysics2\DemoBenchmarks.csproj | ||
| runCategories: 'BepuPhysics' | ||
| runKind: maui_desktop_benchmarks | ||
| projectFileName: maui_desktop_benchmarks.proj | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # ImageSharp benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-performance-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: imagesharp | ||
| targetCsproj: src\benchmarks\real-world\ImageSharp\ImageSharp.Benchmarks.csproj | ||
| runCategories: 'ImageSharp' | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # Akade.IndexedSet benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-performance-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: akadeindexedset | ||
| targetCsproj: src\benchmarks\real-world\Akade.IndexedSet.Benchmarks\Akade.IndexedSet.Benchmarks.csproj | ||
| runCategories: 'AkadeIndexedSet' | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # Roslyn benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-performance-job.yml | ||
| buildMachines: | ||
| - win-x64 | ||
| - ubuntu-x64 | ||
| isPublic: true | ||
| jobParameters: | ||
| runKind: roslyn | ||
| targetCsproj: src\benchmarks\real-world\Roslyn\CompilerBenchmarks.csproj | ||
| runCategories: 'roslyn' | ||
| channels: # for Roslyn jobs we want to check .NET Core 3.1 and 5.0 only | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} |
There was a problem hiding this comment.
In the public jobs section, the existing scenario/micro/real-world jobs appear to have been commented out, leaving only the new MAUI desktop job active under runPublicJobs. This would effectively disable the rest of the public correctness coverage, which seems unrelated to adding a new MAUI desktop scenario. Please restore the original public job templates and add the MAUI desktop benchmarks as an additional job (or gate it independently) rather than replacing/commenting out the existing jobs.
Comment on lines
+167
to
+173
| # Add ProjectReference to BDN.Extensions | ||
| item_group = ET.SubElement(root, f'{ns}ItemGroup') | ||
| item_group.set('Label', 'PerfLabInjected') | ||
| proj_ref = ET.SubElement(item_group, f'{ns}ProjectReference') | ||
| proj_ref.set('Include', bdn_ext_rel) | ||
|
|
||
| tree.write(csproj_path, xml_declaration=True, encoding='utf-8') |
There was a problem hiding this comment.
_inject_bdn_extensions always appends a new ItemGroup + ProjectReference without checking whether the project has already been patched (e.g., by a previous run). Re-running the scenario locally (or a retry in the same checkout) can lead to duplicate ProjectReference entries. Consider making this patch idempotent by detecting an existing ProjectReference to the extensions project (or an ItemGroup with Label="PerfLabInjected") before adding a new one.
Adds a new Helix work item that clones dotnet/maui at the channel's
matching branch, patches the project tree to disable mobile target
frameworks, injects PerfLabExporter via BenchmarkDotNet.Extensions,
and runs the Core, XAML, and Graphics BDN suites. Public CI runs on
win-x64 (channel main); private internal CI runs on win-x64-viper.
Layout:
- eng/performance/maui_desktop_benchmarks.proj
Helix work item definition (TargetsWindows-gated).
- eng/pipelines/sdk-perf-jobs.yml
Two new build-matrix entries (public + private) using the
existing run-scenarios-job.yml template.
- src/scenarios/mauiDesktopBenchmarks/{pre,test,post}.py
Scenario payload. pre.py runs on the AzDO agent (just logs the
framework), test.py runs on Helix (clones MAUI, patches, builds,
runs BDN suites), post.py cleans up the cloned repo.
- src/scenarios/shared/bdndesktop.py
Reusable BDNDesktopHelper with the generic external-repo BDN
workflow (patch Directory.Build.props, inject BDN.Extensions,
build, run, collect/upload reports). Designed so other repos
(e.g. dotnet/runtime libraries) can drop in with just a config
dict.
- src/harness/BenchmarkDotNet.Extensions/ExclusionFilter.cs
Made public so the patched-in Program.cs in external benchmarks
can construct it.
Notable design points:
- test.py acquires source via git sparse-checkout when git is on
PATH, falling back to a curl-based zip download (Windows cert
store) when it is not (Helix machines without git installed).
- patch_directory_build_props regex-replaces Include*TargetFrameworks
in place rather than appending overrides, because MAUI sets these
properties at multiple points and computes MauiPlatforms from
them; appending at the end has no effect.
- _inject_bdn_extensions is idempotent: it removes any prior
ItemGroup labeled PerfLabInjected before adding a new one, so
re-running against an existing checkout doesn't accumulate
duplicate ProjectReferences.
- _patch_program_cs raises if the BenchmarkSwitcher.Run pattern
isn't found, preventing silent runs that produce no
perf-lab-report.json.
- BDNDesktopHelper exposes bdn_run_params and strict_build kwargs
so callers can tune iterations or require all suites to build.
- parse_args() uses parse_known_args() so unknown flags
(--filter, --maxIterationCount, etc.) flow through to BDN.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LoopedBard3
force-pushed
the
AddMauiDesktopTesting
branch
from
2530164 to
671ee64
Compare
| log = getLogger() | ||
| upload_root = os.environ.get('HELIX_WORKITEM_UPLOAD_ROOT', '') | ||
|
|
||
| report_pattern = os.path.join(self.repo_dir, '**', '*perf-lab-report.json') |
Comment on lines
+439
to
+440
| globpath = os.path.join(self.repo_dir, '**', '*perf-lab-report.json') | ||
| upload_code = upload.upload(globpath, UPLOAD_CONTAINER, UPLOAD_QUEUE, UPLOAD_STORAGE_URI) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.