-
Notifications
You must be signed in to change notification settings - Fork 297
Add MAUI Android inner loop deploy measurement scenario #5165
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
davidnguyen-tech
wants to merge
19
commits into
dotnet:main
Choose a base branch
from
davidnguyen-tech:feature/measure-maui-android-deploy
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
8123eed
Add AndroidInnerLoopParser for build/deploy binlog analysis
davidnguyen-tech f6c5869
Add ANDROIDINNERLOOP test type to Python test framework
davidnguyen-tech 7740568
Add MAUI Android inner loop scenario scripts
davidnguyen-tech 3f5e643
Wire MAUI Android inner loop into CI pipeline
davidnguyen-tech 746709a
Make screen timeout configurable and validate cold startup LaunchState
davidnguyen-tech 986c7bf
Address review notes
davidnguyen-tech 26ea3ea
Address review notes 2
davidnguyen-tech 59d3d40
Add SDK version capture + diagnostics to mauiandroidinnerloop
davidnguyen-tech efd2217
Address copilot review notes
davidnguyen-tech d73f5fb
Log rollback_maui.json contents on Helix machine
davidnguyen-tech d5da6c1
Switch MAUI Android Inner Loop to 'dotnet run -p:WaitForExit=false'
davidnguyen-tech c47d731
Migrate MAUI Android Inner Loop startup measurement to logcat polling
davidnguyen-tech 7c1e3bf
Remove stale Android cold startup helper
davidnguyen-tech e2abd8d
Address PR review findings
davidnguyen-tech 27ef49a
Reject non-Debug BuildConfig for MAUI Android Inner Loop
davidnguyen-tech 57d68ff
Address PR review: align with Android verifier-disable + clarify semi…
davidnguyen-tech 0f1a76c
Revert verifier-disable change; keep semicolon-list help text
davidnguyen-tech d019e81
Revert ValidateInnerLoopBuildConfig guard and -c $(BuildConfig) switch
davidnguyen-tech 802b12f
Document why -c Debug is hardcoded in inner loop scenario
davidnguyen-tech File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| <Project Sdk="Microsoft.DotNet.Helix.Sdk" DefaultTargets="Test"> | ||
|
|
||
| <Import Project="Scenarios.Common.props" /> | ||
| <!-- TODO: Scenarios.Common.props sets HelixResultsDestinationDir to a path under | ||
| CorrelationPayloadDirectory, which is cleaned up after SendToHelix. As a result, | ||
| this file's DownloadFilesFromResults artifacts (e.g., output.log) may not appear | ||
| in published pipeline artifacts. A future fix should override | ||
| HelixResultsDestinationDir to a path under ArtifactStagingDirectory that the | ||
| pipeline's "Gather logs" step publishes. --> | ||
|
|
||
| <PropertyGroup> | ||
| <IncludeXHarnessCli>true</IncludeXHarnessCli> | ||
| </PropertyGroup> | ||
|
|
||
| <PropertyGroup> | ||
| <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'mono'">/p:UseMonoRuntime=true</_MSBuildArgs> | ||
| <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'coreclr'">/p:UseMonoRuntime=false</_MSBuildArgs> | ||
|
|
||
| <!-- CoreCLR: disable ReadyToRun so inner loop measures JIT behavior --> | ||
| <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'coreclr'">$(_MSBuildArgs);/p:PublishReadyToRun=false;/p:PublishReadyToRunComposite=false</_MSBuildArgs> | ||
|
|
||
| <!-- AndroidRid: defaults based on target OS but can be overridden. | ||
| Windows targets physical devices; Linux targets emulators. --> | ||
| <AndroidRid Condition="'$(AndroidRid)' == '' and '$(TargetsWindows)' == 'true'">android-arm64</AndroidRid> | ||
| <AndroidRid Condition="'$(AndroidRid)' == '' and '$(TargetsWindows)' != 'true'">android-x64</AndroidRid> | ||
| <_MSBuildArgs>$(_MSBuildArgs) /p:RuntimeIdentifier=$(AndroidRid)</_MSBuildArgs> | ||
|
|
||
| <!-- TODO: https://github.com/dotnet/maui/issues/34706 --> | ||
| <_MSBuildArgs>$(_MSBuildArgs) /p:SupportedOSPlatformVersion=23</_MSBuildArgs> | ||
| <!-- Add TargetFrameworks override so dotnet build's implicit restore | ||
| only evaluates the android TFM, not multi-platform TFMs in the csproj. --> | ||
| <_MSBuildArgs Condition="!$(_MSBuildArgs.Contains('/p:TargetFrameworks='))">$(_MSBuildArgs) /p:TargetFrameworks=$(PERFLAB_Framework)-android</_MSBuildArgs> | ||
|
|
||
| <RunConfigsString>$(RuntimeFlavor)_$(CodegenType)</RunConfigsString> | ||
| <InnerLoopIterations Condition="'$(InnerLoopIterations)' == ''">10</InnerLoopIterations> | ||
| <!-- Screen timeout must be large enough so the display stays on for the entire scenario. | ||
| If the screen turns off mid-run, ActivityTaskManager never emits the 'Displayed' line | ||
| and measure_startup_from_logcat will time out. --> | ||
| <ScreenTimeoutMs Condition="'$(ScreenTimeoutMs)' == ''">1800000</ScreenTimeoutMs> | ||
| </PropertyGroup> | ||
|
|
||
| <!-- Remove .NET SDK packs from the Helix correlation payload to reduce transfer size. | ||
| The workload packs (~1-2 GB) are restored on the Helix machine during the run step, | ||
| so shipping them in the payload is unnecessary and slows down payload upload. --> | ||
| <Target Name="RemoveDotnetFromCorrelationStaging" BeforeTargets="BeforeTest"> | ||
| <Message Text="Removing Dotnet Packs from Correlation Staging" Importance="high" /> | ||
| <RemoveDir Directories="$(CorrelationPayloadDirectory)dotnet\packs" Condition="'$(TargetsWindows)' == 'true'" /> | ||
| <RemoveDir Directories="$(CorrelationPayloadDirectory)dotnet/packs" Condition="'$(TargetsWindows)' != 'true'" /> | ||
| </Target> | ||
|
|
||
| <ItemDefinitionGroup> | ||
| <HelixWorkItem> | ||
| <Timeout>01:00</Timeout> | ||
| </HelixWorkItem> | ||
| </ItemDefinitionGroup> | ||
|
|
||
| <ItemGroup> | ||
| <MAUIAndroidInnerLoopScenario Include="MAUI Android Inner Loop"> | ||
| <ScenarioDirectoryName>mauiandroidinnerloop</ScenarioDirectoryName> | ||
| <PayloadDirectory>$(ScenariosDir)%(ScenarioDirectoryName)</PayloadDirectory> | ||
| </MAUIAndroidInnerLoopScenario> | ||
| </ItemGroup> | ||
|
|
||
| <!-- PreparePayloadWorkItem: create template and modified source on the build machine. | ||
| pre.py creates the MAUI template in app/, fixes .csproj TFMs, and prepares the | ||
| modified MainPage.xaml.cs in src/ for the incremental deploy simulation. --> | ||
| <ItemGroup> | ||
| <PreparePayloadWorkItem Include="@(MAUIAndroidInnerLoopScenario)"> | ||
| <Command>$(Python) pre.py default -f $(PERFLAB_Framework)</Command> | ||
| <WorkingDirectory>%(PreparePayloadWorkItem.PayloadDirectory)</WorkingDirectory> | ||
| </PreparePayloadWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <!-- Environment variables needed on Helix before setup_helix.py runs (so PATH includes adb). --> | ||
| <PropertyGroup> | ||
| <_WindowsEnvVars>set DOTNET_ROOT=%HELIX_CORRELATION_PAYLOAD%\dotnet;set DOTNET_CLI_TELEMETRY_OPTOUT=1;set DOTNET_MULTILEVEL_LOOKUP=0;set NUGET_PACKAGES=%HELIX_WORKITEM_ROOT%\.packages;set ANDROID_HOME=%HELIX_WORKITEM_ROOT%\android-sdk;set ANDROID_SDK_ROOT=%HELIX_WORKITEM_ROOT%\android-sdk;set JAVA_HOME=%HELIX_WORKITEM_ROOT%\jdk;set PATH=%HELIX_CORRELATION_PAYLOAD%\dotnet;;%HELIX_WORKITEM_ROOT%\android-sdk\platform-tools;;%HELIX_WORKITEM_ROOT%\jdk\bin;;%PATH%</_WindowsEnvVars> | ||
| <_LinuxEnvVars>export DOTNET_ROOT=$HELIX_CORRELATION_PAYLOAD/dotnet;export DOTNET_CLI_TELEMETRY_OPTOUT=1;export DOTNET_MULTILEVEL_LOOKUP=0;export NUGET_PACKAGES=$HELIX_WORKITEM_ROOT/.packages;export ANDROID_HOME=$HELIX_WORKITEM_ROOT/android-sdk;export ANDROID_SDK_ROOT=$HELIX_WORKITEM_ROOT/android-sdk;export JAVA_HOME=$HELIX_WORKITEM_ROOT/jdk;export ANDROID_SERIAL=emulator-5554;export PATH=$HELIX_CORRELATION_PAYLOAD/dotnet:$HELIX_WORKITEM_ROOT/android-sdk/platform-tools:$HELIX_WORKITEM_ROOT/jdk/bin:$PATH</_LinuxEnvVars> | ||
|
davidnguyen-tech marked this conversation as resolved.
|
||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup Condition="'$(TargetsWindows)' == 'true'"> | ||
| <!-- NOTE: The scenario is only relevant in Debug mode. Do NOT use Release mode. --> | ||
| <HelixWorkItem Include="@(MAUIAndroidInnerLoopScenario -> 'Inner Loop Device - %(Identity)')"> | ||
| <PreCommands>$(_WindowsEnvVars);$(Python) setup_helix.py $(PERFLAB_Framework)-android "$(_MSBuildArgs)"</PreCommands> | ||
| <Command>$(Python) test.py androidinnerloop --csproj-path app/MauiAndroidInnerLoop.csproj --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" --package-name com.companyname.mauiandroidinnerloop -f $(PERFLAB_Framework)-android -c Debug --msbuild-args "$(_MSBuildArgs)" --scenario-name "%(Identity)" --inner-loop-iterations $(InnerLoopIterations) --screen-timeout-ms $(ScreenTimeoutMs) $(ScenarioArgs)</Command> | ||
| <PostCommands>$(Python) post.py</PostCommands> | ||
| <DownloadFilesFromResults>output.log</DownloadFilesFromResults> | ||
| </HelixWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup Condition="'$(TargetsWindows)' != 'true'"> | ||
| <!-- NOTE: The scenario is only relevant in Debug mode. Do NOT use Release mode. --> | ||
| <HelixWorkItem Include="@(MAUIAndroidInnerLoopScenario -> 'Inner Loop Emulator - %(Identity)')"> | ||
| <PreCommands>$(_LinuxEnvVars);$(Python) setup_helix.py $(PERFLAB_Framework)-android "$(_MSBuildArgs)"</PreCommands> | ||
| <Command>$(Python) test.py androidinnerloop --csproj-path app/MauiAndroidInnerLoop.csproj --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" --package-name com.companyname.mauiandroidinnerloop -f $(PERFLAB_Framework)-android -c Debug --msbuild-args "$(_MSBuildArgs)" --scenario-name "%(Identity)" --inner-loop-iterations $(InnerLoopIterations) --screen-timeout-ms $(ScreenTimeoutMs) $(ScenarioArgs)</Command> | ||
| <PostCommands>$(Python) post.py</PostCommands> | ||
| <DownloadFilesFromResults>output.log</DownloadFilesFromResults> | ||
| </HelixWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <Import Project="PreparePayloadWorkItems.targets" /> | ||
|
|
||
| </Project> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| ''' | ||
| post cleanup script | ||
| ''' | ||
|
|
||
| import subprocess | ||
| import sys | ||
| import traceback | ||
| from performance.logger import setup_loggers, getLogger | ||
| from shared.postcommands import clean_directories | ||
| from shared.util import xharness_adb | ||
| from test import EXENAME | ||
|
|
||
| setup_loggers(True) | ||
| logger = getLogger(__name__) | ||
|
|
||
| try: | ||
| # Uninstall the app from the connected device so re-runs start from a clean state | ||
| package_name = f'com.companyname.{EXENAME.lower()}' | ||
| logger.info(f"Uninstalling {package_name} from device") | ||
| subprocess.run(xharness_adb() + ['uninstall', package_name], check=False) | ||
|
|
||
| logger.info("Shutting down dotnet build servers") | ||
| subprocess.run(['dotnet', 'build-server', 'shutdown'], check=False) | ||
|
|
||
| clean_directories() | ||
| except Exception as e: | ||
| logger.error(f"Post cleanup failed: {e}\n{traceback.format_exc()}") | ||
| sys.exit(1) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file is redundant with the existing
eng/performance/maui_scenarios_android.proj. Could we just add the new HelixWorkItem to themaui_scenarios_android.projand condition on the inner dev loop?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we could - I had a prototype for that approach and from ultra-high level, but the Inner Loop scenario seemed too different from the other ones and it looked approximately like this:
Don't have a strong opinion on this, happy to switch to using just one
.projfile :)