-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Add emulator process kill timeout with ADB restart and comprehensive timeout protection #30941
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add emulator process kill timeout with ADB restart and comprehensive timeout protection #30941
Conversation
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking good
Can you address this exception as well?
at Submission#0.PrepareDevice(Boolean waitForBoot)
at Submission#0.InstallApk(String testApp, String testAppPackageName, String testResultsDirectory, String skin)
at Submission#0.ExecutePrepareUITests(String project, String app, String appPackageName, String device, String resultsDir, String binDir, String config, String tfm, String rid, String ver, String toolPath, String instrumentation)
at Submission#0.<<Initialize>>b__0_25()
at Cake.Core.CakeTaskBuilderExtensions.<>c__DisplayClass36_0.<Does>b__0(ICakeContext _) in C:\projects\cake\src\Cake.Core\CakeTaskBuilder.Execution.cs:line 26
at Cake.Core.CakeTaskBuilderExtensions.<>c__DisplayClass39_0.<Does>b__0(ICakeContext context) in C:\projects\cake\src\Cake.Core\CakeTaskBuilder.Execution.cs:line 81
at Cake.Core.CakeTask.Execute(ICakeContext context) in C:\projects\cake\src\Cake.Core\CakeTask.cs:line 119
at Cake.Core.DefaultExecutionStrategy.ExecuteAsync(CakeTask task, ICakeContext context) in C:\projects\cake\src\Cake.Core\DefaultExecutionStrategy.cs:line 69
at Cake.Core.CakeEngine.ExecuteTaskAsync(ICakeContext context, IExecutionStrategy strategy, Stopwatch stopWatch, CakeTask task, CakeReport report) in C:\projects\cake\src\Cake.Core\CakeEngine.cs:line 318
at Cake.Core.CakeEngine.ExecuteTaskAsync(ICakeContext context, IExecutionStrategy strategy, Stopwatch stopWatch, CakeTask task, CakeReport report) in C:\projects\cake\src\Cake.Core\CakeEngine.cs:line 341
at Cake.Core.CakeEngine.ExecuteTaskAsync(ICakeContext context, IExecutionStrategy strategy, Stopwatch stopWatch, CakeTask task, CakeReport report) in C:\projects\cake\src\Cake.Core\CakeEngine.cs:line 353
at Cake.Core.CakeEngine.RunTask(ICakeContext context, IExecutionStrategy strategy, CakeTask task, String target, Stopwatch stopWatch, CakeReport report) in C:\projects\cake\src\Cake.Core\CakeEngine.cs:line 254
at Cake.Core.CakeEngine.RunTarget(ICakeContext context, IExecutionStrategy strategy, CakeTask[] orderedTasks, String target, Boolean exclusive, Stopwatch stopWatch, CakeReport report) in C:\projects\cake\src\Cake.Core\CakeEngine.cs:line 229
at Cake.Core.CakeEngine.RunTargetAsync(ICakeContext context, IExecutionStrategy strategy, ExecutionSettings settings) in C:\projects\cake\src\Cake.Core\CakeEngine.cs:line 199
at Cake.Cli.BuildScriptHost`1.internalRunTargetAsync() in C:\projects\cake\src\Cake.Cli\Hosts\BuildScriptHost.cs:line 87
at Cake.Cli.BuildScriptHost`1.RunTargetAsync(String target) in C:\projects\cake\src\Cake.Cli\Hosts\BuildScriptHost.cs:line 74
at Cake.Core.Scripting.ScriptHost.RunTarget(String target) in C:\projects\cake\src\Cake.Core\Scripting\ScriptHost.cs:line 110
at Submission#0.<<Initialize>>d__0.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.RunSubmissionsAsync(ScriptExecutionState executionState, ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, Func`2 catchExceptionOpt, CancellationToken cancellationToken)
at Cake.Infrastructure.Scripting.RoslynScriptSession.Execute(Script script) in C:\projects\cake\src\Cake\Infrastructure\Scripting\RoslynScriptSession.cs:line 206
at Cake.Core.Scripting.ScriptRunner.Run(IScriptHost host, FilePath scriptPath) in C:\projects\cake\src\Cake.Core\Scripting\ScriptRunner.cs:line 172
at Cake.Features.Building.BuildFeature.RunCore(ICakeArguments arguments, BuildFeatureSettings settings) in C:\projects\cake\src\Cake\Features\Building\BuildFeature.cs:line 99
at Cake.Features.Building.BuildFeature.Run(ICakeArguments arguments, BuildFeatureSettings settings) in C:\projects\cake\src\Cake\Features\Building\BuildFeature.cs:line 49
at Cake.Commands.DefaultCommand.Execute(CommandContext context, DefaultCommandSettings settings) in C:\projects\cake\src\Cake\Commands\DefaultCommand.cs:line 76
I've added comprehensive timeout protection and error handling to the PrepareDevice function to address the exception you mentioned. The changes include:
These changes should prevent the PrepareDevice function from hanging indefinitely and provide better resilience in CI/CD environments. Commit: 5cb4cba |
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
once this is happening
Otherwise check for a confirmation dialog on your device.
Waiting 90/120 seconds for the emulator to boot up.
Resolved tool to path /usr/local/lib/android/sdk/platform-tools/adb
Resolved tool to path /usr/local/lib/android/sdk/platform-tools/adb
Executing: /usr/local/lib/android/sdk/platform-tools/adb shell getprop sys.boot_completed
adb: device unauthorized.
This adb server's $ADB_VENDOR_KEYS is not set
Try 'adb kill-server' if that seems wrong.
Otherwise check for a confirmation dialog on your device.
Waiting 91/120 seconds for the emulator to boot up.
Once it gets to about 91/120 let's have it restart the adb server and then keep trying again from that point
... I've added targeted ADB server restart at exactly 90 seconds during emulator boot wait to address the authorization issues you mentioned. The implementation:
This should resolve the "adb: device unauthorized" and "$ADB_VENDOR_KEYS is not set" issues that occur around 91/120 seconds. Commit: 6a3975b |
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds comprehensive timeout protection and recovery mechanisms to Android emulator management in the build automation script. The changes address critical issues where emulator process operations and ADB commands can hang indefinitely, causing CI/CD pipelines to get stuck.
Key changes include:
- Timeout protection for emulator kill operations with automatic ADB server restart recovery
- Safe wrapper functions for ADB commands with timeout handling
- Targeted ADB server restart at 90 seconds during emulator boot to address authorization issues
…timeout protection (dotnet#30941) * Initial plan * Implement emulator process kill timeout with ADB restart functionality Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Add timeout protection and error handling to PrepareDevice function - Added AdbCommandTimeoutSeconds constant (30 seconds) following existing timeout patterns - Created SafeAdbShell helper function with timeout protection for ADB shell commands - Created SafeAdbLogcat helper function with timeout protection for logcat operations - Updated PrepareDevice function to use safe ADB operations with proper error handling - Added try-catch blocks around EnsureAdbKeys, logcat setup, and property configuration - Operations continue gracefully on non-critical failures (logcat, properties) instead of throwing - Maintains existing functionality while preventing indefinite hangs in CI/CD environments Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Add ADB server restart at 90 seconds during emulator boot wait to recover from authorization issues Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Fix emulator process kill timeout mechanism to use WaitForExit instead of Task.Run Co-authored-by: jsuarezruiz <6755973+jsuarezruiz@users.noreply.github.com> * Fix AndroidEmulatorProcess WaitForExit compilation error by using Task.Run on WaitForExit instead of Kill Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Remove timeout protection from PrepareDevice function as requested Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Revert removal of timeout protection from PrepareDevice function Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Remove non-android.cake formatting changes and wrap emulator Kill() operation with timeout Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> Co-authored-by: jsuarezruiz <6755973+jsuarezruiz@users.noreply.github.com>
…timeout protection (#30941) * Initial plan * Implement emulator process kill timeout with ADB restart functionality Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Add timeout protection and error handling to PrepareDevice function - Added AdbCommandTimeoutSeconds constant (30 seconds) following existing timeout patterns - Created SafeAdbShell helper function with timeout protection for ADB shell commands - Created SafeAdbLogcat helper function with timeout protection for logcat operations - Updated PrepareDevice function to use safe ADB operations with proper error handling - Added try-catch blocks around EnsureAdbKeys, logcat setup, and property configuration - Operations continue gracefully on non-critical failures (logcat, properties) instead of throwing - Maintains existing functionality while preventing indefinite hangs in CI/CD environments Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Add ADB server restart at 90 seconds during emulator boot wait to recover from authorization issues Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Fix emulator process kill timeout mechanism to use WaitForExit instead of Task.Run Co-authored-by: jsuarezruiz <6755973+jsuarezruiz@users.noreply.github.com> * Fix AndroidEmulatorProcess WaitForExit compilation error by using Task.Run on WaitForExit instead of Kill Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Remove timeout protection from PrepareDevice function as requested Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Revert removal of timeout protection from PrepareDevice function Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> * Remove non-android.cake formatting changes and wrap emulator Kill() operation with timeout Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: PureWeen <5375137+PureWeen@users.noreply.github.com> Co-authored-by: jsuarezruiz <6755973+jsuarezruiz@users.noreply.github.com>
The Android emulator process kill operation in the build automation script can hang indefinitely, causing CI/CD pipelines and local builds to get stuck. When
emulatorProcess.Kill()fails to complete, there's no recovery mechanism, leading to build timeouts and requiring manual intervention.A specific issue also occurs during emulator boot where authorization problems manifest around 90 seconds with errors like:
Additionally, various ADB commands in the PrepareDevice function can hang indefinitely, causing the entire device preparation process to fail without recovery.
Solution
This PR implements comprehensive timeout mechanisms and error handling for both emulator cleanup operations and device preparation with automatic ADB server restart as a recovery strategy, including targeted intervention for boot-time authorization issues.
Changes Made
Added emulator kill timeout protection:
EmulatorKillTimeoutSeconds = 1 * 60constant following existing timeout patternsemulatorProcess.Kill()operation itself with timeout usingTask.Run()to prevent indefinite hangingWaitForExit()to ensure process actually terminatesImplemented ADB recovery for emulator cleanup:
AdbKillServer()andAdbStartServer()methodsAdded targeted ADB server restart during emulator boot wait:
Added comprehensive timeout protection for PrepareDevice function:
AdbCommandTimeoutSeconds = 30constant for ADB command timeoutsSafeAdbShell()andSafeAdbLogcat()helper functions with timeout protectionEnhanced logging throughout:
Benefits
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
This pull request was created as a result of the following prompt from Copilot chat.
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.