[ci] Run xharness DeviceTests on dnceng-public Helix #30544

rmarinho · 2025-07-10T18:22:47Z

Description of Change

Enable a way to run device tests on Helix with xharness.
Right now we going to support both modes, helix and non helix as we are doing the move.

Example run https://dev.azure.com/dnceng-public/public/_build/results?buildId=1115383&view=results

About helix

Check available queues https://helix.dot.net (right now we just going to use open)

Number of Tests (from visual test runner)

iOS: 3676
Controls: 814
Core: 2529
Graphics: 33
Blazor: 6
Essentials: 294

Catalyst: 3373
Controls:658
Core: 2388
Graphics: 33
Blazor:6
Essentials: 288

Android: 3435
Controls:999
Core: 2995
Graphics: 33
Blazor:6
Essentials: 302

Documentation about xharness on Helix https://github.com/dotnet/arcade/blob/main/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/Readme.md#android-apk-payloads

Run locally from the Mac

The arcade system uses ./build.sh and /build.cmd as the start point, and it will look at the Build.props file for information of what projects to build

#build the Build tasks
./build.sh -restore -build -configuration Release -projects '$(Build.SourcesDirectory)/Microsoft.Maui.BuildTasks.slnf' /bl:BuildBuildTasks.binlog -warnAsError false
 
 #build the devicetests
./build.sh -restore -build -configuration Release /p:BuildDeviceTests=true /bl:BuildDeviceTests.binlog -warnAsError false

#send to helix
./eng/common/msbuild.sh /Users/ruimarinho/dotnet/maui/eng/helix_xharness.proj /restore /p:TreatWarningsAsErrors=false /t:Test /p:TargetOS=android /bl:sendhelix.binlog -verbosity:diag

This pull request introduces improvements to how device tests are built and run in the MAUI repository, with a focus on pipeline and infrastructure updates. The changes standardize the location of device test projects, add new build and test orchestration for device tests, and enhance pipeline configuration for running tests on iOS, Android, and MacCatalyst via Helix and XHarness.

Device Test Infrastructure Improvements

Added new pipeline template stage-device-tests.yml to orchestrate building and running device tests on iOS, Android, and MacCatalyst using Helix and XHarness. This enables automated device test runs in CI for multiple platforms.
Introduced helix_xharness.proj to define scenarios and test payloads for device tests, supporting discovery and execution of test bundles for Controls, Core, Graphics, Essentials, and MauiBlazorWebView.
Updated Build.props to conditionally build device test projects when the BuildDeviceTests property is set, supporting parallel and platform-specific builds for Controls, Core, Graphics, Essentials, and MauiBlazorWebView device tests.

Project Structure and Solution Updates

Standardized the location of MauiBlazorWebView.DeviceTests.csproj by moving it from src/BlazorWebView/tests/MauiDeviceTests/ to src/BlazorWebView/tests/DeviceTests/ in all relevant solution and solution filter files. This improves consistency and discoverability of device test projects. [1] [2] [3] [4] [5] [6]

Pipeline and Build Script Enhancements

Improved Helix pipeline configuration in helix.proj to set default build numbers, queue selection, and Azure Pipelines reporting, as well as correcting paths for unit test projects. [1] [2]

These changes collectively modernize and automate the device test workflow for MAUI, making it easier to run comprehensive tests across multiple platforms in CI.

Issues Fixed

Fixes #30466

src/TestUtils/src/DeviceTests.Runners/HeadlessRunnerOptions.cs

Copilot

Pull Request Overview

This PR enables running device tests on Helix with xharness by introducing support for both traditional and Helix-based testing modes. The changes include correcting project path inconsistencies for MauiBlazorWebView.DeviceTests across solution files, enhancing build configurations for device tests, and adding comprehensive Helix/XHarness testing infrastructure.

Key changes:

Updated project paths for MauiBlazorWebView.DeviceTests across multiple solution files to use the correct DeviceTests directory
Added new Helix and XHarness testing infrastructure with pipeline templates and project configurations
Enhanced memory testing utilities with improved garbage collection handling and async support

Reviewed Changes

Copilot reviewed 24 out of 53 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
Multiple solution files	Corrected MauiBlazorWebView.DeviceTests project path references
eng/helix_xharness.proj	New Helix project configuration for iOS, macOS Catalyst, and Android device testing
eng/pipelines/arcade/stage-device-tests.yml	New pipeline template for Helix-based device testing
eng/Build.props	Added conditional build support for device test projects
src/TestUtils/src/DeviceTests/AssertionExtensions.cs	Enhanced garbage collection testing utilities with async support
Multiple test files	Updated test code to use new collection syntax and skip problematic tests on Helix

src/TestUtils/src/DeviceTests/AssertionExtensions.cs

Copilot · 2025-08-06T13:47:02Z

src/TestUtils/src/DeviceTests/AssertionExtensions.cs

+		public static async Task AssertEventuallyAsync(this Task<bool> assertion, int timeout = 1000, int interval = 100, string message = "Assertion timed out")
+		{
+			do
+			{
+				if (await assertion)


The AssertEventuallyAsync method evaluates the assertion task multiple times by awaiting it repeatedly. Since Task can only be awaited once, subsequent awaits will throw an InvalidOperationException. The task should be stored and its result cached, or the method should accept a Func<Task> instead.

Suggested change

public static async Task AssertEventuallyAsync(this Task<bool> assertion, int timeout = 1000, int interval = 100, string message = "Assertion timed out")

{

do

{

if (await assertion)

public static async Task AssertEventuallyAsync(this Func<Task<bool>> assertion, int timeout = 1000, int interval = 100, string message = "Assertion timed out")

{

do

{

if (await assertion())

src/Controls/tests/DeviceTests/Elements/Border/BorderTests.cs

rmarinho · 2025-08-06T13:47:49Z

eng/pipelines/arcade/variables.yml

  value: $(Build.SourcesDirectory)\build.cmd -ci
 - name: _buildScriptMacOS
-  value: $(Build.SourcesDirectory)/build.sh -ci -warnAsError 0
+  value: $(Build.SourcesDirectory)/build.sh -ci -warnAsError false


i m still not sure about this one

eng/helix_xharness.proj

rmarinho · 2025-08-06T15:59:18Z

eng/helix_xharness.proj

+        <IncludeXHarnessCli>true</IncludeXHarnessCli>
+        <!-- Optional: Specific version of Xcode to use. If omitted, xcode-select is used to
+        determine the version -->
+        <!-- <XHarnessXcodeVersion>16.3</XHarnessXcodeVersion> -->


Do the queues can have more than 1 Xcode version @akoeplinger ?

rmarinho · 2025-08-06T16:00:39Z

eng/test-configuration.json

@@ -0,0 +1,12 @@
+{
+  "version" : 1,


This is for retries.. not sure if it works right now

rmarinho · 2025-08-20T10:37:57Z

/rebase

# Conflicts: # eng/pipelines/device-tests.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

This reverts commit b676c31.

matouskozak · 2025-09-12T13:46:38Z

There seems to be an extra / in the helix URL, see

Would have thought this a helix thing but we have the correct url in runtime CI.

akoeplinger · 2025-09-12T17:16:57Z

@matouskozak this was fixed with dotnet/arcade#15672 in arcade 10.0/main, but maui is still on arcade 9.0

rmarinho commented Jul 10, 2025

View reviewed changes

src/TestUtils/src/DeviceTests.Runners/HeadlessRunnerOptions.cs Show resolved Hide resolved

rmarinho force-pushed the devicetests-dnceng branch from 414b00a to d7d6eb4 Compare July 22, 2025 19:04

rmarinho force-pushed the devicetests-dnceng branch from d844e98 to 3ea6049 Compare August 5, 2025 13:10

rmarinho marked this pull request as ready for review August 6, 2025 13:46

Copilot AI review requested due to automatic review settings August 6, 2025 13:46

rmarinho requested a review from a team as a code owner August 6, 2025 13:46

rmarinho requested review from StephaneDelcroix and jfversluis August 6, 2025 13:46

Copilot AI reviewed Aug 6, 2025

View reviewed changes

rmarinho commented Aug 6, 2025

View reviewed changes

mattleibow previously approved these changes Aug 6, 2025

View reviewed changes

mattleibow reviewed Aug 6, 2025

View reviewed changes

eng/helix_xharness.proj Outdated Show resolved Hide resolved

rmarinho dismissed mattleibow’s stale review via 5cb8df4 August 6, 2025 15:45

rmarinho commented Aug 6, 2025

View reviewed changes

eng/test-configuration.json

@@ -0,0 +1,12 @@

{

"version" : 1,

Copy link

Member Author

rmarinho Aug 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for retries.. not sure if it works right now

rmarinho force-pushed the devicetests-dnceng branch from 49ef863 to e13c25e Compare August 7, 2025 12:03

rmarinho force-pushed the devicetests-dnceng branch from b217e54 to bf05ba6 Compare August 20, 2025 10:44

rmarinho added 12 commits September 10, 2025 20:08

[ci] no yaml templates

ef527d7

# Conflicts: # eng/pipelines/device-tests.yml

[ci] Update variables

356de60

[ci] No variables group

efd41a0

[ci] Enable variables again

2e703a7

[ci] Add arcade device tests

4a0fb32

[ci] Add VM_IMAGE_HOST

9fe1d19

Try use Windows pool

da3cafb

[ci] try fix build

f019c8a

[ci] Fix build parameter

c4a742d

[ci] Run windows device tests on azure pipelines

72a5233

Build device tests

08bc4a7

[ci] Try helix devicetests

d0ba72d

rmarinho and others added 9 commits September 10, 2025 20:10

Try add $(TargetFrameworkToTest)

0f74435

[ci]Try fix results on existing devicetests run

8242a90

fix typo

9e94b34

Try rename

e146f3b

Other fix

e4dce7f

Apply suggestion from @Copilot

2d96464

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Apply suggestion from @Copilot

6f11da5

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update AssertionExtensions.cs

6193946

Update AssertionExtensions.cs

edb319b

rmarinho force-pushed the devicetests-dnceng branch from f39c289 to edb319b Compare September 10, 2025 19:10

rmarinho and others added 8 commits September 10, 2025 20:17

Fix merge

2ef4304

Revert assertion changes

b676c31

fix length count on for loop

d8e7b8e

try find results better

96fce17

Revert "Revert assertion changes"

8894645

This reverts commit b676c31.

Allow to run filter, and wait for test result file

efff7ea

Update windows.cake

e6ec8c6

Add some docs for helix

b66cfb6

rmarinho added this to MAUI SDK Ongoing Sep 11, 2025

github-project-automation bot moved this to Todo in MAUI SDK Ongoing Sep 11, 2025

rmarinho moved this from Todo to Ready To Review in MAUI SDK Ongoing Sep 11, 2025

jfversluis approved these changes Sep 12, 2025

View reviewed changes

github-project-automation bot moved this from Ready To Review to Approved in MAUI SDK Ongoing Sep 12, 2025

vitek-karas requested a review from matouskozak September 12, 2025 11:32

rmarinho merged commit 0923806 into main Sep 12, 2025
145 checks passed

rmarinho deleted the devicetests-dnceng branch September 12, 2025 11:43

github-project-automation bot moved this from Approved to Done in MAUI SDK Ongoing Sep 12, 2025

github-actions bot locked and limited conversation to collaborators Oct 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ci] Run xharness DeviceTests on dnceng-public Helix #30544

[ci] Run xharness DeviceTests on dnceng-public Helix #30544

rmarinho commented Jul 10, 2025 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Aug 6, 2025

Uh oh!

Uh oh!

rmarinho Aug 6, 2025

Uh oh!

Uh oh!

rmarinho Aug 6, 2025

Uh oh!

rmarinho Aug 6, 2025

Uh oh!

rmarinho commented Aug 20, 2025

Uh oh!

Uh oh!

matouskozak commented Sep 12, 2025

Uh oh!

akoeplinger commented Sep 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[ci] Run xharness DeviceTests on dnceng-public Helix #30544

[ci] Run xharness DeviceTests on dnceng-public Helix #30544

Conversation

rmarinho commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Change

About helix

Run locally from the Mac

Issues Fixed

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rmarinho Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rmarinho Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

rmarinho Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

rmarinho commented Aug 20, 2025

Uh oh!

Uh oh!

matouskozak commented Sep 12, 2025

Uh oh!

akoeplinger commented Sep 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rmarinho commented Jul 10, 2025 •

edited

Loading