Batch libraries test Helix work items by compatibility metadata#129632
Draft
mmitche wants to merge 2 commits into
Draft
Batch libraries test Helix work items by compatibility metadata#129632mmitche wants to merge 2 commits into
mmitche wants to merge 2 commits into
Conversation
Compose desktop batch commands from structured runner parts and group library Helix work items by compatibility metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Emit non-batch-compatible desktop library work items directly so stress scenarios keep the original runner and timeout instead of flowing through ComputeBatchTimeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Tagging subscribers to this area: @akoeplinger, @matouskozak, @simonrozsival |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR changes the libraries Helix submission flow to batch multiple compatible test payload ZIPs into fewer Helix work items, and adds a desktop batch runner that executes each suite inside a single work item while keeping per-suite results separated.
Changes:
- Extend the
GroupWorkItemsMSBuild task to partition candidates by a compatibility-metadata tuple (and optionally force “solo” batches). - Update
sendtohelixhelp.projto emit either classic per-assembly work items or new batched work items (with per-batch payload creation and timeouts). - Add
DesktopBatchRunnerscripts and a newHelixTestTasks.Testsproject with unit tests for the grouping logic.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/tasks/HelixTestTasks/GroupWorkItems.cs | Adds compatibility partitioning + “solo” batching logic and assigns globally-unique batch IDs across partitions. |
| src/tasks/HelixTestTasks.Tests/HelixTestTasks.Tests.csproj | Introduces a new xUnit test project for HelixTestTasks. |
| src/tasks/HelixTestTasks.Tests/GroupWorkItemsTests.cs | Adds unit tests validating partitioning and solo-batch behaviors. |
| src/libraries/sendtohelixhelp.proj | Adds batching opt-in default + emits batched work items, payload staging, and batch-runner command composition. |
| src/libraries/pretest.proj | Ensures HelixTestTasks is built for desktop library Helix batching scenarios (not just browser). |
| eng/testing/DesktopBatchRunner.sh | Adds Unix batch runner that unpacks suite ZIPs, sets per-suite upload roots, runs each suite, and summarizes outcomes. |
| eng/testing/DesktopBatchRunner.cmd | Adds Windows batch runner with similar behavior to the Unix runner. |
Comment on lines
+63
to
+66
| string[] compatibilityKeys = CompatibilityMetadataKeys | ||
| .Split(s_compatibilityMetadataKeySeparators, StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); | ||
| bool hasCompatibilityKeys = compatibilityKeys.Length > 0; | ||
| string batchCompatibleMetadataKey = BatchCompatibleMetadataKey.Trim(); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reduces the number of libraries-test Helix work items by batching multiple test assemblies into a single work item, cutting per-work-item overhead (queueing, machine provisioning, payload/runtime download) while keeping total test execution time roughly flat.
Today each libraries test assembly is sent to Helix as its own work item, so the fixed per-work-item overhead is paid thousands of times. This change groups compatible desktop libraries work items together and runs them through a structured batch runner.
How it works
GroupWorkItemsMSBuild task (src/tasks/HelixTestTasks/GroupWorkItems.cs): greedy bin-packs assemblies into batches by file size. Items above a size threshold get their own solo batch. Assemblies are first partitioned by a compatibility-key tuple so only co-runnable assemblies share a work item:RunnerType;PlatformFamily;RuntimeFlavor;TargetOS;TargetArchitecture;Scenario;TimeoutClass;PayloadShapeDesktopBatchRunner.sh/.cmd(eng/testing/): inside a single work item, iterates the batched suites, gives each its own upload root so results don't collide, runs each suite's existingRunTestsscript, and aggregates per-suite exit codes/durations.sendtohelixhelp.proj: composes the desktop batch command from structured runner parts and emits batched work items. Non-batch-compatible (e.g. stressScenario) items are emitted directly so they keep their original runner and timeout instead of flowing throughComputeBatchTimeout.HelixTestTasks.Tests/GroupWorkItemsTests.cs.Batching can be disabled via
BatchLibraryHelixWorkItems=false.Validation
A manual pipeline run of this branch showed:
mainrun, with total test-execution time roughly flat.Opening as draft to get full CI coverage and review on the approach.
Note
This pull request description was drafted with the assistance of GitHub Copilot (AI-generated).