feat: parallelize bakery ci publish across image targets#641
Open
ianpittwood wants to merge 9 commits into
Open
feat: parallelize bakery ci publish across image targets#641ianpittwood wants to merge 9 commits into
ianpittwood wants to merge 9 commits into
Conversation
…sh ordered Rewrites ImageToolsPlugin.publish() to fan Stage 1 (wait + index-create + soci-convert) out across targets via the parallel ShellJob executor, while keeping Stage 2 (index-copy, the real registry push) and Stage 3 (verify) as sequential push_sort_key-ordered loops -- one target's Stage 1 failure no longer blocks any other target's publish. Also fixes test/cli/test_ci.py's merge.feature fixture (patch_image_target_merge_method), whose MockOrasIndexCreateWorkflow.run() didn't accept the runner kwarg that publish() now always passes through _run_publish_stage1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rrupt handling Adds three test cases that were in the task brief but missing from the implementation: - test_job_value_is_callable_return: Verifies job callable return values are captured - test_on_result_called_once_per_job_on_main_thread: Confirms on_result callback runs once per job on main thread - test_sigint_terminates_in_flight_children: Verifies interrupt safety for run_jobs() matches run() Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Test job runs pytest against imagetools code that shells out to soci, but never installed it (unlike bakery-build-native.yml), so SOCI-dependent tests failed with BakeryToolNotFoundError. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Jul 2, 2026
Contributor
Author
|
Tracked by #648. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #619 (unmerged) — targets that branch rather than
mainso this diff only shows the new work.Parallelizes
bakery ci publish's per-target pipeline across image targets, while keeping each target's own operation sequence (wait-for-sources → index-create → soci-convert → index-copy → verify) strictly ordered:ParallelShellExecutor.run_jobs()/ShellJob/CommandRunner(parallel/executor.py) built on the same tracked-spawn/Ctrl-C-safety primitive the existing dgossrun()path uses. One target's failure is isolated — it no longer aborts the whole run.push_sort_keyorder, so registries that display tags by push time (e.g. Docker Hub) still show the latest version as most recent.--jobs/-jtobakery ci publish, matching the existing dgoss flag.Out of scope (deferred to follow-up work, per the design spec): retry-with-backoff plumbing in the
parallelmodule (transient registry errors are already handled independently byposit_bakery/retry.py), and unifyingoras merge/soci convertinto this same pipeline.Design spec and implementation plan (local-only, gitignored under
docs/superpowers/) drove this; happy to share their content in review if useful.Test plan
uv run pytest test/parallel/ test/plugins/builtin/imagetools/ test/cli/test_ci_publish.py -q— 174 passeduv run ruff check ./uv run ruff format --check .— clean🤖 Generated with Claude Code