Skip to content

feat: retry-with-backoff for the parallel publish path#642

Open
ianpittwood wants to merge 16 commits into
feat/parallel-publishfrom
feat/parallel-publish-retry
Open

feat: retry-with-backoff for the parallel publish path#642
ianpittwood wants to merge 16 commits into
feat/parallel-publishfrom
feat/parallel-publish-retry

Conversation

@ianpittwood

Copy link
Copy Markdown
Contributor

Stacked on #641. Part of splitting #599 (parallelize + retry + unify oras/soci into imagetools) into smaller, independently-reviewable pieces.

What & why

Retry-with-backoff for transient registry errors (RetryPolicy/retry_on_transient in posit_bakery/retry.py) already covered the riskiest phases of the parallel publish path introduced in #641 — index-create and the SOCI pull step, both carried forward from a pre-existing GHCR-flakiness fix. Two gaps remained:

  • Stage 3 (verify) had no retry. OrasIndexVerifyWorkflow did a bare oras manifest fetch — a transient read-after-write blip failed the whole publish with zero backoff, even though every phase before it was protected.
  • Retry backoff wasn't interruptible. retry_on_transient slept via plain time.sleep. A Stage-1 job (running on a ParallelShellExecutor worker thread) blocked mid-backoff wouldn't notice Ctrl-C — the executor's interrupt handling returns immediately on the main thread, but ThreadPoolExecutor registers an atexit hook that joins every thread it ever spawned, so the process wouldn't actually exit until that sleep (and any remaining retry attempts) finished, up to ~62s per stuck job with default settings.

Changes

  • retry_on_transient() gains an injectable sleep callable (defaults to time.sleep, looked up fresh per call so existing patch("posit_bakery.retry.time.sleep") tests keep working unmodified).
  • CommandRunner.sleep() (new, in the parallel module): sleeps in short slices, checking the bound executor's shutdown flag between slices, raising a new ExecutorInterrupted as soon as shutdown is observed — so a mid-backoff job unwinds within about one slice interval of Ctrl-C instead of blocking process exit.
  • OrasIndexCreateWorkflow and SociConvertWorkflow's pull-step retries (the only retries reachable from a Stage-1 ShellJob worker thread) now pass sleep=runner.sleep when a runner is available.
  • OrasIndexVerifyWorkflow gains a retry_policy field and now retries transient errors on the final tag-resolution check, mirroring the existing Create/Copy pattern. No runner/interrupt plumbing there — it runs on the main thread, not inside the executor, so there's no process-exit-blocking risk to fix.

Explicitly out of scope (see design doc): unifying OrasWaitForSourcesWorkflow's poll loop onto RetryPolicy (different, already-correct mechanism), adding jitter, changing RetryPolicy defaults.

Verification

  • just test — 1947 passed; ruff check + ruff format --check clean.
  • Each commit individually reviewed (spec compliance + code quality) before merge into this branch; a final whole-branch review confirmed the cross-task composition (retry → CommandRunner.sleepExecutorInterrupted propagation through _run_job) is correct end to end and all stated non-goals were honored.
  • Known follow-up (not blocking): OrasWaitForSourcesWorkflow's own poll sleep is the one Stage-1 wait left non-interruptible (~5s Ctrl-C latency bound vs. the ~62s this PR fixes for retry backoff) — same one-line fix pattern would close it if desired.

🤖 Generated with Claude Code

ianpittwood and others added 9 commits June 30, 2026 14:24
…sh ordered

Rewrites ImageToolsPlugin.publish() to fan Stage 1 (wait + index-create +
soci-convert) out across targets via the parallel ShellJob executor, while
keeping Stage 2 (index-copy, the real registry push) and Stage 3 (verify)
as sequential push_sort_key-ordered loops -- one target's Stage 1 failure
no longer blocks any other target's publish.

Also fixes test/cli/test_ci.py's merge.feature fixture
(patch_image_target_merge_method), whose MockOrasIndexCreateWorkflow.run()
didn't accept the runner kwarg that publish() now always passes through
_run_publish_stage1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rrupt handling

Adds three test cases that were in the task brief but missing from the implementation:
- test_job_value_is_callable_return: Verifies job callable return values are captured
- test_on_result_called_once_per_job_on_main_thread: Confirms on_result callback runs once per job on main thread
- test_sigint_terminates_in_flight_children: Verifies interrupt safety for run_jobs() matches run()

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Test job runs pytest against imagetools code that shells out to
soci, but never installed it (unlike bakery-build-native.yml), so
SOCI-dependent tests failed with BakeryToolNotFoundError.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ianpittwood ianpittwood requested a review from bschwedler as a code owner July 2, 2026 17:44
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

Test Results

1 970 tests  +11   1 970 ✅ +11   8m 16s ⏱️ ±0s
    1 suites ± 0       0 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 5672335. ± Comparison against base commit 69ee43c.

♻️ This comment has been updated with latest results.

@ianpittwood

Copy link
Copy Markdown
Contributor Author

Tracked by #648.

bschwedler and others added 7 commits July 9, 2026 13:36
publish() had two related problems with how it resolved the soci
binary:

- find_soci_bin was only imported at module load time, so
  unittest.mock.patch() on the source module never took effect at
  call time inside publish() (unlike find_oras_bin, which is
  imported locally for exactly this reason). CI worked around the
  resulting test failures by installing a real soci binary
  (68fa1db) instead of fixing the mock.
- soci_bin was resolved unconditionally, so any bakery ci
  publish/merge call became a hard failure without soci installed,
  even for configs where no target has SOCI enabled. This broke the
  "Merging multiplatform builds" functional test, which exercises
  ORAS-only merging with no SOCI configuration anywhere in its
  fixture.

Import find_soci_bin locally to match find_oras_bin, and only treat
a missing binary as fatal when at least one target in the run
actually has SOCI enabled. Also memoize find_soci_bin with
functools.cache, mirroring #552's convention for per-invocation
idempotent lookups, so a run with multiple SOCI-enabled targets
resolves the binary once instead of once per target.
68fa1db installed a real soci binary in the Test job to work around
BakeryToolNotFoundError failures, but that papered over two actual
bugs in how publish() resolved the soci binary rather than fixing
them:

- find_soci_bin was only imported at module load time, so
  unittest.mock.patch() on the source module never took effect at
  call time, causing mocked unit tests to fall through to the real
  function instead of the mock.
- soci_bin was resolved unconditionally, even when no target in the
  run had SOCI enabled, making the binary a hard requirement for
  every bakery ci publish/merge call regardless of whether it was
  ever used.

Both are fixed in 4f25a18, so the test suite no longer needs a real
soci binary. Confirmed by running the full suite with no soci binary
on PATH: no soci-related failures remain.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add retry_policy field to OrasIndexVerifyWorkflow and wrap the manifest
fetch in retry_on_transient, matching the pattern already used by
OrasIndexCreateWorkflow and OrasIndexCopyWorkflow. Runs on the main
thread, so no runner/interruptible-sleep plumbing is needed.

Also update test_run_failure_reports_partial's second failure to a
non-transient stderr message ("unauthorized: authentication required")
since "not found" is now a retryable transient pattern.
@bschwedler bschwedler force-pushed the feat/parallel-publish-retry branch from 3feb2da to 5672335 Compare July 9, 2026 19:16
@ianpittwood ianpittwood force-pushed the feat/parallel-publish branch from 1a65e9a to 8eec3fd Compare July 14, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants