Skip to content

feat(bazel): capture & forward Bazel stderr (phase 1)#1197

Draft
gregmagolan wants to merge 1 commit into
mainfrom
capture-bazel-stderr
Draft

feat(bazel): capture & forward Bazel stderr (phase 1)#1197
gregmagolan wants to merge 1 commit into
mainfrom
capture-bazel-stderr

Conversation

@gregmagolan
Copy link
Copy Markdown
Member

Today every Bazel invocation in AXL tasks (ctx.bazel.build / ctx.bazel.test, both via bazel_runner.axl) spawns the Bazel child with cmd.stderr(Stdio::inherit()), so its output reaches the terminal untouched and we can't pre-process it.

This adds an opt-in capture → process → forward path for the child's stderr: the runtime captures it, runs each record through an extensible pipeline, and forwards the survivors to the real stderr. Phase 1 forwards verbatim — the goal here is to get capture mode, ordering, flushing/snappiness, memory bounds, and lifecycle correct. It's the foundation for follow-ups: output cleanup (dedup repeated lines + count), pattern-matched hooks, and hung/locked-up-server detection.

Off by default. With no opt-in, stderr stays Stdio::inherit() and behavior is byte-for-byte unchanged.

How it works

  • Conditional capture mode. When BazelTrait.capture_output is set, bazel_runner.axl builds a processor and decouples --isatty from stdout to match the captured fd:
    • destination stderr is a TTY → allocate a PTY so Bazel keeps its live curses UI (--isatty=1); forward the master bytes near-verbatim.
    • otherwise (CI, redirected) → a plain pipe + --curses=no --isatty=0, so Bazel emits clean newline-terminated lines (the tractable substrate for the deferred dedup/match work).
  • Reader/forwarder thread (stream/output.rs, modeled on the BES stream): splits records on \n and \r (so the \r-driven progress UI stays live), flushes once per read, runs a LineProcessor pipeline (pass-through in phase 1 — the seam where dedup/matchers plug in), and writes to the real stderr. Single producer→consumer with a blocking write, so back-pressure is automatic — deliberately not the unbounded Broadcaster used for structured BES events. Handles Linux EIO-as-EOF, caps the in-flight line buffer, and maintains a last_activity atomic for the deferred stall watchdog.
  • Lifecycle: the OutputStream is joined in Build::wait after the child is reaped, so all forwarded stderr is flushed before the task prints its terminal summary. The PTY slave is dropped right after spawn so the master observes EOF (the classic PTY-hang pitfall).

Deferred to follow-ups (room left, not wired): dedup+count, output_match hooks, and the kill/unhealthy decision — which per design will live in the existing bazel_attempt_end hook.


Changes are visible to end-users: no

  • Searched for relevant documentation and updated as needed: yes (Starlark API docstrings on ctx.bazel.build/.test, bazel.output.processor, and the BazelTrait.capture_output field)
  • Breaking change (forces users to change their own code or config): no
  • Suggested release notes appear below: no

Test plan

  • New test cases added — 7 OutputStream unit tests (record splitting, CR/LF, partial-line flush, carry cap, processor drop) and 2 real-subprocess round-trips through the full capture machinery for both pipe and PTY (these prove no hang at EOF).
  • Covered by existing test cases — 771 AXL unit tests pass; no-opt-in build verified unchanged.
  • Manual testing:
    • Pipe path: ASPECT_CAPTURE_TEST=1 aspect build //examples/deliverable:py_deliverable 2>err.log (via a temp config gate, reverted) — Bazel's stderr forwarded to the file, exit 0, no hang; confirmed --curses=no --isatty=0 via ASPECT_DEBUG=1.
    • PTY path: same under script -q (real TTY) — confirmed --isatty=1, output forwarded through the PTY master, exit 0, no hang.

@gregmagolan gregmagolan force-pushed the capture-bazel-stderr branch from d02a892 to 3a19c27 Compare June 3, 2026 14:47
Spawn Bazel with a runtime-owned stderr instead of Stdio::inherit() when
a task opts in, so its output can be pre-processed before reaching the
terminal. Phase 1 forwards verbatim; the pipeline seam, capture modes,
ordering, flushing, and memory bounds are the foundation for later
output cleanup (dedup + count), pattern-matched hooks, and hung-server
detection.

- stream/output.rs: OutputStream reader thread — \n/\r record splitting,
  per-read flush (keeps the curses progress UI live), LineProcessor
  pipeline seam (pass-through only), EIO-as-EOF, 1 MiB carry cap,
  last_activity atomic (stall-watchdog substrate). Single
  producer->consumer with a blocking write so back-pressure is automatic
  (deliberately not the unbounded Broadcaster).
- capture.rs: conditional capture fd — plain pipe (non-TTY) or PTY
  (interactive, via nix::pty::openpty) with slave-drop / CLOEXEC /
  winsize discipline; pipe fallback off-Unix.
- build.rs: OutputProcessor type + CaptureMode; `output` param on
  Build::spawn overrides the stderr Stdio and starts the OutputStream
  post-spawn; output_stream joined in wait() after the child is reaped.
- mod.rs: `output` arg on ctx.bazel.build/.test; bazel.output.processor()
  constructor + type.
- axl-types: Writable::to_boxed_write() for the forwarder thread.
- bazel.axl: capture_output opt-in on BazelTrait (default off).
- bazel_runner.axl: build/pass the processor; decouple --isatty from
  stdout (--isatty=1 for PTY, --isatty=0 --curses=no for the pipe path).

Off by default: with no opt-in, stderr stays Stdio::inherit() and
behavior is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gregmagolan gregmagolan force-pushed the capture-bazel-stderr branch from 3a19c27 to 4457f55 Compare June 4, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant