Skip to content

Add BC PR-Review engine arm with BCQuality ref override#739

Open
gggdttt wants to merge 11 commits into
mainfrom
engine-review-arm
Open

Add BC PR-Review engine arm with BCQuality ref override#739
gggdttt wants to merge 11 commits into
mainfrom
engine-review-arm

Conversation

@gggdttt

@gggdttt gggdttt commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

What

Lands the BC PR-Review engine arm (faithful convergence arm) in BC-Bench and wires it into the CLI.

The engine (microsoft/BC-ALReviewAgent) now owns orchestration + the default BCQuality policy. This PR makes BC-Bench a first-class consumer of that engine, running its production Invoke-CopilotPRReview.ps1 in local mode as the review agent and scoring the result with BC-Bench's own CodeReviewPipeline.

Changes

  • src/bcbench/agent/engine/ (new): the engine adapter
    • agent.pyrun_engine_review(...): commits the patched worktree, fetches+filters BCQuality, invokes the engine, and normalizes al-code-review-findings.json into BC-Bench's flat review.json.
    • fetch_bcquality.ps1 — clones + layer-filters BCQuality at the configured ref.
  • src/bcbench/agent/__init__.py — export run_engine_review.
  • src/bcbench/commands/evaluate.py — new bcbench evaluate engine command.

BCQuality ref override (no engine edits)

run_engine_review accepts optional bcquality_repo / bcquality_ref, passed to the engine's fetch via the BCQUALITY_REPO / BCQUALITY_REF env vars (which the engine's Get-BCQualityConfig already honors on top of its baseline config). This lets a CI run review against a modified BCQuality branch/SHA without editing the engine.

Exposed on the CLI (all env-bound for Actions):

  • --engine-scripts-dir (env ENGINE_SCRIPTS_DIR)
  • --bcquality-repo (env BCQUALITY_REPO)
  • --bcquality-ref (env BCQUALITY_REF)

Not in this PR (follow-ups)

  • A GitHub Actions workflow entry to run the engine arm on CI.
  • Recording the resolved BCQuality SHA + engine ref into ExperimentConfiguration for reproducibility.

wenjiefan added 9 commits July 10, 2026 12:58
Lands the faithful convergence arm (engine adapter + BCQuality fetch/filter + review.json transform) and wires it into the CLI as 'bcbench evaluate engine'. The adapter accepts optional bcquality_repo/bcquality_ref that are passed to the engine's fetch via BCQUALITY_REPO/BCQUALITY_REF env vars, so a CI run can review against a modified BCQuality branch/SHA without editing the engine.
…oggle

Adds CODE_REVIEW_BCQUALITY_REF (default empty) to the existing 'Evaluation with GitHub Copilot' workflow. Empty keeps the current Copilot code-review; a branch that sets it runs the code-review through the BC PR-Review engine arm against that BCQuality ref. No new workflow or command surface for the user - flip one committed env on a branch.
Comment thread src/bcbench/commands/evaluate.py Fixed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant