Add BC PR-Review engine arm with BCQuality ref override#739
Open
gggdttt wants to merge 11 commits into
Open
Conversation
added 9 commits
July 10, 2026 12:58
Lands the faithful convergence arm (engine adapter + BCQuality fetch/filter + review.json transform) and wires it into the CLI as 'bcbench evaluate engine'. The adapter accepts optional bcquality_repo/bcquality_ref that are passed to the engine's fetch via BCQUALITY_REPO/BCQUALITY_REF env vars, so a CI run can review against a modified BCQuality branch/SHA without editing the engine.
…oggle Adds CODE_REVIEW_BCQUALITY_REF (default empty) to the existing 'Evaluation with GitHub Copilot' workflow. Empty keeps the current Copilot code-review; a branch that sets it runs the code-review through the BC PR-Review engine arm against that BCQuality ref. No new workflow or command surface for the user - flip one committed env on a branch.
…drop separate engine command
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Lands the BC PR-Review engine arm (faithful convergence arm) in BC-Bench and wires it into the CLI.
The engine (
microsoft/BC-ALReviewAgent) now owns orchestration + the default BCQuality policy. This PR makes BC-Bench a first-class consumer of that engine, running its productionInvoke-CopilotPRReview.ps1in local mode as the review agent and scoring the result with BC-Bench's ownCodeReviewPipeline.Changes
src/bcbench/agent/engine/(new): the engine adapteragent.py—run_engine_review(...): commits the patched worktree, fetches+filters BCQuality, invokes the engine, and normalizesal-code-review-findings.jsoninto BC-Bench's flatreview.json.fetch_bcquality.ps1— clones + layer-filters BCQuality at the configured ref.src/bcbench/agent/__init__.py— exportrun_engine_review.src/bcbench/commands/evaluate.py— newbcbench evaluate enginecommand.BCQuality ref override (no engine edits)
run_engine_reviewaccepts optionalbcquality_repo/bcquality_ref, passed to the engine's fetch via theBCQUALITY_REPO/BCQUALITY_REFenv vars (which the engine'sGet-BCQualityConfigalready honors on top of its baseline config). This lets a CI run review against a modified BCQuality branch/SHA without editing the engine.Exposed on the CLI (all env-bound for Actions):
--engine-scripts-dir(envENGINE_SCRIPTS_DIR)--bcquality-repo(envBCQUALITY_REPO)--bcquality-ref(envBCQUALITY_REF)Not in this PR (follow-ups)
ExperimentConfigurationfor reproducibility.