Feat/koth engine lane - #569
Merged
Merged
Conversation
the contributor-onboarding half of the beat-ditto plan: ditto's mining loop (tweak a retrieval kit, get scored, dethrone the champion, earn while on top) rebuilt on github primitives. stacks on the bench from the session-answer-mode branch. the ladder surface is one schema-validated data file, competition/kits/current/kit.yaml, which is the whole retrieval config of the throwaway bench kb. a PR touching only that file is scored by vouchbench, paired per seed against the reigning kit; a dethrone over max(0.007, 1.96 x paired se) enables github auto-merge. the review gate stays load-bearing. code never auto-merges - the ladder surface is data, closed-world allowlisted (config can name executables, so unknown keys are an error). and the trunk is never auto-written - auto-merge fires only for PRs whose base is a dedicated non-shipped ladder branch (repo var koth_ladder_base); a kit-only PR against main is scored then merged by a human. promoting a champion into shipped defaults is a periodic human PR. the daily ladder is provisional by design: public seeds (base sha + utc date) make it offline-reproducible and thus overfittable, so payout rank is settled by the monthly sealed commit-reveal run, not the daily throne. hardened against an adversarial review: the empty/oversized-kit contents- api bypass, a stray 1-hour real sleep that out-slept the ci job, the auto-merge-to-trunk gate bypass (now ladder-branch-only), band calibration (12 paired seeds, z=1.96), the draft-pr trigger gap, and a branch-protection command that 422'd on a stringified boolean.
the kit's own comment still read 1.64; the band is 1.96 everywhere else (koth_score, docs, leaderboard). also note auto-merge is ladder-branch scoped.
benchmark v2 grows the axis recall-only benchmarks cannot measure: citation-correctness (the surfaced answer must be spelled by a quote whose byte-offset receipt verifies), receipt-coverage (fraction of surfaced claim items carrying a verifying receipt), and supersede-hygiene (a landed update must retire the stale value from the live claim set). the first two are guards that score 1.00 stock; the third is a lever at 0.00 alongside knowledge-update, pointing at lifecycle-driven supersession. memory_contract.MemoryContract implements ditto's five mining-contract tools (save, search, search subjects, fetch by id, search in subject) over the real store, so a shared-contract head-to-head can run both engines on one task set. saves route through the receipt-gated capture loop and return nothing durable when the gate is off — the review gate is never silently bypassed, even inside a harness. reference baseline refreshed to 0.52 +- 0.03 over seeds 1-6 (the category set change is a bench-contract version bump).
the verifiability categories change the category set, so seeds 1-6 rescore: composite 0.52 +- 0.03 replaces 0.57 +- 0.04 as row 0. the koth fold compares only max-bench-version scores, so the reigning-kit number and the challenger number stay on the same contract.
…dbox the engine-submission half of the koth ladder, and the real ditto equivalent: contributors submit a retrieval STRATEGY as python code (a rank(query, candidates, *, limit) -> ids function under contrib/strategies/), and vouchbench scores it. the kit lane only tuned coefficients; this is where a new fusion or reranker actually competes. the load-bearing rule: a kit is data and auto-merges; a strategy is code and NEVER auto-merges. vouch is a library people install, so a winning strategy earns the leaderboard and payout and ships only after human review merges it as a default - the review gate applied to engine code. scoring runs untrusted code, so vouch.strategy.run_sandboxed executes each submission in a python -I child that sets rlimits and installs an audit hook blocking network, subprocess, os.exec/fork, ctypes.dlopen, and filesystem writes before importing the file; the parent enforces a wall-clock timeout and treats any failure as no-reordering. the engine hook in build_context_pack is off by default and byte-identical when unset (retrieval.strategy, a dotted path, wires a merged strategy). koth-engine-gate.yml runs pull_request_target with contents:read, no secrets, no write token, no merge. hardened against an adversarial review. two confirmed defects fixed with regression tests: the audit hook read its blocklists from module globals (disarmable in two lines by reassigning __main__ globals) - now bound in closure cells; and the result travelled on the strategy's own stdout (any print corrupted it, silently scoring a good strategy as a no-op) - the child now writes the result to a private dup of stdout with fd 1 pointed at /dev/null. the docstring is honest that in-process python isolation is best-effort and the real boundary is the ephemeral runner plus human merge.
vouch bench run gains --strategy (score a sandboxed ranking strategy exactly like ci) and --against (paired dethrone verdict vs a champion strategy before pushing a pr). run_seeds threads the strategy through. the dethrone test now lives once, in bench.paired_verdict, with the floor and z constants exported; both ci scorers call it instead of carrying their own copies, so the margin math cannot drift between the local practice loop and the gates.
mining-on-vouch.md is the canonical from-fork-to-shipped walkthrough: fork, practice locally with the exact ci scoring loop, open a pr, read the scorecard, get paid by the season split — and the carrot ditto cannot offer, a reviewed winner ships as the default in every install. the kit ladder is repositioned as the 10-minute on-ramp (its ceiling is low by construction; the bench's zero categories are only reachable from code), the strategy lane doc gets the headline and the local practice command, and the readme points at the walkthrough.
every dethrone that lands on the ladder branch now writes its own LEADERBOARD.md row instead of waiting for the maintainer at season close. the scoring gates stay read-only by design; the ledger workflow runs only on push to the ladder branch, after the merge, from base-branch code, so an untrusted pr can never reach its write token. the row comes from the gate's own scorecard comment (comments by any other author are ignored, so a scorecard-shaped comment cannot spoof a row), and update_leaderboard.py is idempotent per pr, so the workflow triggering on its own ledger push converges instead of looping.
… events round 1 (pr #566) proved the gap live: the gate arms auto-merge with the workflow GITHUB_TOKEN, and github suppresses workflow triggers for pushes caused by that token, so the merge push never reached the ledger. the workflow now sweeps recently merged ladder prs (schedule + manual dispatch + the still-working push path for human merges); the appender stays idempotent per pr, so overlapping sweeps converge.
the ladder's required gate check applies to direct pushes too, so the bot's row append was rejected (round 1 proved it). protection moved from classic branch protection to the koth-ladder-gate ruleset (same gate requirement, strict), and the ledger checkout uses an owner token with admin bypass. the workflow never runs on pr events and never executes untrusted code, so the token stays out of untrusted reach.
with a ranking strategy active, retrieval fetched exactly `limit` hits before the strategy ran, so ordering could never change pack membership - and the bench grades presence, not order, which made every strategy score byte-identical to baseline (measured: 12 seeds, diff 0.0000). retrieval now over-fetches a bounded pool (5x limit, floor 50) when a strategy is active and the top `limit` of the strategy's order survive the cut. de-prioritising below the window excludes the candidate; a strategy still cannot fabricate a result or shrink the pack. contract language in the lane doc and baseline docstring updated to match, plus a regression test that a demoted candidate leaves the pack.
open contributors kept needing the lane rules from three scattered docs. contributing.md now states the two pr shapes the gates enforce: engine lane = one new file under contrib/strategies/ against koth-ladder, sandboxed, scored, never auto-merged, human review for benchmark-keyed logic; kit lane = kit.yaml only, auto-merge on dethrone, low ceiling by construction. includes the local ci-exact practice command and the reproducibility and payout pointers.
workflow lint only runs on prs that touch workflows, so the koth gates' pull_request_target findings never surfaced until the engine-gate registration pr woke the linter. both gates now carry the ignore annotation with the justification inline (base-branch code only, pr content handled as data, read-only tokens), and the scorecard comment script declares its literal backticks for shellcheck.
|
Warning Review limit reached
Next review available in: 41 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (90)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Why
What might break
VEP
Tests
make checkpasses locally (lint + mypy + pytest)CHANGELOG.mdupdated under## [Unreleased]