chore: move the submission pipeline out to lean-eval-submissions#278
Open
kim-em wants to merge 1 commit into
Open
chore: move the submission pipeline out to lean-eval-submissions#278kim-em wants to merge 1 commit into
kim-em wants to merge 1 commit into
Conversation
This PR removes the hosted submission pipeline from this repository, which now keeps only the benchmark problem set and the comparator/ sandbox integration. The pipeline — issue intake, fetch/evaluate/record scripts, the submission and reconciler workflows, and the stored results — moves to the new leanprover/lean-eval-submissions repository. Delete `submission.yml`, `submission-reconciler.yml`, the submission issue template, `fetch_submission.py`, `evaluate_submission.py`, `update_leaderboard.py`, `reconcile_orphan_submissions.py`, and their Python tests. Drop the three submission test steps from `ci.yml` and fix its Python-syntax-check glob now that `tests/` holds only Lean tests. The security probes (`sandbox_engaged_probe.py`, `security_probes/`) stay: they guard sandbox regressions introduced by changes to this repo, and the submission pipeline runs them from a checkout of this repo. Split `SECURITY.md`: this repo keeps the comparator/sandbox security model (architecture, where untrusted code runs, the trust model, the pin policy, the probes), and the submission-pipeline threat model moves to lean-eval-submissions. Trim `docs/ci-secrets.md` to the `lean-eval-regenerator` App and the `LEADERBOARD_WRITE_TOKEN` used by `notify-leaderboard.yml`. Point `README.md` and `PLAN.md` at the new repository for submissions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kim-em
force-pushed
the
split/move-submission-pipeline-out
branch
from
be11d94 to
e9652a8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR removes the hosted submission pipeline from this repository, which now keeps only the benchmark problem set and the comparator/sandbox integration. The pipeline — issue intake, fetch/evaluate/record scripts, the submission and reconciler workflows, and the stored results — moves to the new
leanprover/lean-eval-submissionsrepository.It deletes
submission.yml,submission-reconciler.yml, the submission issue template,fetch_submission.py,evaluate_submission.py,update_leaderboard.py,reconcile_orphan_submissions.py, and their Python tests; drops the three submission test steps fromci.ymland fixes its Python-syntax-check glob now thattests/holds only Lean tests. The security probes (sandbox_engaged_probe.py,security_probes/) stay — they guard sandbox regressions introduced by changes to this repo, and the submission pipeline runs them from a checkout of this repo.It splits
SECURITY.md(this repo keeps the comparator/sandbox security model; the submission-pipeline threat model moves to lean-eval-submissions), trimsdocs/ci-secrets.mdto thelean-eval-regeneratorApp and theLEADERBOARD_WRITE_TOKENused bynotify-leaderboard.yml, and pointsREADME.md/PLAN.mdat the new repository.Important
Part of a three-repo split. Do not merge before the cutover steps are coordinated:
leanprover/lean-eval-submissionsmust be live and configured, and the leaderboard must read results from it, before old submission intake here is frozen. See the cutover checklist in the planning notes.🤖 Prepared with Claude Code