feat(trail): add entire trail tune to tailor runner prompts to the repo#1506
feat(trail): add entire trail tune to tailor runner prompts to the repo#1506Soph wants to merge 2 commits into
entire trail tune to tailor runner prompts to the repo#1506Conversation
The shipped risk/confidence/review-focus runner templates were written for a generic web/backend app (payments, DB migrations, TypeScript). Rewrite them around what actually makes changes risky in this Go CLI: - risk: destructive git ops, hook handlers, checkpoint/session integrity, transcript egress, blast radius; score bands re-anchored accordingly. - confidence: Go test coverage, isolation hygiene, mise/golangci gates and the Vogon e2e canary instead of the TypeScript type-safety dimension. - review-focus: hotspot list realigned to the same CLI risk surface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Entire-Checkpoint: 55f8c01f7de4
…repo
`entire trail tune [<runner>]` gathers signal about the current repo across
four best-effort, gracefully-degrading tiers — repo docs/structure, merged PRs
and issues (via gh), checkpoint churn hotspots, and past trail findings — and
produces a prompt that rewrites the .entire/runners/*.json templates so their
dimensions and score bands fit this repo instead of the generic defaults.
By default it prints the prompt for pasting into an agent. With --run it
executes the prompt headlessly through the configured summary provider and
surgically rewrites only each runner's prompt.template via byte-level
replacement, leaving all other fields and formatting byte-for-byte intact
(minimal git diff; files are git-tracked so the user reviews via git diff).
The CLI has no runner struct/loader (the backend consumes these files and
substitutes {{placeholders}}), so runners are treated as opaque text.
Unit tests cover output parsing, surgical template replacement, runner
loading/filtering, source-flag parsing, and prompt assembly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 34f054b55a0d
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 7752487. Configure here.
| }) | ||
| if err != nil { | ||
| return skip("trails unavailable: " + oneLine(err.Error())) | ||
| } |
There was a problem hiding this comment.
Cancellation swallowed during signal gather
Medium Severity
When the user cancels during gatherTuningContext, errors including context.Canceled from gh, checkpoints, or the trails API are turned into _skipped: …_ brief text instead of aborting. runTrailTune then prints the prompt or runs --run as if gathering succeeded, ignoring the cancellation.
Additional Locations (3)
Reviewed by Cursor Bugbot for commit 7752487. Configure here.
| out, err := textGen.GenerateText(ctx, prompt, provider.Model) | ||
| if err != nil { | ||
| return fmt.Errorf("agent run failed: %w", err) | ||
| } |
There was a problem hiding this comment.
Agent cancel not silent error
Low Severity
If the user interrupts --run while GenerateText is running, the failure is returned as fmt.Errorf("agent run failed: %w", err) instead of NewSilentError, so Cobra prints a noisy Error: agent run failed: context canceled after Ctrl+C.
Triggered by learned rule: Map context.Canceled to NewSilentError on user cancellation
Reviewed by Cursor Bugbot for commit 7752487. Configure here.
There was a problem hiding this comment.
Pull request overview
Adds a new entire trail tune [<runner>] subcommand that gathers repo-specific signal and generates (or applies) rewritten evaluator prompt.template strings for .entire/runners/*.json, making trail scoring prompts tailored to the current repository (this Go CLI) rather than generic web-app defaults.
Changes:
- Introduces
trail tunecommand with--sources,--limit, and--runto print a tuning prompt or apply model output headlessly via the configured summary provider. - Implements runner discovery + prompt assembly (with untrusted-data JSON framing) and safe application logic (JSON-object output parsing, placeholder-set preservation, surgical byte-level template replacement).
- Retunes this repo’s shipped
trail-risk,trail-confidence, andtrail-review-focusrunner templates for the CLI’s actual risk surface (git ops, hooks, checkpoints, transcript egress).
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| cmd/entire/cli/trail_tune_cmd.go | New trail tune command wiring; prompt generation vs --run apply path. |
| cmd/entire/cli/trail_tune_gather.go | Best-effort signal gathering from repo statics, gh, checkpoints, and trails. |
| cmd/entire/cli/trail_tune_prompt.go | Builds the model instruction prompt with JSON-embedded untrusted data blocks. |
| cmd/entire/cli/trail_tune_apply.go | Parses model output, validates placeholders, and surgically replaces prompt.template bytes. |
| cmd/entire/cli/trail_tune_prompt_test.go | Tests runner loading, source parsing, prompt content, and injection-breakout framing. |
| cmd/entire/cli/trail_tune_apply_test.go | Tests output parsing, surgical replacement, and placeholder validation. |
| cmd/entire/cli/trail_cmd.go | Registers the new tune subcommand under entire trail. |
| .entire/runners/trail-risk.json | Updates risk evaluator template for Entire CLI-specific risk factors. |
| .entire/runners/trail-confidence.json | Updates confidence evaluator template to reflect Go CLI testing/CI patterns. |
| .entire/runners/trail-review-focus.json | Updates review-focus template to point reviewers at CLI-specific hotspots. |
| fmt.Fprintf(&b, "- Top-level dirs: %s\n", strings.Join(dirs, ", ")) | ||
| } | ||
| } | ||
| b.WriteString("\n") |
| s := string(data) | ||
| if len(s) > maxLen { | ||
| s = s[:maxLen] + "\n…(truncated)…" | ||
| } | ||
| return s, true |
https://entire.io/gh/entireio/cli/trails/648
What
Adds
entire trail tune [<runner>]— a command that tailors the.entire/runners/*.jsonevaluator prompts (risk / confidence / drift / review-focus …) to this repository instead of the generic web-app defaults they ship with. Also rewrites this repo's own risk/confidence/review-focus runner prompts as the first consumer.Why
The shipped runner templates are written for a generic web/backend app (payments, DB migrations, TypeScript). For a Go CLI that manipulates the user's git repo, the real risk surface is destructive git ops, hook handlers, checkpoint/session integrity, and transcript egress. Calibrating the prompts by hand (as done here for this repo) is exactly the kind of thing that should be automatable for any repo adopting trails.
How it works
tunegathers signal about the current repo across four best-effort, gracefully-degrading tiers:gh pr/issue liststrategy.ListCheckpoints→ churn hotspotsIt then assembles a prompt that rewrites each
prompt.templateto fit the repo.--print): emits a paste-ready prompt.--run: executes the prompt headlessly through the configured summary provider and rewrites the runner files surgically — only eachprompt.templatevalue is swapped (via byte-level replacement), leaving every other field and the file formatting byte-for-byte intact, so thegit diffis scoped to the prompt change. Files are git-tracked, so the user reviews viagit diff.The CLI has no runner struct/loader (the backend consumes these files and substitutes
{{placeholders}}), so runners are treated as opaque text.Safety / hardening
{{placeholder}}set exactly (no drops, no invented placeholders).{}proposal is a clean no-op (exit 0); proposals that are all rejected/out-of-scope return an error.--insecure-http-authis threaded through to the trails API client.Tests
Unit tests cover output parsing (incl. fenced/prose-wrapped/
{}), surgical template replacement (special-char preservation), placeholder validation (drop + invent), runner loading/filtering, source-flag parsing, prompt assembly, and an injection-breakout regression. Build + lint clean.🤖 Generated with Claude Code
Note
Medium Risk
--runcan rewrite committed.entire/runnersJSON from model output (mitigated by placeholder checks and surgical edits), and the updated eval prompts change how future trails score branches.Overview
Adds
entire trail tune [<runner>], which collects repo-specific signal (docs/layout,ghPRs/issues, checkpoint churn, trail review findings) and either prints a paste-ready rewrite prompt or, with--run, drives the configured summary provider to update only each runner’sprompt.templatevia byte-level JSON replacement (other fields and formatting unchanged).--sourcesand--limitcontrol which signal tiers are included; tiers degrade gracefully whengh, checkpoints, or trails are unavailable. Before writes, rewritten templates must keep the original{{placeholder}}set; gathered signal and existing templates are embedded as JSON with explicit untrusted-data framing to reduce prompt injection.Shipped risk, confidence, and review-focus runner prompts in
.entire/runners/are retuned for this Go CLI (destructive git ops, hooks, checkpoints, transcript egress) as the first in-repo consumer. Unit tests cover parsing, surgical replacement, validation, loading/filtering, and injection-breakout behavior.Reviewed by Cursor Bugbot for commit 7752487. Configure here.