TEMP: scratch workflow for investigate-prompt iteration - #662
TEMP: scratch workflow for investigate-prompt iteration #662JakeSCahill wants to merge 3 commits into
Conversation
…re prompt Deliberately breaks a real selector to produce a realistic failure, then runs the same Claude-investigation logic being developed in PR #651. Delete once a good prompt is found. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
✅ Deploy Preview for rp-cloud ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
📝 WalkthroughWalkthroughAdds a temporary GitHub Actions workflow with manual and pull-request triggers. The workflow installs dependencies, creates a preprod environment, and changes a selector to cause a controlled Doc Detective failure. It runs the inline tests and, on failure, invokes Claude to reproduce, classify, and resolve the failure or print a diagnosis. Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant DocDetective
participant Claude
participant GitHubRepository
GitHubActions->>DocDetective: Run inline tests
DocDetective-->>GitHubActions: Report expected failure
GitHubActions->>Claude: Send repository and run context
Claude->>DocDetective: Reproduce and classify failure
Claude->>GitHubRepository: Create validated labeled PR when applicable
Claude-->>GitHubActions: Print diagnosis otherwise
Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
JakeSCahill
changed the title
|
@CodeRabbit review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/_temp-test-investigate-prompt.yml:
- Around line 11-15: Remove the pull_request trigger and its paths configuration
from the workflow’s on block, leaving only workflow_dispatch so the temporary
workflow runs exclusively when manually dispatched.
- Around line 68-101: The heredoc content in the workflow is not indented within
the `run: |` block, causing GitHub Actions to parse it as YAML. In the workflow
step containing the `cat > /tmp/investigate-prompt.md <<PROMPT` command, indent
every heredoc line from the prompt text through the closing `PROMPT` marker to
the run-block indentation, while preserving the heredoc contents.
- Around line 49-52: Update the workflow’s ephemeral mutation and subsequent
verification flow around the deliberate selector break so a rerun that restores
the repository content is accepted as a verified local repair. Remove any
requirement to commit the restoration or create a GitHub pull request, while
preserving checks that confirm the working tree is clean and the base content
has been restored.
- Around line 103-107: Install Claude Code in the workflow before the invocation
containing claude -p, since the current npm ci step does not provide the claude
executable. Add the required installation step or command using the supported
`@anthropic-ai/claude-code` package, then preserve the existing command and
options.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 326b86bc-7b8b-45a6-bb84-8bd4913334cb
📒 Files selected for processing (1)
.github/workflows/_temp-test-investigate-prompt.yml
micheleRP
left a comment
micheleRP
left a comment
There was a problem hiding this comment.
Review: TEMP scratch workflow for investigate-prompt iteration
Status: the workflow has never executed a single step. Head is still 6c64bcaf, and all three recorded runs are 0-second startup failures ("This run likely failed because of a workflow file issue"). No run was ever created for the PR itself.
I reviewed this independently of CodeRabbit, then reconciled the two passes. We agree on three items, CodeRabbit found one I missed, and I found one it missed. Net: 5 issues, and this needs a rebase plus a re-indent before it can do anything at all.
Confirmed by both passes
1. The YAML is invalid (CodeRabbit: Critical). Line 69 starts outside run: |, so lines 69 through 101 parse as YAML and GitHub cannot load the workflow. CodeRabbit backs this with actionlint ("could not parse as YAML: did not find expected key", line 72) and YAMLlint ("syntax error: expected <block end>, but found -", line 73). That matches my own parse exactly: expected <block end>, but found '-' ... line 73, column 1. Three independent tools, same conclusion.
Fix (verified): indent lines 69 through 101, the heredoc body and the closing PROMPT, to the run: block indentation. That is the whole change. YAML strips the block scalar's common indentation, so bash still receives the heredoc at column 0 and the ${ENVIRONMENT} / ${RUN_URL} expansions and \$HOME escapes keep working. I confirmed that round-trip with a parser rather than reasoning about it.
Longer term, moving the prompt into a committed file (tests/doc-detective/investigate-prompt.md) plus envsubst would make it reviewable on its own and reusable by the real workflow, which is where it's ultimately headed.
2. claude is not installed (CodeRabbit: Major). The workflow only runs npm ci, package.json doesn't include @anthropic-ai/claude-code, and node_modules/.bin isn't on PATH in a run: step anyway. CodeRabbit's proposed global install step plus a command -v claude guard, pinned to an approved version, is the right shape. This one carries over to the real workflow whenever the prompt graduates into it.
Related and unverified: once the CLI is actually installed, check that --effort, --max-budget-usd, --output-format json, and --permission-mode bypassPermissions are all valid for the pinned version. They're currently untested, since the step has never run.
3. Remove the pull_request trigger (CodeRabbit: Minor). 3b1d4958 added it and 6c64bcaf added workflow_dispatch back without removing it, so the PR description ("workflow_dispatch-only ... no schedule, no PR trigger") no longer matches the file.
I'd rate this higher than Minor. CodeRabbit frames it as objective drift. The substance is that permissions: contents/pull-requests/issues: write plus persist-credentials: true plus claude --permission-mode bypassPermissions means that once the YAML is fixed, an autonomous agent holding a write-capable token runs automatically on any same-repo PR touching that path. Fork PRs get no secrets so they'd fail harmlessly, but this is a permissions question rather than a description-accuracy one.
Only CodeRabbit found this one
4. A correct repair leaves no diff, so the required PR cannot be created (CodeRabbit: Major). This is a real design flaw and I missed it. I verified the logic before endorsing it:
The sed rewrites "elementText":"Add input" to "Add Input Source" in the ephemeral checkout only. The correct diagnosis is that the selector text doesn't match the UI, and the minimal correct fix is to change it back to "Add input", which restores the file to its committed state byte for byte. git status is then clean, git commit has nothing to commit, and gh pr create has no diff to open a PR from.
Meanwhile the prompt's path A demands "Create a branch, commit exactly the files that needed to change ..., push it, and open a PR", and closes with "Produce exactly one outcome: a PR, or a printed diagnosis. Not both, not neither." So an agent that diagnoses and fixes perfectly is handed an impossible instruction. The likely failure modes are worse than an error: it either gives up, or invents an unrelated change so it has something to commit, which is exactly the behavior the prompt's other guardrails work hard to prevent.
That matters because the PR is this harness's success signal, so as written the signal can't fire. CodeRabbit's suggestion, accepting "restored to base content, rerun passes, tree clean" as the verified-repair outcome instead, looks right to me. Note this only bites after finding 5 is fixed: on the current branch the sed matches nothing at all.
Only I found this one
5. The branch doesn't contain the framework it tests. git ls-tree origin/temp-investigate-prompt-test shows main plus the one workflow file. There is no tests/doc-detective/ directory, and connect-quickstart.adoc has no inline test comments. So even with valid YAML, in order:
- "Create .env file" redirects into a directory that doesn't exist, and the step fails.
sedmatches nothing, thengrep -n "Add Input Source"exits 1 and fails the step.node tests/doc-detective/run-inline.jsdoesn't exist.
Fix: branch this from doc-2235-visual-pipeline-editor rather than main.
Also worth changing
The preprod cluster ID (d9f1nf0d6v8fd1orsa00) is hardcoded in a public repo, same point as #651. Moving it to a repo variable keeps org resource identifiers out of the open and makes a recreated cluster a variable edit.
And if this is ever merged rather than dispatched from the branch, a pull_request-triggered agent workflow lands on main. The description already commits to deleting it, so keeping it unmerged and dispatching from the branch is the safer shape.
The prompt itself is good
The transient / real-and-fixable / real-but-not-fixable-here trichotomy, "run it two more times before calling it transient", the explicit ban on widening maxVariation or deleting steps to force a pass, requiring the agent to watch the test pass before opening a PR, and --max-budget-usd 3 are all the right guardrails. Finding 4 is a flaw in the harness around the prompt, not in the prompt's judgment criteria. Once the blockers are cleared this should give a genuine signal.
|
Closing this without landing the review fixes. This was always a scratch harness ("delete this file once a good prompt is found"), and two things make fixing it the wrong move:
All four review findings were genuinely valid (the file did not even parse as YAML — the heredoc block-scalar indentation broke it at line 73). They are moot now that the file is not landing. If the investigate-prompt work resumes, it should start from the branch that actually contains |
2 participants
Temporary, workflow_dispatch-only scratch workflow for iterating on the "Investigate failure with Claude" prompt being developed in #651. Not a real change to any existing workflow — no schedule, no PR trigger, doesn't touch the real cron.
Deliberately breaks a real selector on an ephemeral checkout to produce a realistic failure, then runs the same investigate-prompt logic so we can judge diagnosis/fix quality via real
gh workflow runexecutions.Delete this file (and this branch) once a good prompt is found — this is not meant to be a lasting addition to
main.