feat(EC-1862): add AI skills for ec-cli - #3434
Conversation
📝 WalkthroughWalkthroughSix new Claude skill documents provide Conforma CLI guidance for test development, test execution, failure debugging, builds, benchmarks, and pull request preparation. ChangesConforma CLI development skills
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
🤖 Review · |
PR Summary by QodoAdd repo-specific Claude AI skills for ec-cli workflows
AI Description
Diagram
High-Level Assessment
Files changed (6)
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/skills/build-and-lint/SKILL.md:
- Around line 29-30: Update the lint commands in the build-and-lint skill to
invoke golangci-lint through the repository-pinned tooling defined by
tools/go.mod rather than an unqualified global binary. Replace the single-file
example with an appropriate package path that provides reliable coverage, and
keep the gofmt check aligned with that target.
In @.claude/skills/pr-checklist/SKILL.md:
- Around line 21-24: Update the commit-message example fenced block in SKILL.md
to include the text language identifier, preserving its existing example
content.
In @.claude/skills/run-tests/SKILL.md:
- Around line 67-70: Update the fenced /etc/hosts blocks in
.claude/skills/run-tests/SKILL.md lines 67-70 and
.claude/skills/debug-failure/SKILL.md lines 38-41 to declare the text language.
No other content changes are needed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 86a00619-00a3-4f91-a273-6a3a0a7bee66
📒 Files selected for processing (6)
.claude/skills/benchmark/SKILL.md.claude/skills/build-and-lint/SKILL.md.claude/skills/debug-failure/SKILL.md.claude/skills/pr-checklist/SKILL.md.claude/skills/run-tests/SKILL.md.claude/skills/write-tests/SKILL.md
Code Review by Qodo
Context used✅ Compliance rules (platform):
27 rules 1.
|
![qodo-for-conforma[bot]](https://avatars.githubusercontent.com/in/3609154?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
🤖 Finished Review · ✅ Success · Started 11:18 AM UTC · Completed 11:35 AM UTC |
ReviewFindingsMedium
Previous runReviewFindingsHigh
Medium
Low
Previous run (2)ReviewFindingsMedium
Low
Previous run (3)ReviewFindingsMedium
Low
Labels: PR adds AI agent skill files under .claude/skills/ |
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fullsend-ai-review
Bot
added
requires-manual-review
|
🤖 Finished Review · ✅ Success · Started 12:31 PM UTC · Completed 12:47 PM UTC |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
These are all good ideas for skills, but are written more as how-to guides than skills. Skills should be reusable workflows. Here is an example of the ---
name: pr-checklist
description: >
Verify a Conforma CLI pull request is ready for review. Use when users ask
"is this PR ready", "definition of done", "PR checklist", "before merging",
"review checklist", or when preparing a PR for review.
---
# Verify PR Readiness for Conforma CLI
Run through each step below, report pass/fail for each, and summarize what remains.
## Step 1: Check for uncommitted generated code
```bash
make generate
git diff --exit-codeIf there are uncommitted changes after Step 2: Check module tidinessRun go mod tidy
git diff --exit-code go.mod go.sumIf acceptance/ or tools/ files were modified, repeat in those directories: cd acceptance && go mod tidy && git diff --exit-code go.mod go.sum
cd tools && go mod tidy && git diff --exit-code go.mod go.sumStep 3: Run lintmake lintZero warnings required. If there are failures, run Step 4: Run testsmake testIf acceptance-relevant code changed (features/, acceptance/, CLI commands), also run: make acceptanceIf behavior changed, check whether snapshots need updating: git diff features/__snapshots__/If snapshots are stale, update them with Step 5: Verify build tags on new test filesFind any new test files in the diff and verify each starts with a build tag ( git diff --name-only --diff-filter=A HEAD~1 | grep '_test\.go$'Read the first line of each new test file and flag any missing build tags. Step 6: Check PR descriptionIf a PR already exists for this branch, fetch its description and verify it covers:
Compare against the template in Step 7: ReportProduce a summary table:
List any items that need attention before the PR is ready for review. |
|
🤖 Review · |
|
🤖 Review · |
0f51eff to
83e15ea
Compare
|
🤖 Finished Review · ✅ Success · Started 9:23 AM UTC · Completed 9:43 AM UTC |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fullsend-ai-review
Bot
removed
the
requires-manual-review
Add 6 reusable workflow skills under .claude/skills/ following the pattern established in conforma/e2e-tests (EC-1929). Each skill is a step-by-step executable workflow with bash commands and pass/fail reporting. Skills: run-tests, debug-failure, build-and-lint, pr-checklist, write-tests, benchmark. Also fixes AGENTS.md: correct --debug claim (only EC_DEBUG=1 preserves temp dirs) and remove conventional commits enforcement.
83e15ea to
ba2469b
Compare
|
🤖 Finished Review · ✅ Success · Started 11:21 AM UTC · Completed 11:38 AM UTC |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/skills/build-and-lint/SKILL.md:
- Around line 34-38: Update the generation verification instructions in
.claude/skills/build-and-lint/SKILL.md (lines 34-38) and
.claude/skills/pr-checklist/SKILL.md (lines 15-20) to check only expected
generated outputs or compare repository status before and after generation,
rather than using an unscoped git diff. Remove all instructions to stage or
commit generated changes in both sites.
- Around line 76-79: Use independent subshells for the acceptance and tools Go
module checks in both .claude/skills/build-and-lint/SKILL.md lines 76-79 and
.claude/skills/pr-checklist/SKILL.md lines 26-36, so each cd starts from the
original working directory and both go mod tidy/git diff checks execute
correctly.
- Around line 62-65: Update the module-detection command in the build-and-lint
skill to inspect the PR branch diff range, matching the range used by the other
skills, rather than relying only on unstaged git diff output. Ensure changed
acceptance/ and tools/ modules are detected on clean multi-commit branches,
while preserving separate handling of any local modifications if required.
In @.claude/skills/run-tests/SKILL.md:
- Around line 62-72: Update the direct acceptance test commands in the run-tests
instructions so each runs independently in a subshell, using the acceptance
module’s current package target: `(cd acceptance && go test . ...)`. Apply this
consistently to the persist, restore, and tag variants, preserving their
existing arguments.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: ea281057-548c-40de-a492-e6b9f1134cfa
📒 Files selected for processing (7)
.claude/skills/benchmark/SKILL.md.claude/skills/build-and-lint/SKILL.md.claude/skills/debug-failure/SKILL.md.claude/skills/pr-checklist/SKILL.md.claude/skills/run-tests/SKILL.md.claude/skills/write-tests/SKILL.mdAGENTS.md
🚧 Files skipped from review as they are similar to previous changes (3)
- .claude/skills/benchmark/SKILL.md
- .claude/skills/write-tests/SKILL.md
- .claude/skills/debug-failure/SKILL.md
| make generate | ||
| git diff --exit-code | ||
| ``` | ||
|
|
||
| If there are changes, stage and commit them. CI will fail if generated code is out of sync. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Avoid mutating the repository from an unscoped generation diff.
Both workflows can include unrelated edits in git diff and omit untracked generated files, then instruct the agent to stage changes. Restrict verification to expected generated outputs or compare status before/after, and do not stage or commit automatically.
.claude/skills/build-and-lint/SKILL.md#L34-L38: remove the automatic staging/commit instruction and scope the generation check..claude/skills/pr-checklist/SKILL.md#L15-L20: remove the automatic staging instruction and scope the generation check.
📍 Affects 2 files
.claude/skills/build-and-lint/SKILL.md#L34-L38(this comment).claude/skills/pr-checklist/SKILL.md#L15-L20
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/build-and-lint/SKILL.md around lines 34 - 38, Update the
generation verification instructions in .claude/skills/build-and-lint/SKILL.md
(lines 34-38) and .claude/skills/pr-checklist/SKILL.md (lines 15-20) to check
only expected generated outputs or compare repository status before and after
generation, rather than using an unscoped git diff. Remove all instructions to
stage or commit generated changes in both sites.
| Identify which modules have changes: | ||
|
|
||
| ```bash | ||
| git diff --name-only | grep -E '^(acceptance/|tools/)' || echo "root module only" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Use the branch diff when identifying changed modules.
git diff --name-only only reports local unstaged changes, so a clean working tree on a multi-commit PR incorrectly reports “root module only” and skips acceptance/ or tools/ tidiness checks. Use the PR range, consistent with the other skills, and separately account for local modifications if needed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/build-and-lint/SKILL.md around lines 62 - 65, Update the
module-detection command in the build-and-lint skill to inspect the PR branch
diff range, matching the range used by the other skills, rather than relying
only on unstaged git diff output. Ensure changed acceptance/ and tools/ modules
are detected on clean multi-commit branches, while preserving separate handling
of any local modifications if required.
| cd acceptance && go mod tidy && git diff --exit-code go.mod go.sum | ||
|
|
||
| # If tools/ files changed | ||
| cd tools && go mod tidy && git diff --exit-code go.mod go.sum |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Use independent shells for each module command.
In both workflows, the second cd is evaluated relative to the first module directory, so the tools check is skipped or fails.
.claude/skills/build-and-lint/SKILL.md#L76-L79: wrap theacceptanceandtoolscommands in separate subshells..claude/skills/pr-checklist/SKILL.md#L26-L36: wrap theacceptanceandtoolscommands in separate subshells.
📍 Affects 2 files
.claude/skills/build-and-lint/SKILL.md#L76-L79(this comment).claude/skills/pr-checklist/SKILL.md#L26-L36
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/build-and-lint/SKILL.md around lines 76 - 79, Use independent
subshells for the acceptance and tools Go module checks in both
.claude/skills/build-and-lint/SKILL.md lines 76-79 and
.claude/skills/pr-checklist/SKILL.md lines 26-36, so each cd starts from the
original working directory and both go mod tidy/git diff checks execute
correctly.
| These require `go test` directly, not `make`: | ||
|
|
||
| ```bash | ||
| # Keep test containers running after failure | ||
| cd acceptance && go test ./acceptance -args -persist | ||
|
|
||
| # Reattach to persisted containers | ||
| cd acceptance && go test ./acceptance -args -restore | ||
|
|
||
| # Run with specific tags | ||
| cd acceptance && go test ./acceptance -args -tags=@focus |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Make each direct acceptance command independently runnable.
This block fails when executed sequentially: after line 66 changes into acceptance, the later cd acceptance commands target acceptance/acceptance. Also, from the acceptance module, go test ./acceptance targets the wrong package. Use subshells and go test ., for example (cd acceptance && go test . -args -persist).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/run-tests/SKILL.md around lines 62 - 72, Update the direct
acceptance test commands in the run-tests instructions so each runs
independently in a subshell, using the acceptance module’s current package
target: `(cd acceptance && go test . ...)`. Apply this consistently to the
persist, restore, and tag variants, preserving their existing arguments.
Superseded by updated review
|
|
||
| ```bash | ||
| cd acceptance && go test ./acceptance -args -persist | ||
| ``` |
There was a problem hiding this comment.
[medium] incorrect-command
The command cd acceptance && go test ./acceptance -args -persist will fail. After cd acceptance, the module root is github.com/conforma/cli/acceptance and ./acceptance resolves to a nonexistent acceptance/acceptance/ subdirectory. The Makefile's focus-acceptance target confirms the correct form: cd acceptance && go test . -args .... Same issue at line 33 with -restore.
Suggested fix: Change ./acceptance to . in both commands (lines 27 and 33).
|
|
||
| ## Step 4: Debug acceptance test options | ||
|
|
||
| These require `go test` directly, not `make`: |
There was a problem hiding this comment.
[medium] incorrect-command
Same incorrect go test ./acceptance path after cd acceptance. Affects lines 62 (-persist), 66 (-restore), and 69 (-tags=@focus).
Suggested fix: Replace ./acceptance with . in all three commands.
fullsend-ai-review
Bot
added
the
requires-manual-review
3 participants
Add 6 repo-specific AI skills under
.claude/skills/, following the pattern established in conforma/e2e-tests (EC-1929).Skills: run-tests, debug-failure, build-and-lint, pr-checklist, write-tests, benchmark.
Jira: EC-1862