-
Notifications
You must be signed in to change notification settings - Fork 61
feat(EC-1862): add AI skills for ec-cli #3434
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| --- | ||
| name: benchmark | ||
| description: > | ||
| Run performance benchmarks for the Conforma CLI. Use when users ask "run benchmark", | ||
| "performance test", "stress test", "how fast", "benchmark data", "make benchmark", | ||
| or need help with performance measurement and profiling. | ||
| --- | ||
|
|
||
| # Run Performance Benchmarks for the Conforma CLI | ||
|
|
||
| Prepare data, execute benchmarks, and report results. | ||
|
|
||
| ## Step 1: Check prerequisites | ||
|
|
||
| Verify Docker or Podman is running (needed for testcontainers local registry): | ||
|
|
||
| ```bash | ||
| podman info > /dev/null 2>&1 || docker info > /dev/null 2>&1 | ||
| echo $? | ||
| ``` | ||
|
|
||
| If neither is available, start Podman or Docker before proceeding. | ||
|
|
||
| ## Step 2: Prepare benchmark data | ||
|
|
||
| ```bash | ||
| make benchmark_data | ||
| ``` | ||
|
|
||
| Or manually: | ||
|
|
||
| ```bash | ||
| cd benchmark/simple | ||
| ./prepare_data.sh | ||
| ``` | ||
|
|
||
| This pulls data from quay.io (~760MB). Only needed once. | ||
|
|
||
| ## Step 3: Run the simple benchmark | ||
|
|
||
| Single-component validation against the `@redhat` policy collection: | ||
|
|
||
| ```bash | ||
| make benchmark | ||
| ``` | ||
|
|
||
| Or with a specific iteration count: | ||
|
|
||
| ```bash | ||
| cd benchmark/simple | ||
| go run . -benchnum 5 | ||
| ``` | ||
|
|
||
| ## Step 4: Run the stress benchmark (optional) | ||
|
|
||
| Multi-component snapshot with configurable parallelism: | ||
|
|
||
| ```bash | ||
| cd benchmark/stress | ||
| ./prepare_data.sh | ||
| go run . | ||
| ``` | ||
|
|
||
| Configure scale: | ||
|
|
||
| ```bash | ||
| EC_STRESS_COMPONENTS=50 EC_STRESS_WORKERS=20 go run . | ||
| ``` | ||
|
|
||
| Defaults: 10 components, 35 workers. | ||
|
|
||
| ## Step 5: Profile if needed | ||
|
|
||
| Use the CLI's built-in profiling: | ||
|
|
||
| ```bash | ||
| ec validate image --trace=perf ... # Go runtime trace | ||
| ec validate image --trace=cpu ... # pprof CPU profile | ||
| ec validate image --trace=mem ... # heap profile | ||
| ``` | ||
|
|
||
| ## Step 6: Report results | ||
|
|
||
| Output is in standard Go benchmark format (ns/op, memory stats). Summarize: | ||
| - Benchmark type run (simple/stress) | ||
| - Iteration count | ||
| - Key metrics (ns/op, allocs/op, bytes/op) | ||
| - Comparison with previous results if available |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| --- | ||
| name: build-and-lint | ||
| description: > | ||
| Build and lint the Conforma CLI. Use when users ask "how to build", "make build", | ||
| "lint errors", "lint fix", "golangci-lint", "make generate", "generated files", | ||
| "go mod tidy", or need help with the build process and code quality checks. | ||
| --- | ||
|
|
||
| # Build and Lint the Conforma CLI | ||
|
|
||
| Run build, lint, and code generation steps, then report results. | ||
|
|
||
| ## Step 1: Build the binary | ||
|
|
||
| ```bash | ||
| make build | ||
| ``` | ||
|
|
||
| This produces `dist/ec_<os>_<arch>` for the current platform. Verify the binary exists: | ||
|
|
||
| ```bash | ||
| ls -la dist/ec_* | ||
| ``` | ||
|
|
||
| For a debug build with debugger symbols: | ||
|
|
||
| ```bash | ||
| DEBUG_BUILD=1 make build | ||
| ``` | ||
|
|
||
| ## Step 2: Run code generation | ||
|
|
||
| ```bash | ||
| make generate | ||
| git diff --exit-code | ||
| ``` | ||
|
|
||
| If there are changes, stage and commit them. CI will fail if generated code is out of sync. | ||
|
|
||
| ## Step 3: Run lint | ||
|
|
||
| ```bash | ||
| make lint | ||
| ``` | ||
|
|
||
| Zero warnings are enforced. If lint fails, try auto-fix: | ||
|
|
||
| ```bash | ||
| make lint-fix | ||
| ``` | ||
|
|
||
| Report what was auto-fixed vs what needs manual attention. | ||
|
Comment on lines
+46
to
+52
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win Re-run lint after auto-fix.
📍 Affects 2 files
🤖 Prompt for AI Agents |
||
|
|
||
| To lint a specific package: | ||
|
|
||
| ```bash | ||
| go run -modfile tools/go.mod github.com/golangci/golangci-lint/v2/cmd/golangci-lint run ./internal/evaluator/... | ||
| ``` | ||
|
|
||
| ## Step 4: Check module tidiness | ||
|
|
||
| Identify which modules have changes: | ||
|
|
||
| ```bash | ||
| git diff --name-only origin/main...HEAD | grep -E '^(acceptance/|tools/)' || echo "root module only" | ||
| ``` | ||
|
|
||
| Run `go mod tidy` in each affected module: | ||
|
|
||
| ```bash | ||
| # Root module | ||
| go mod tidy && git diff --exit-code go.mod go.sum | ||
|
|
||
| # If acceptance/ files changed | ||
| (cd acceptance && go mod tidy && git diff --exit-code go.mod go.sum) | ||
|
|
||
| # If tools/ files changed | ||
| (cd tools && go mod tidy && git diff --exit-code go.mod go.sum) | ||
|
|
||
| # If tools/kubectl/ files changed | ||
| (cd tools/kubectl && go mod tidy && git diff --exit-code go.mod go.sum) | ||
| ``` | ||
|
|
||
| ## Step 5: Report | ||
|
|
||
| Summarize: | ||
|
|
||
| | Check | Status | Notes | | ||
| |-------|--------|-------| | ||
| | Build | pass/fail | | | ||
| | Generated code | pass/fail | | | ||
| | Lint | pass/fail | | | ||
| | Module tidiness | pass/fail | | | ||
|
|
||
| List any items that need attention. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,128 @@ | ||
| --- | ||
| name: debug-failure | ||
| description: > | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] description-field-formatting Trigger phrases in description fields vary in style across files: some use quoted command strings, others mix quoted commands with bare phrases. Minor inconsistency. |
||
| Debug Conforma CLI test failures and runtime issues. Use when users ask "test | ||
| failed", "debug failure", "why is this failing", "preserve temp dir", "podman | ||
| error", "DNS resolution", "container failure", or need help troubleshooting. | ||
| --- | ||
|
|
||
| # Debug a Failing Test or CLI Issue | ||
|
|
||
| Investigate the failure, preserve debug state, identify the root cause, and report findings. | ||
|
|
||
| ## Step 1: Reproduce and preserve debug state | ||
|
|
||
| For CLI runtime issues, preserve temp directories: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [low] naming-consistency Uses EC_DEBUG=1 while some test code uses EC_DEBUG=true. Both are functionally equivalent (the code checks for any non-empty value), but consistency across the codebase would reduce confusion. |
||
|
|
||
| ```bash | ||
| EC_DEBUG=1 ec validate image ... | ||
| ``` | ||
|
|
||
| This keeps `ec-work-*` temp dirs for inspection. The `--debug` flag only increases log verbosity. | ||
|
|
||
| For acceptance test failures, keep containers running: | ||
|
|
||
| ```bash | ||
| (cd acceptance && go test . -args -persist) | ||
| ``` | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [medium] incorrect-command The command Suggested fix: Change |
||
|
|
||
| Reattach later with: | ||
|
|
||
| ```bash | ||
| (cd acceptance && go test . -args -restore) | ||
| ``` | ||
|
|
||
| ## Step 2: Read the failure output | ||
|
|
||
| - **Unit/integration tests**: check the assertion message and stack trace | ||
| - **Acceptance tests**: compare `features/__snapshots__/` for expected vs actual output | ||
| - **CLI runtime**: inspect `ec-work-*` temp dirs for downloaded policies, OPA data, and evaluation artifacts | ||
|
|
||
| ```bash | ||
| ls -la /tmp/ec-work-* | ||
| ``` | ||
|
|
||
| ## Step 3: Check common failure patterns | ||
|
|
||
| Run through each check and report which applies: | ||
|
|
||
| ### DNS resolution failures | ||
|
|
||
| Binary is built with `CGO_ENABLED=0` (native Go DNS resolver). Check `/etc/hosts`: | ||
|
|
||
| ```bash | ||
| grep -E 'apiserver\.localhost|rekor\.localhost' /etc/hosts | ||
| ``` | ||
|
|
||
| If missing, add: | ||
|
|
||
| ```text | ||
| 127.0.0.1 apiserver.localhost | ||
| 127.0.0.1 rekor.localhost | ||
| ``` | ||
|
|
||
| ### Podman issues | ||
|
|
||
| ```bash | ||
| # Check podman socket (Linux) | ||
| systemctl --user status podman.socket | ||
|
|
||
| # macOS: verify podman machine | ||
| podman machine info | ||
| # If not configured: | ||
| ./hack/macos/setup-podman-machine.sh | ||
| ``` | ||
|
|
||
| ### inotify / key limit errors | ||
|
|
||
| ```bash | ||
| cat /proc/sys/fs/inotify/max_user_watches | ||
| # If below 524288: | ||
| sudo sysctl fs.inotify.max_user_watches=524288 | ||
|
|
||
| cat /proc/sys/kernel/keys/maxkeys | ||
| # If below 1000: | ||
| sudo sysctl kernel.keys.maxkeys=1000 | ||
| ``` | ||
|
|
||
| ### Go checksum mismatch | ||
|
|
||
| ```bash | ||
| go env GOPROXY | ||
| # If not set correctly: | ||
| go env -w GOPROXY='https://proxy.golang.org,direct' | ||
| ``` | ||
|
|
||
| ### Snapshot mismatch | ||
|
|
||
| ```bash | ||
| UPDATE_SNAPS=true make acceptance | ||
| git diff features/__snapshots__/ | ||
| ``` | ||
|
|
||
| Review the diff to confirm changes are expected. | ||
|
|
||
| ### Generated code out of sync | ||
|
|
||
| ```bash | ||
| make generate | ||
| git diff --exit-code | ||
| ``` | ||
|
|
||
| If CI fails with "File was modified in build", commit the generated changes. | ||
|
|
||
| ## Step 4: Check CI-specific issues | ||
|
|
||
| If the failure is CI-only: | ||
|
|
||
| - Harden Runner is disabled for acceptance tests (DNS conflicts) | ||
| - CI installs tkn and kubectl from pinned versions, not Go tools | ||
| - CI runs `hack/ubuntu-podman-update.sh` before acceptance tests | ||
|
|
||
| ## Step 5: Report findings | ||
|
|
||
| Summarize: | ||
| - Root cause identified or suspected | ||
| - Which check from Step 3 matched | ||
| - Fix applied or recommended | ||
| - Whether the fix needs to be committed | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Avoid mutating the repository from an unscoped generation diff.
Both workflows can include unrelated edits in
git diffand omit untracked generated files, then instruct the agent to stage changes. Restrict verification to expected generated outputs or compare status before/after, and do not stage or commit automatically..claude/skills/build-and-lint/SKILL.md#L34-L38: remove the automatic staging/commit instruction and scope the generation check..claude/skills/pr-checklist/SKILL.md#L15-L20: remove the automatic staging instruction and scope the generation check.📍 Affects 2 files
.claude/skills/build-and-lint/SKILL.md#L34-L38(this comment).claude/skills/pr-checklist/SKILL.md#L15-L20🤖 Prompt for AI Agents