Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 88 additions & 0 deletions .claude/skills/benchmark/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
---
name: benchmark
description: >
Run performance benchmarks for the Conforma CLI. Use when users ask "run benchmark",
"performance test", "stress test", "how fast", "benchmark data", "make benchmark",
or need help with performance measurement and profiling.
---

# Run Performance Benchmarks for the Conforma CLI

Prepare data, execute benchmarks, and report results.

## Step 1: Check prerequisites

Verify Docker or Podman is running (needed for testcontainers local registry):

```bash
podman info > /dev/null 2>&1 || docker info > /dev/null 2>&1
echo $?
```

If neither is available, start Podman or Docker before proceeding.

## Step 2: Prepare benchmark data

```bash
make benchmark_data
```

Or manually:

```bash
cd benchmark/simple
./prepare_data.sh
```

This pulls data from quay.io (~760MB). Only needed once.

## Step 3: Run the simple benchmark

Single-component validation against the `@redhat` policy collection:

```bash
make benchmark
```

Or with a specific iteration count:

```bash
cd benchmark/simple
go run . -benchnum 5
```

## Step 4: Run the stress benchmark (optional)

Multi-component snapshot with configurable parallelism:

```bash
cd benchmark/stress
./prepare_data.sh
go run .
```

Configure scale:

```bash
EC_STRESS_COMPONENTS=50 EC_STRESS_WORKERS=20 go run .
```

Defaults: 10 components, 35 workers.

## Step 5: Profile if needed

Use the CLI's built-in profiling:

```bash
ec validate image --trace=perf ... # Go runtime trace
ec validate image --trace=cpu ... # pprof CPU profile
ec validate image --trace=mem ... # heap profile
```

## Step 6: Report results

Output is in standard Go benchmark format (ns/op, memory stats). Summarize:
- Benchmark type run (simple/stress)
- Iteration count
- Key metrics (ns/op, allocs/op, bytes/op)
- Comparison with previous results if available
95 changes: 95 additions & 0 deletions .claude/skills/build-and-lint/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
name: build-and-lint
description: >
Build and lint the Conforma CLI. Use when users ask "how to build", "make build",
"lint errors", "lint fix", "golangci-lint", "make generate", "generated files",
"go mod tidy", or need help with the build process and code quality checks.
---

# Build and Lint the Conforma CLI

Run build, lint, and code generation steps, then report results.

## Step 1: Build the binary

```bash
make build
```

This produces `dist/ec_<os>_<arch>` for the current platform. Verify the binary exists:

```bash
ls -la dist/ec_*
```

For a debug build with debugger symbols:

```bash
DEBUG_BUILD=1 make build
```

## Step 2: Run code generation

```bash
make generate
git diff --exit-code
```

If there are changes, stage and commit them. CI will fail if generated code is out of sync.
Comment on lines +34 to +38

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Avoid mutating the repository from an unscoped generation diff.

Both workflows can include unrelated edits in git diff and omit untracked generated files, then instruct the agent to stage changes. Restrict verification to expected generated outputs or compare status before/after, and do not stage or commit automatically.

  • .claude/skills/build-and-lint/SKILL.md#L34-L38: remove the automatic staging/commit instruction and scope the generation check.
  • .claude/skills/pr-checklist/SKILL.md#L15-L20: remove the automatic staging instruction and scope the generation check.
📍 Affects 2 files
  • .claude/skills/build-and-lint/SKILL.md#L34-L38 (this comment)
  • .claude/skills/pr-checklist/SKILL.md#L15-L20
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.claude/skills/build-and-lint/SKILL.md around lines 34 - 38, Update the
generation verification instructions in .claude/skills/build-and-lint/SKILL.md
(lines 34-38) and .claude/skills/pr-checklist/SKILL.md (lines 15-20) to check
only expected generated outputs or compare repository status before and after
generation, rather than using an unscoped git diff. Remove all instructions to
stage or commit generated changes in both sites.


## Step 3: Run lint

```bash
make lint
```

Zero warnings are enforced. If lint fails, try auto-fix:

```bash
make lint-fix
```

Report what was auto-fixed vs what needs manual attention.
Comment on lines +46 to +52

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Re-run lint after auto-fix.

  • .claude/skills/build-and-lint/SKILL.md#L46-L52: run make lint after make lint-fix and report that result.
  • .claude/skills/pr-checklist/SKILL.md#L43-L43: use the post-fix lint result for the checklist status.
📍 Affects 2 files
  • .claude/skills/build-and-lint/SKILL.md#L46-L52 (this comment)
  • .claude/skills/pr-checklist/SKILL.md#L43-L43
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.claude/skills/build-and-lint/SKILL.md around lines 46 - 52, Update the lint
workflow in .claude/skills/build-and-lint/SKILL.md lines 46-52 to run make lint
after make lint-fix, report the post-fix result alongside auto-fixed and
manually actionable issues, and update .claude/skills/pr-checklist/SKILL.md line
43 to use that post-fix lint result for checklist status.


To lint a specific package:

```bash
go run -modfile tools/go.mod github.com/golangci/golangci-lint/v2/cmd/golangci-lint run ./internal/evaluator/...
```

## Step 4: Check module tidiness

Identify which modules have changes:

```bash
git diff --name-only origin/main...HEAD | grep -E '^(acceptance/|tools/)' || echo "root module only"
```

Run `go mod tidy` in each affected module:

```bash
# Root module
go mod tidy && git diff --exit-code go.mod go.sum

# If acceptance/ files changed
(cd acceptance && go mod tidy && git diff --exit-code go.mod go.sum)

# If tools/ files changed
(cd tools && go mod tidy && git diff --exit-code go.mod go.sum)

# If tools/kubectl/ files changed
(cd tools/kubectl && go mod tidy && git diff --exit-code go.mod go.sum)
```

## Step 5: Report

Summarize:

| Check | Status | Notes |
|-------|--------|-------|
| Build | pass/fail | |
| Generated code | pass/fail | |
| Lint | pass/fail | |
| Module tidiness | pass/fail | |

List any items that need attention.
128 changes: 128 additions & 0 deletions .claude/skills/debug-failure/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
---
name: debug-failure
description: >

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] description-field-formatting

Trigger phrases in description fields vary in style across files: some use quoted command strings, others mix quoted commands with bare phrases. Minor inconsistency.

Debug Conforma CLI test failures and runtime issues. Use when users ask "test
failed", "debug failure", "why is this failing", "preserve temp dir", "podman
error", "DNS resolution", "container failure", or need help troubleshooting.
---

# Debug a Failing Test or CLI Issue

Investigate the failure, preserve debug state, identify the root cause, and report findings.

## Step 1: Reproduce and preserve debug state

For CLI runtime issues, preserve temp directories:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] naming-consistency

Uses EC_DEBUG=1 while some test code uses EC_DEBUG=true. Both are functionally equivalent (the code checks for any non-empty value), but consistency across the codebase would reduce confusion.


```bash
EC_DEBUG=1 ec validate image ...
```

This keeps `ec-work-*` temp dirs for inspection. The `--debug` flag only increases log verbosity.

For acceptance test failures, keep containers running:

```bash
(cd acceptance && go test . -args -persist)
```

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] incorrect-command

The command cd acceptance && go test ./acceptance -args -persist will fail. After cd acceptance, the module root is github.com/conforma/cli/acceptance and ./acceptance resolves to a nonexistent acceptance/acceptance/ subdirectory. The Makefile's focus-acceptance target confirms the correct form: cd acceptance && go test . -args .... Same issue at line 33 with -restore.

Suggested fix: Change ./acceptance to . in both commands (lines 27 and 33).


Reattach later with:

```bash
(cd acceptance && go test . -args -restore)
```

## Step 2: Read the failure output

- **Unit/integration tests**: check the assertion message and stack trace
- **Acceptance tests**: compare `features/__snapshots__/` for expected vs actual output
- **CLI runtime**: inspect `ec-work-*` temp dirs for downloaded policies, OPA data, and evaluation artifacts

```bash
ls -la /tmp/ec-work-*
```

## Step 3: Check common failure patterns

Run through each check and report which applies:

### DNS resolution failures

Binary is built with `CGO_ENABLED=0` (native Go DNS resolver). Check `/etc/hosts`:

```bash
grep -E 'apiserver\.localhost|rekor\.localhost' /etc/hosts
```

If missing, add:

```text
127.0.0.1 apiserver.localhost
127.0.0.1 rekor.localhost
```

### Podman issues

```bash
# Check podman socket (Linux)
systemctl --user status podman.socket

# macOS: verify podman machine
podman machine info
# If not configured:
./hack/macos/setup-podman-machine.sh
```

### inotify / key limit errors

```bash
cat /proc/sys/fs/inotify/max_user_watches
# If below 524288:
sudo sysctl fs.inotify.max_user_watches=524288

cat /proc/sys/kernel/keys/maxkeys
# If below 1000:
sudo sysctl kernel.keys.maxkeys=1000
```

### Go checksum mismatch

```bash
go env GOPROXY
# If not set correctly:
go env -w GOPROXY='https://proxy.golang.org,direct'
```

### Snapshot mismatch

```bash
UPDATE_SNAPS=true make acceptance
git diff features/__snapshots__/
```

Review the diff to confirm changes are expected.

### Generated code out of sync

```bash
make generate
git diff --exit-code
```

If CI fails with "File was modified in build", commit the generated changes.

## Step 4: Check CI-specific issues

If the failure is CI-only:

- Harden Runner is disabled for acceptance tests (DNS conflicts)
- CI installs tkn and kubectl from pinned versions, not Go tools
- CI runs `hack/ubuntu-podman-update.sh` before acceptance tests

## Step 5: Report findings

Summarize:
- Root cause identified or suspected
- Which check from Step 3 matched
- Fix applied or recommended
- Whether the fix needs to be committed
Loading
Loading