diff --git a/.claude/skills/benchmark/SKILL.md b/.claude/skills/benchmark/SKILL.md new file mode 100644 index 000000000..8f6a2e78a --- /dev/null +++ b/.claude/skills/benchmark/SKILL.md @@ -0,0 +1,88 @@ +--- +name: benchmark +description: > + Run performance benchmarks for the Conforma CLI. Use when users ask "run benchmark", + "performance test", "stress test", "how fast", "benchmark data", "make benchmark", + or need help with performance measurement and profiling. +--- + +# Run Performance Benchmarks for the Conforma CLI + +Prepare data, execute benchmarks, and report results. + +## Step 1: Check prerequisites + +Verify Docker or Podman is running (needed for testcontainers local registry): + +```bash +podman info > /dev/null 2>&1 || docker info > /dev/null 2>&1 +echo $? +``` + +If neither is available, start Podman or Docker before proceeding. + +## Step 2: Prepare benchmark data + +```bash +make benchmark_data +``` + +Or manually: + +```bash +cd benchmark/simple +./prepare_data.sh +``` + +This pulls data from quay.io (~760MB). Only needed once. + +## Step 3: Run the simple benchmark + +Single-component validation against the `@redhat` policy collection: + +```bash +make benchmark +``` + +Or with a specific iteration count: + +```bash +cd benchmark/simple +go run . -benchnum 5 +``` + +## Step 4: Run the stress benchmark (optional) + +Multi-component snapshot with configurable parallelism: + +```bash +cd benchmark/stress +./prepare_data.sh +go run . +``` + +Configure scale: + +```bash +EC_STRESS_COMPONENTS=50 EC_STRESS_WORKERS=20 go run . +``` + +Defaults: 10 components, 35 workers. + +## Step 5: Profile if needed + +Use the CLI's built-in profiling: + +```bash +ec validate image --trace=perf ... # Go runtime trace +ec validate image --trace=cpu ... # pprof CPU profile +ec validate image --trace=mem ... # heap profile +``` + +## Step 6: Report results + +Output is in standard Go benchmark format (ns/op, memory stats). Summarize: +- Benchmark type run (simple/stress) +- Iteration count +- Key metrics (ns/op, allocs/op, bytes/op) +- Comparison with previous results if available diff --git a/.claude/skills/build-and-lint/SKILL.md b/.claude/skills/build-and-lint/SKILL.md new file mode 100644 index 000000000..5a16c3e5b --- /dev/null +++ b/.claude/skills/build-and-lint/SKILL.md @@ -0,0 +1,95 @@ +--- +name: build-and-lint +description: > + Build and lint the Conforma CLI. Use when users ask "how to build", "make build", + "lint errors", "lint fix", "golangci-lint", "make generate", "generated files", + "go mod tidy", or need help with the build process and code quality checks. +--- + +# Build and Lint the Conforma CLI + +Run build, lint, and code generation steps, then report results. + +## Step 1: Build the binary + +```bash +make build +``` + +This produces `dist/ec__` for the current platform. Verify the binary exists: + +```bash +ls -la dist/ec_* +``` + +For a debug build with debugger symbols: + +```bash +DEBUG_BUILD=1 make build +``` + +## Step 2: Run code generation + +```bash +make generate +git diff --exit-code +``` + +If there are changes, stage and commit them. CI will fail if generated code is out of sync. + +## Step 3: Run lint + +```bash +make lint +``` + +Zero warnings are enforced. If lint fails, try auto-fix: + +```bash +make lint-fix +``` + +Report what was auto-fixed vs what needs manual attention. + +To lint a specific package: + +```bash +go run -modfile tools/go.mod github.com/golangci/golangci-lint/v2/cmd/golangci-lint run ./internal/evaluator/... +``` + +## Step 4: Check module tidiness + +Identify which modules have changes: + +```bash +git diff --name-only origin/main...HEAD | grep -E '^(acceptance/|tools/)' || echo "root module only" +``` + +Run `go mod tidy` in each affected module: + +```bash +# Root module +go mod tidy && git diff --exit-code go.mod go.sum + +# If acceptance/ files changed +(cd acceptance && go mod tidy && git diff --exit-code go.mod go.sum) + +# If tools/ files changed +(cd tools && go mod tidy && git diff --exit-code go.mod go.sum) + +# If tools/kubectl/ files changed +(cd tools/kubectl && go mod tidy && git diff --exit-code go.mod go.sum) +``` + +## Step 5: Report + +Summarize: + +| Check | Status | Notes | +|-------|--------|-------| +| Build | pass/fail | | +| Generated code | pass/fail | | +| Lint | pass/fail | | +| Module tidiness | pass/fail | | + +List any items that need attention. diff --git a/.claude/skills/debug-failure/SKILL.md b/.claude/skills/debug-failure/SKILL.md new file mode 100644 index 000000000..32ea026d8 --- /dev/null +++ b/.claude/skills/debug-failure/SKILL.md @@ -0,0 +1,128 @@ +--- +name: debug-failure +description: > + Debug Conforma CLI test failures and runtime issues. Use when users ask "test + failed", "debug failure", "why is this failing", "preserve temp dir", "podman + error", "DNS resolution", "container failure", or need help troubleshooting. +--- + +# Debug a Failing Test or CLI Issue + +Investigate the failure, preserve debug state, identify the root cause, and report findings. + +## Step 1: Reproduce and preserve debug state + +For CLI runtime issues, preserve temp directories: + +```bash +EC_DEBUG=1 ec validate image ... +``` + +This keeps `ec-work-*` temp dirs for inspection. The `--debug` flag only increases log verbosity. + +For acceptance test failures, keep containers running: + +```bash +(cd acceptance && go test . -args -persist) +``` + +Reattach later with: + +```bash +(cd acceptance && go test . -args -restore) +``` + +## Step 2: Read the failure output + +- **Unit/integration tests**: check the assertion message and stack trace +- **Acceptance tests**: compare `features/__snapshots__/` for expected vs actual output +- **CLI runtime**: inspect `ec-work-*` temp dirs for downloaded policies, OPA data, and evaluation artifacts + +```bash +ls -la /tmp/ec-work-* +``` + +## Step 3: Check common failure patterns + +Run through each check and report which applies: + +### DNS resolution failures + +Binary is built with `CGO_ENABLED=0` (native Go DNS resolver). Check `/etc/hosts`: + +```bash +grep -E 'apiserver\.localhost|rekor\.localhost' /etc/hosts +``` + +If missing, add: + +```text +127.0.0.1 apiserver.localhost +127.0.0.1 rekor.localhost +``` + +### Podman issues + +```bash +# Check podman socket (Linux) +systemctl --user status podman.socket + +# macOS: verify podman machine +podman machine info +# If not configured: +./hack/macos/setup-podman-machine.sh +``` + +### inotify / key limit errors + +```bash +cat /proc/sys/fs/inotify/max_user_watches +# If below 524288: +sudo sysctl fs.inotify.max_user_watches=524288 + +cat /proc/sys/kernel/keys/maxkeys +# If below 1000: +sudo sysctl kernel.keys.maxkeys=1000 +``` + +### Go checksum mismatch + +```bash +go env GOPROXY +# If not set correctly: +go env -w GOPROXY='https://proxy.golang.org,direct' +``` + +### Snapshot mismatch + +```bash +UPDATE_SNAPS=true make acceptance +git diff features/__snapshots__/ +``` + +Review the diff to confirm changes are expected. + +### Generated code out of sync + +```bash +make generate +git diff --exit-code +``` + +If CI fails with "File was modified in build", commit the generated changes. + +## Step 4: Check CI-specific issues + +If the failure is CI-only: + +- Harden Runner is disabled for acceptance tests (DNS conflicts) +- CI installs tkn and kubectl from pinned versions, not Go tools +- CI runs `hack/ubuntu-podman-update.sh` before acceptance tests + +## Step 5: Report findings + +Summarize: +- Root cause identified or suspected +- Which check from Step 3 matched +- Fix applied or recommended +- Whether the fix needs to be committed diff --git a/.claude/skills/pr-checklist/SKILL.md b/.claude/skills/pr-checklist/SKILL.md new file mode 100644 index 000000000..48c497ba4 --- /dev/null +++ b/.claude/skills/pr-checklist/SKILL.md @@ -0,0 +1,98 @@ +--- +name: pr-checklist +description: > + Verify a Conforma CLI pull request is ready for review. Use when users ask + "is this PR ready", "definition of done", "PR checklist", "before merging", + "review checklist", or when preparing a PR for review. +--- + +# Verify PR Readiness for Conforma CLI + +Run through each step below, report pass/fail for each, and summarize what remains. + +## Step 1: Check for uncommitted generated code + +```bash +make generate +git diff --exit-code +``` + +If there are uncommitted changes after `make generate`, flag them and stage the results. + +## Step 2: Check module tidiness + +List all modules and run `go mod tidy` in each, then check for drift: + +```bash +git ls-files go.mod '*/go.mod' +``` + +```bash +go mod tidy && git diff --exit-code go.mod go.sum +(cd acceptance && go mod tidy && git diff --exit-code go.mod go.sum) +(cd tools && go mod tidy && git diff --exit-code go.mod go.sum) +(cd tools/kubectl && go mod tidy && git diff --exit-code go.mod go.sum) +``` + +## Step 3: Run lint + +```bash +make lint +``` + +Zero warnings required. If there are failures, run `make lint-fix` and report what was auto-fixed vs what needs manual attention. + +## Step 4: Run tests + +```bash +make test +``` + +If acceptance-relevant code changed (features/, acceptance/, CLI commands), also run: + +```bash +make acceptance +``` + +If behavior changed, check whether snapshots need updating: + +```bash +git diff features/__snapshots__/ +``` + +If snapshots are stale, update them with `UPDATE_SNAPS=true make acceptance` and report which snapshots changed. + +## Step 5: Verify build tags on new test files + +Find any new test files in the diff and verify each starts with a build tag (`//go:build unit`, `integration`, or `generative`): + +```bash +git diff --name-only --diff-filter=A origin/main...HEAD | grep '_test\.go$' +``` + +Read the first line of each new test file and flag any missing build tags. + +## Step 6: Check PR description + +If a PR already exists for this branch, fetch its description and verify it covers: + +- **What:** What the change does +- **Why:** Context and background +- **Tickets:** Link to Jira issue + +Compare against the template in `.github/pull_request_template.md`. + +## Step 7: Report + +Produce a summary table: + +| Check | Status | Notes | +|-------|--------|-------| +| Generated code | pass/fail | | +| Module tidiness | pass/fail | | +| Lint | pass/fail | | +| Tests | pass/fail | | +| Build tags | pass/fail | | +| PR description | pass/fail/skip | | + +List any items that need attention before the PR is ready for review. diff --git a/.claude/skills/run-tests/SKILL.md b/.claude/skills/run-tests/SKILL.md new file mode 100644 index 000000000..7fca06489 --- /dev/null +++ b/.claude/skills/run-tests/SKILL.md @@ -0,0 +1,97 @@ +--- +name: run-tests +description: > + Run Conforma CLI tests. Use when users ask "how to run tests", "run unit tests", + "run acceptance tests", "make test", "test tags", "test timeout", "run a single + test", "ginkgo", or need help with test execution and environment setup. +--- + +# Run Conforma CLI Tests + +Determine which tests to run, execute them, and report results. + +## Step 1: Determine test scope + +Check what files changed to decide which tests are needed: + +```bash +git diff --name-only origin/main...HEAD +``` + +- Changes in `internal/`, `cmd/`, `pkg/` → run unit + integration tests +- Changes in `features/`, `acceptance/` → run acceptance tests +- Changes in `tools/` → run `make tools-ci` + +## Step 2: Run unit and integration tests + +```bash +make test +``` + +This runs unit (10s timeout), integration (15s), and generative (30s) tests. If a specific test is needed: + +```bash +go test -tags=unit ./internal/evaluator -run TestName +``` + +Report any failures with the test name and error message. + +## Step 3: Run acceptance tests (if needed) + +Full suite (~20 min): + +```bash +make acceptance +``` + +Single feature or scenario: + +```bash +make feature_validate_image +make scenario_inline_policy +``` + +Focused tests (tag scenarios with `@focus`): + +```bash +make focus-acceptance +``` + +## Step 4: Debug acceptance test options + +These require `go test` directly, not `make`: + +```bash +# Keep test containers running after failure +(cd acceptance && go test . -args -persist) + +# Reattach to persisted containers +(cd acceptance && go test . -args -restore) + +# Run with specific tags +(cd acceptance && go test . -args -tags=@focus) + +# Update snapshot files +UPDATE_SNAPS=true make acceptance +``` + +## Step 5: Verify macOS setup (if acceptance tests fail on macOS) + +```bash +./hack/macos/setup-podman-machine.sh +``` + +Check `/etc/hosts` has required entries: + +```text +127.0.0.1 apiserver.localhost +127.0.0.1 rekor.localhost +``` + +## Step 6: Report results + +Summarize: +- Which test suites ran and their pass/fail status +- Any failing test names with error messages +- Whether snapshot updates are needed +- Environment issues encountered (Podman, DNS, etc.) diff --git a/.claude/skills/write-tests/SKILL.md b/.claude/skills/write-tests/SKILL.md new file mode 100644 index 000000000..19bcdca5a --- /dev/null +++ b/.claude/skills/write-tests/SKILL.md @@ -0,0 +1,122 @@ +--- +name: write-tests +description: > + Write tests for the Conforma CLI. Use when users ask "add a test", "write a test", + "new test case", "how to test", "test pattern", "build tags", "snapshot testing", + "acceptance test", "cucumber", or need guidance on the test framework. +--- + +# Write a Test for the Conforma CLI + +Determine the test type, create the file with correct conventions, write the test, and verify it passes. + +## Step 1: Determine test type + +Check what is being tested: + +- **Unit test** (`//go:build unit`, 10s timeout): isolated function, mocked dependencies +- **Integration test** (`//go:build integration`, 15s timeout): real OPA engine, no mocks +- **Acceptance test** (Cucumber/Gherkin, 20m timeout): end-to-end CLI behavior + +## Step 2: Create the test file + +### For unit or integration tests + +Create a `_test.go` file alongside the source. Every test file must start with a build tag: + +```go +//go:build unit + +package mypackage + +import ( + "testing" + "github.com/stretchr/testify/assert" +) +``` + +Follow existing naming patterns in the package: +- `evaluator.go` → `evaluator_test.go` +- Multiple test files per concern: `evaluator_unit_core_test.go`, `evaluator_unit_data_test.go` + +### For acceptance tests + +Create or modify a `.feature` file in `features/`: + +```gherkin +Feature: My feature + Scenario: it works + Given a known good image + When ec validates the image + Then the exit status is 0 +``` + +Step definitions go in `acceptance/` subdirectories, registered via `AddStepsTo(sc *godog.ScenarioContext)` functions. State is passed between steps via `context.Context` values — no global state. + +## Step 3: Write the test + +Use table-driven tests for multiple cases: + +```go +func TestMyFunction(t *testing.T) { + cases := []struct { + name string + input string + expected string + expectErr string + }{ + {name: "valid input", input: "good", expected: "result"}, + {name: "invalid input", input: "bad", expectErr: "error message"}, + } + + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + result, err := MyFunc(c.input) + if c.expectErr != "" { + assert.ErrorContains(t, err, c.expectErr) + return + } + assert.NoError(t, err) + assert.Equal(t, c.expected, result) + }) + } +} +``` + +For snapshot testing, use `go-snaps`: + +```go +import "github.com/gkampitakis/go-snaps/snaps" + +func TestOutput(t *testing.T) { + result := GenerateOutput() + snaps.MatchJSON(t, result) +} +``` + +## Step 4: Run the test and verify + +```bash +# Unit test +go test -tags=unit ./internal/mypackage -run TestMyFunction + +# Integration test +go test -tags=integration ./internal/mypackage -run TestMyFunction + +# Acceptance scenario +make scenario_it_works +``` + +Verify: +- Test passes +- Build tag is present on first line +- Test uses `testify/assert` for assertions +- No global state in acceptance tests + +## Step 5: Report + +Summarize: +- Test file created/modified and its location +- Test type and build tag used +- Pass/fail result +- Any snapshot files created or updated diff --git a/AGENTS.md b/AGENTS.md index ff645be5f..2f3e7e27b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,8 +41,8 @@ Tests use build tags with different timeouts: - **Multi-module project:** root, `acceptance/`, `tools/` each have their own go.mod. Run `go mod tidy` in the right module. -- **Debug mode:** `--debug` or `EC_DEBUG=1` preserves `ec-work-*` temp directories for inspection. -- Conventional commits with Jira key encouraged (e.g., `feat(EC-1234): description`). +- **Debug mode:** `EC_DEBUG=1` preserves `ec-work-*` temp directories for inspection. + The `--debug` flag only increases log verbosity. ## CGO and DNS Resolution