Skip to content

Commit e86cd34

Browse files
authored
chore: retire src/eval.ts and npm run eval (#24)
`src/eval.ts` and the `npm run eval` script have been dead-code-walking since Stack E shipped `npm run sim` (improvements.md #16, resolved 2026-04-30). The two runners coexisting under similar names actively misled engineers and customer forks. This change deletes the legacy runner and scrubs all doc references. The Vapi `/eval` REST endpoint, the `evals` resource type, the `/eval/simulation/*` simulation API, and `docs/learnings/simulations.md` are all unaffected — only the local CLI runner is gone. If customer-fork automation still invokes `npm run eval`, swap to `npm run sim -- <org> --suite <name> --target <name>`.
1 parent 170e331 commit e86cd34

6 files changed

Lines changed: 4 additions & 613 deletions

File tree

AGENTS.md

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -775,8 +775,6 @@ npm run rollback -- <org> --list # List available snapshots
775775
# Testing
776776
npm run call -- <org> -a <assistant-name> # Call an assistant via WebSocket
777777
npm run call -- <org> -s <squad-name> # Call a squad via WebSocket
778-
npm run eval -- <org> -s <squad-name> # Run evals against a squad
779-
npm run eval -- <org> -a <assistant-name> # Run evals against an assistant
780778
781779
# Maintenance
782780
npm run cleanup -- <org> # Dry-run: show orphaned remote resources

README.md

Lines changed: 0 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,6 @@ Every command works in two modes:
7676
| `npm run apply` || `npm run apply -- <org> [--force]` | Pull → Merge → Push in one shot |
7777
| `npm run call` || `npm run call -- <org> -a <name>` | Start a WebSocket call |
7878
| `npm run cleanup` || `npm run cleanup -- <org> [--force --confirm <org>]` | Delete orphaned remote resources (destructive run requires `--confirm <org>`) |
79-
| `npm run eval` || `npm run eval -- <org> -s <squad>` | Run evals against an assistant/squad |
8079
| `npm run build` ||| Type-check the codebase |
8180
| `npm test` ||| Run regression tests (`node:test`) |
8281

@@ -142,10 +141,6 @@ npm run call -- my-org -a my-assistant
142141

143142
# Call a squad
144143
npm run call -- my-org -s my-squad
145-
146-
# Run evals
147-
npm run eval -- my-org -s my-squad
148-
npm run eval -- my-org -a my-assistant --filter booking
149144
```
150145

151146
---
@@ -282,29 +277,6 @@ Squad push
282277
└─ all references resolved → create the squad ✓
283278
```
284279

285-
### Running Evals
286-
287-
Evals run mock conversations against an assistant or squad and check assertions.
288-
289-
```bash
290-
# Run all evals against a squad (transient — loaded from local files)
291-
npm run eval -- my-org -s my-squad
292-
293-
# Run a specific eval by name filter
294-
npm run eval -- my-org -a my-assistant --filter booking
295-
296-
# Use stored assistant/squad IDs from state (already pushed)
297-
npm run eval -- my-org -s my-squad --stored
298-
299-
# Load assistant from a specific file path
300-
npm run eval -- my-org -a resources/my-org/assistants/qa-tester.md
301-
302-
# Provide variable overrides
303-
npm run eval -- my-org -s my-squad -v eval-variables.json
304-
```
305-
306-
Evals must be pushed first (`npm run push -- my-org evals`). Eval definitions live in `resources/<org>/evals/*.yml`.
307-
308280
---
309281

310282
## File Formats
@@ -536,7 +508,6 @@ vapi-gitops/
536508
│ ├── push.ts # Push local state to platform
537509
│ ├── apply.ts # Orchestrator: pull → merge → push
538510
│ ├── call.ts # WebSocket call script
539-
│ ├── eval.ts # Eval runner
540511
│ ├── cleanup.ts # Orphan cleanup
541512
│ ├── pull-cmd.ts # Entry point: interactive or direct pull
542513
│ ├── push-cmd.ts # Entry point: interactive or direct push

package.json

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@
1212
"pull": "tsx src/pull-cmd.ts",
1313
"call": "bash -c 'exec tsx src/call-cmd.ts \"$@\" 2> >(grep --line-buffered -v \"buffer underflow\" >&2)' --",
1414
"cleanup": "tsx src/cleanup-cmd.ts",
15-
"eval": "tsx src/eval.ts",
1615
"validate": "tsx src/validate-cmd.ts",
1716
"sim": "tsx src/sim-cmd.ts",
1817
"rollback": "tsx src/rollback-cmd.ts",

0 commit comments

Comments
 (0)