feat(eval): add batch-evaluation simulate (dataset simulations) - #2000

Closed

jariy17 wants to merge 13 commits into

aws:refactorfrom

jariy17:feat/eval-batch-simulate

Commits on Aug 13, 2026

feat(eval): add ondemand evaluate (synchronous, client-side)
jariy17
committed
59e81dd
Merge remote-tracking branch 'origin/refactor' into feat/eval-ondemand-evaluate-clean
jariy17
committed
9b57081
feat(eval): add batch-evaluation simulate (dataset replay)
jariy17
committed
e265d99
fix(eval): omit empty ground-truth arrays (service rejects length 0)
jariy17
committed
95d341a
fix(eval): drop --session-id from simulate, add ingestion wait + reject multi-turn
jariy17
committed
80291f9

Commits on Aug 14, 2026

feat(eval): support multi-turn scenarios (invoke each turn on one session, in order)
jariy17
committed
028bd0c
feat(eval): thread AbortSignal through simulate (Ctrl-C cancels invokes + ingestion wait)
jariy17
committed
6ba1994
fix(eval): unlink temp dataset file after simulate (dataset-id path)
jariy17
committed
a8675a2

Commits on Aug 18, 2026

refactor(eval): replace core.eval.simulate with invokeDataset + per-type example classes
jariy17
committed
889ea9b
fix(eval): reject non-object dataset rows and turn entries instead of crashing
jariy17
committed
cb7b78b
test(eval): cover runExamples failure isolation + handler sessionMetadata golden
jariy17
committed
3475347
style(eval): trim code comments to why-not-how, drop restatements
jariy17
committed
98f4b1a
Merge pull request #2 from jariy17/feat/eval-invoke-dataset
jariy17
authored
7d1f9b1