Translation bench

[Dominic Nguyen (datduyng)]

Summary

Empty-gold TB negatives (expectedActions: []) are scored as zero actions across the full loaded tool catalog. Label-fair Q&A/definition negatives were not zero-action-scorer-fair: models correctly fire chat/help/history, tanking negative pass.

This PR ships a pipeline fairness contract so empty gold is only accepted for hard abstains:

• Synthesizer + quality-verifier prompts: empty-gold kind is pure_refusal only (leave-alone / don't / never / cancel of the target; no questions, no alternate task).
• negativeFairness module: path-keyed 1:1 negativeAssessments (no regex path parse); fail-closed on missing/duplicate paths; fairEmptyGold only when kind is in the allowlist (TRANSLATION_BENCH_FAIR_EMPTY_GOLD_KINDS).
• Partial generation: when complete coverage is optional, emitted caseCount / coverage metadata match the cases actually produced (so validation can succeed).
• No gold-row hand edits and no empty-gold scorer loosening.

Measured 3-model eval (honest)

Regenerated draft under the new policy (998/1000 rows; 2 slots exhausted data-quality, unrelated to fairness), then evaluated azure/gpt-5.6-{sol,terra,luna}.

Metric	Result
Rows / empty-gold negs	998 / 998
Kind mix	100% pure_refusal
Unfair rate (LLM audit)	0%
Overall pass	36.8%
Negative pass (zero fire on empty gold)	13.9%
Neg fire-on-empty	47.9%
Positive pass	59.8%
Pos abstention FNR	3.2%
Tool / param	83.5% / ~71.6%
Eval cost	~$190

Finding: fairness regen fixed the dataset (kind mix + unfair rate), but did not produce a large negative-pass lift. Checkpoint inspection shows models often emit the refused action itself on clear hard-abstains (negation-following failure). Translator/prompt work is out of scope here.

Test plan

• pnpm run build in ts/packages/benchmarks
• pnpm run jest-esm --testPathPattern='(negativeFairness|datasetGenerator)\\.spec\\.js' (43/43)
• Smoke + full regen fairness audit (0% unfair, 100% pure_refusal)
• 3-model headless eval on regenerated draft; scale-metrics recorded

---

Infra: extract cross-process TPM limiter + run config to core

Follow-on refactor moving the run harness's rate limiter and config loader into @typeagent/benchmarks as tested, reusable modules.

Interface (one owner per concern):

• createRateLimiter(tpmLimits, { dbPath, estTokensPerCall?, maxWaitMs?, onWait? }) → { disabledFor, run, close }. dbPath is required so callers own persistence; no hidden default DB. Optional knobs are orthogonal.
• runConfig.ts is pure: reads a JSON file (schema-validated), merges batch over base, derives per-model concurrency. No env, no process globals — runtime overrides are commander flags prop-drilled by the local runners (TB_* env removed).
• Credential env (OPENAI_*/AZURE_*) is intentionally kept as the aiclient contract; documented in AGENTS.md.

Correctness:

• TPM is enforced by a shared SQLite ledger over a rolling 60s window (BEGIN IMMEDIATE, WAL): concurrent awaited run() calls reserve then settle to actual token usage, staying within quota across processes.
• In-flight claims survive purge to STALE_MS (purge guarded on pending = 0); settle falls back to insert if the claim row was purged.
• Fail-closed: throws on missing positive token estimate; omits non-positive tpm limits.

Tests: translationBench.rateLimiter.spec + translationBench.runConfig.spec (17 cases) — passthrough for unquota'd models, budget admit/throttle, cross-instance shared budget, estimate fallbacks, batch merge, concurrency derivation, malformed-json handling.

Test plan (infra)

• pnpm run build + pnpm run prettier clean in ts/packages/benchmarks
• pnpm run jest-esm --testPathPattern='translationBench.(rateLimiter|runConfig).spec.js' (17/17)
• Commander parse + resolveRunnerConfig smoke against real config.local.json

[Copilot]

Pull request overview

Adds a fairness contract ensuring empty-gold translation-bench negatives represent only pure refusals.

Changes:

• Adds path-keyed LLM fairness assessments and enforcement.
• Updates generation/verifier prompts and gold-action validation.
• Supports concurrent generation and accurate partial-result metadata.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
README.AUTOGEN.md	Refreshes generated documentation metadata.
actionValidation.ts	Normalizes gold actions for schema validation.
benchmark.ts	Uses normalized gold-action validation.
dataQualityVerifier.ts	Enforces semantic negative-fairness assessments.
datasetGenerator.ts	Adds concurrency and partial-generation handling.
generationCandidate.ts	Preserves required parameter objects during cleanup.
index.ts	Exports the fairness module.
negativeFairness.ts	Implements fairness parsing and enforcement.
quality-verifier.prompt.yaml	Defines full-catalog fairness review requirements.
synthesizer.prompt.yaml	Restricts empty-gold negatives to pure refusals.
utteranceDisambiguation.ts	Documents separate negative handling.
translationBench.datasetGenerator.spec.ts	Updates generation fixtures for fairness assessments.
translationBench.negativeFairness.spec.ts	Tests fairness validation and semantic enforcement.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.