bt scorers create/update

[Cedric / ViaDézo1er (viadezo1er)]

messages.json

[
  { "role": "system", "content": "You are a strict grader. Reply with a single label." },
  { "role": "user", "content": "Question: {{input}}\nAnswer: {{output}}\nIs the answer correct?" }
]

metadata.yaml

owner: test-team
purpose: demo

scorers_demo.sh

#!/usr/bin/env bash
# Showcase for `bt scorers`. Run `bt auth` and select a project first.
# Reads messages.json and metadata.yaml from this directory (@ = read from file).

# 1. Minimal score scorer: --choice-scores maps each label to a score in [0,1].
bt scorers create "Correctness" --slug correctness \
  --model gpt-5.4-nano \
  --messages @messages.json \
  --choice-scores '{"correct":1,"incorrect":0}' \
  --if-exists replace

# 2. Fully-configured LLM judge (every knob mirrors the web UI).
bt scorers create "Quality judge" --slug quality-judge \
  --model gpt-5.4-nano \
  --messages @messages.json \
  --choice-scores '{"pass":1,"fail":0}' \
  --max-tokens 512 --top-p 0.9 \
  --template-format mustache \
  --pass-threshold 0.7 \
  --metadata @metadata.yaml \
  --if-exists replace

# 3. Classifier: label output instead of a score. --allow-no-match lets it abstain.
bt scorers create "Safety label" --slug safety-label \
  --model gpt-5.4-nano \
  --messages @messages.json \
  --classifications '["safe","unsafe"]' \
  --allow-no-match \
  --if-exists replace

# 4. Update in place; only the fields you pass change. -y skips the prompt.
bt scorers update quality-judge --pass-threshold 0.8 -y
bt scorers update quality-judge --patch '{"prompt_data":{"options":{"params":{"top_p":0.95}}}}' -y

# 5. Inspect.
bt scorers list
bt scorers view quality-judge

# Cleanup (optional).
# bt scorers delete correctness --force
# bt scorers delete quality-judge --force
# bt scorers delete safety-label --force

After using the script:

When trying to update a parameter that isn't supported:

❯ bt scorers update --temperature 1
✔ Select scorer · Correctness [slug: correctness]
error: --temperature is not supported by model 'gpt-5.4-nano' unless reasoning effort is 'none'; pass `--reasoning-effort none` or omit `--temperature`
If this seems like a bug, file an issue at https://github.com/braintrustdata/bt/issues/new and include `bt --version`, `bt status --json`, and the command you ran.

Note: the check is done on bt's side, soon even if it passes it will be done in the backend too. The check is imperfect.

TODO API URL ISN'T LISTENED TO

[github-actions]

Latest downloadable build artifacts for this PR commit 505f1a9d3987:

• Workflow run: https://github.com/braintrustdata/bt/actions/runs/30603360630
• Download all artifacts (GitHub CLI): gh run download 30603360630 --repo braintrustdata/bt
• Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

• artifacts-build-global
• artifacts-build-local-x86_64-apple-darwin
• artifacts-build-local-aarch64-pc-windows-msvc
• artifacts-build-local-x86_64-pc-windows-msvc
• artifacts-build-local-x86_64-unknown-linux-gnu
• artifacts-build-local-x86_64-unknown-linux-musl
• artifacts-build-local-aarch64-apple-darwin
• artifacts-build-local-aarch64-unknown-linux-gnu
• artifacts-plan-dist-manifest
• cargo-dist-cache