Add multi-dataset model run prediction uploads

[luke-e-schaefer]

Summary — v0.20.0: multi-dataset model runs

Stacked on #467 (update-nuc-sdk-for-new-eval-stuff-pt1) — review that first; this PR's diff is only the 7 files below. The SDK-visible half of the scaleapi change that gives a model run a resolved set of datasets instead of one declared dataset.

The one new method

dataset_b.upload_predictions_for_model_run(run_id, predictions)

Posts to dataset/{dataset_id}/modelRun/{model_run_id}/uploadPredictions, adding dataset_b to the run's dataset set. That is what lets a single run be scored against a benchmark whose items span several datasets — previously create_benchmark_evaluation_v2 returned a 400 for exactly that case.

Same update / asynchronous / batch_size / file-batching / trained_slice_id arguments as upload_predictions, and it runs the same duplicate-id check.

What is deliberately not changed

Dataset.upload_predictions still cannot widen a run, and that's the point. It identifies the run by (dataset, model), so it finds the run already on this dataset or creates a new one. Widening got its own endpoint rather than loosening this one — mirroring the server, where the existing route kept its never-widen contract exactly as it was.

ModelRun.predict also stays on the old route. It's deprecated, and switching it would silently turn a stale dataset_id passed to get_model_run() into a widening upload. Its docstring now says it fails for multi-dataset runs and points at the new method.

Routing

PredictionUploader now accepts dataset_id together with model_run_id — previously an assertion rejected the pair — and picks the endpoint from which identifiers are present:

Arguments	Route	Can widen?
dataset_id + model_run_id	dataset/{ds}/modelRun/{run}/uploadPredictions	Yes
dataset_id + model_id	dataset/{ds}/model/{model}/uploadPredictions	No
model_run_id alone	modelRun/{run}/predict	No (deprecated)

The two pre-existing forms route exactly as before.

Access, and the sharp edge

Widening requires write on this dataset and on every dataset the run already covers. That's stricter than it looks necessary, for a reason worth knowing: a run is visible only to users who can read all of its datasets, so adding a dataset to a run can remove that run from a collaborator's view. The set also only ever grows — a later upload never drops a dataset, so it can't widen who can read the run.

Docstring corrections

Both are wrong as of the server change, not merely incomplete:

• create_benchmark_evaluation_v2 / Benchmark.create_evaluation_v2 said the run's predictions "must cover items from the benchmark's datasets". The server enforced that with a 400; it no longer does. Coverage may be partial or empty, and uncovered members score as false negatives.
• ModelRun.predict — see above.

Tests

tests/test_multi_dataset_model_runs.py — 9 mock-based tests pinning all three routes (the whole difference between them is the route), the async route, trained_slice_id forwarding, and that the new entry point runs the duplicate-id check rather than bypassing it.

Ran locally: 68 passed across the new file plus the benchmark / eval-v2 / preset / leaderboard suites. black and isort clean on the changed files. Diff is purely additive — no reformatting churn from a newer local black.

Server dependency: the new route ships with the multi-dataset model-run work in scaleapi (#154100). Unit tests pass regardless; live calls 404 until that deploys.

🤖 Generated with Claude Code

Greptile Summary

Adds multi-dataset model-run prediction uploads.

• Adds Dataset.upload_predictions_for_model_run, including synchronous, asynchronous, batching, duplicate-ID, and trained-slice handling.
• Extends PredictionUploader routing to support uploads identified by both dataset and model-run IDs.
• Documents revised benchmark coverage behavior and the limitation of deprecated ModelRun.predict.
• Adds route-focused tests and bumps the package version to 0.20.0.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
nucleus/dataset.py	Adds the public synchronous and asynchronous multi-dataset prediction-upload entry point using existing upload machinery.
nucleus/annotation_uploader.py	Extends prediction routing to accept dataset and model-run identifiers together while retaining both existing route forms.
tests/test_multi_dataset_model_runs.py	Covers all three routing forms, asynchronous routing, option forwarding, and duplicate-ID validation.
nucleus/init.py	Updates benchmark-evaluation documentation to describe partial or empty dataset coverage.
nucleus/benchmark.py	Updates the benchmark convenience API documentation for partial model-run coverage.
nucleus/model_run.py	Documents why the deprecated prediction method cannot support multi-dataset runs.
pyproject.toml	Bumps the package version from 0.19.1 to 0.20.0.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Prediction upload] --> B{Identifiers supplied}
    B -->|dataset_id + model_run_id| C[dataset/dataset_id/modelRun/model_run_id/uploadPredictions]
    B -->|dataset_id + model_id| D[dataset/dataset_id/model/model_id/uploadPredictions]
    B -->|model_run_id only| E[modelRun/model_run_id/predict]
    C --> F[Run may gain the selected dataset]
    D --> G[Resolve or create run for dataset and model]
    E --> H[Deprecated single-dataset inference path]

Loading

_{Reviews (2): Last reviewed commit: "Add multi-dataset model run prediction u..." | Re-trigger Greptile}