Skip to content

Support baseline-referenced SGES for single-trial experiments - #5267

Closed
umutgunduzfb wants to merge 1 commit into
facebook:mainfrom
umutgunduzfb:export-D114376941
Closed

Support baseline-referenced SGES for single-trial experiments#5267
umutgunduzfb wants to merge 1 commit into
facebook:mainfrom
umutgunduzfb:export-D114376941

Conversation

@umutgunduzfb

Copy link
Copy Markdown
Contributor

Summary:
Stability-gated early stopping requires at least two comparable curves. Add an explicit baseline-reference hook so integrations can provide a comparison curve when an experiment has one candidate.

Inject the cleaned raw baseline as a pseudo-trial before progression normalization. Ax already aligns curves sampled at different progressions and SGES interpolates them at decision checkpoints, so no integration-specific resampling is needed. Preserve unknown baseline uncertainty as NaN. Existing multi-candidate behavior remains unchanged.

Expose a reference-needed query based on the same attached-data condition used for injection. It returns true only when the selected metric has exactly one candidate curve and no reference is cached, allowing integrations to avoid unused and repeated baseline fetches.

Context:
A one-candidate experiment has no peer curve, so SGES cannot satisfy its comparison requirements even when GAIN has a completed baseline. The reference hook supplies only the missing comparator. Restricting injection and fetching to exactly one attached candidate curve avoids changing leader selection, top-k protection, and inferred stability thresholds for existing multi-candidate experiments.

Reviewed By: saitcakmak

Differential Revision: D114376941

Summary:
Stability-gated early stopping requires at least two comparable curves. Add an explicit baseline-reference hook so integrations can provide a comparison curve when an experiment has one candidate.

Inject the cleaned raw baseline as a pseudo-trial before progression normalization. Ax already aligns curves sampled at different progressions and SGES interpolates them at decision checkpoints, so no integration-specific resampling is needed. Preserve unknown baseline uncertainty as NaN. Existing multi-candidate behavior remains unchanged.

Expose a reference-needed query based on the same attached-data condition used for injection. It returns true only when the selected metric has exactly one candidate curve and no reference is cached, allowing integrations to avoid unused and repeated baseline fetches.

Context:
A one-candidate experiment has no peer curve, so SGES cannot satisfy its comparison requirements even when GAIN has a completed baseline. The reference hook supplies only the missing comparator. Restricting injection and fetching to exactly one attached candidate curve avoids changing leader selection, top-k protection, and inferred stability thresholds for existing multi-candidate experiments.

Reviewed By: saitcakmak

Differential Revision: D114376941
@meta-cla meta-cla Bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Aug 4, 2026
@meta-codesync

meta-codesync Bot commented Aug 4, 2026

Copy link
Copy Markdown

@umutgunduzfb has exported this pull request. If you are a Meta employee, you can view the originating Diff in D114376941.

@meta-codesync

meta-codesync Bot commented Aug 4, 2026

Copy link
Copy Markdown

This pull request has been merged in d389183.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.73684% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.58%. Comparing base (1ffbb75) to head (be23398).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
ax/early_stopping/strategies/stability_gated.py 90.32% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5267      +/-   ##
==========================================
- Coverage   96.59%   96.58%   -0.01%     
==========================================
  Files         621      621              
  Lines       70745    70802      +57     
==========================================
+ Hits        68335    68384      +49     
- Misses       2410     2418       +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. Merged meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants