Skip to content

benchmarks: add baseline planner comparison#20

Merged
programmablemanufacturing merged 2 commits into
programmablemanufacturing:mainfrom
kgarg2468:kgarg/baseline-planner-comparison
May 29, 2026
Merged

benchmarks: add baseline planner comparison#20
programmablemanufacturing merged 2 commits into
programmablemanufacturing:mainfrom
kgarg2468:kgarg/baseline-planner-comparison

Conversation

@kgarg2468
Copy link
Copy Markdown
Contributor

Summary\n- add a beginner-friendly baseline planner comparison example for the toy process-window benchmark\n- compare random sampling against grid search using the existing synthetic physics objective\n- print objective, quality, defect risk, feasibility, and selected physical process settings\n\nFixes #15\n\n## Test Report\n- cd benchmarks/toy-process-window/industrial-world-model && PYTHONPATH=src python3 examples/compare_baseline_planners.py

@programmablemanufacturing
Copy link
Copy Markdown
Owner

Thanks Krish. This is a useful direction and fits the goal of making the toy benchmark easier for new contributors to understand.

Before merging, could you make one small correction to the feasibility reporting? Right now feasible appears to mean whether any feasible candidate was found in the evaluated set, while the selected candidate is chosen by best objective. That could be misleading in the printed comparison.

Could you either:

  1. set feasible=best.feasible so it reports whether the selected candidate is feasible, or
  2. keep a separate field such as any_feasible / feasible_count?

Also, GitHub is flagging hidden/bidirectional Unicode text in the file. Please clean that up as well.

After those changes, I think this is a good beginner-friendly baseline example to merge.

@programmablemanufacturing
Copy link
Copy Markdown
Owner

programmablemanufacturing commented May 26, 2026

Thanks, Krish! I rechecked locally, and the file looks fine regarding hidden Unicode / line endings, so please ignore that part of my earlier comment.

The only remaining point is the feasibility reporting. Right now, feasible=feasible_count > 0 means any evaluated candidate was feasible, not whether the selected candidate is feasible. Could you change it to feasible=best.feasible, or rename it to something like any_feasible?

@kgarg2468
Copy link
Copy Markdown
Contributor Author

Addressed the feasibility-reporting review note.

The comparison result now reports feasible for the selected/best candidate action rather than whether any evaluated candidate was feasible. This makes the table match the action that is actually being reported.

Validation run: PYTHONPATH=src python3 examples/compare_baseline_planners.py completed successfully.

@programmablemanufacturing programmablemanufacturing merged commit bffdb6d into programmablemanufacturing:main May 29, 2026
@programmablemanufacturing
Copy link
Copy Markdown
Owner

Merged. Thanks for the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants