Skip to content

Simplify the evels that were needed and make them pass on OpenCode#769

Merged
potiuk merged 2 commits into
apache:mainfrom
justinmclean:improve-eval-tests
Jul 7, 2026
Merged

Simplify the evels that were needed and make them pass on OpenCode#769
potiuk merged 2 commits into
apache:mainfrom
justinmclean:improve-eval-tests

Conversation

@justinmclean

Copy link
Copy Markdown
Member

Summary

Run all 1000+ evals on OpenCode and Claude - all pass on Claude, after these changes only 3 fail on OpenCode.

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other: evals

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other:

@justinmclean justinmclean self-assigned this Jul 7, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coool!

@potiuk potiuk merged commit e90649c into apache:main Jul 7, 2026
8 checks passed
@justinmclean

justinmclean commented Jul 7, 2026

Copy link
Copy Markdown
Member Author

Note that, as the tests are not 100% deterministic, we may need a few more tweaks, but it's in a good place, I feel.

@potiuk

potiuk commented Jul 7, 2026

Copy link
Copy Markdown
Member

Note that, as the tests are not 100% deterministic, we may need a few more tweaks, but it's in a good place, I feel.

Yeah. We might want to improve it and automate further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants