feat: eval results#23
Draft
Ki-Seki wants to merge 69 commits into
Draft
Conversation
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
Member
Author
Note: Eval Bench Sizes#(Sculpt-AI/GIM-SFT/*/train) = 3157829 |
for more information, see https://pre-commit.ci
Ki-Seki
force-pushed
the
feat/eval-results
branch
from
3dd512e to
c9eef17
Compare
for more information, see https://pre-commit.ci
* chore: backup results * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * chore: remove gim prompt about gim models * chore: update api model eval scripts * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * chore: fix api model eval scripts * chore: fix api model eval scripts * chore: ignore eval.log.* files (#76) * feat: add upper bound to auto reason_budget (#78) * chore: backup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Shichao Song <60967965+Ki-Seki@users.noreply.github.com>
…odel configurations (#90) * feat: add humaneval infilling benchmark support Agent-Logs-Url: https://github.com/SculptAI/GIMBench/sessions/3aa3039a-4ca0-489f-be58-5c1f5197d1ad Co-authored-by: Ki-Seki <60967965+Ki-Seki@users.noreply.github.com> * feat: add code infilling eval task for humaneval_infilling (generate + unit test execution) Agent-Logs-Url: https://github.com/SculptAI/GIMBench/sessions/7bc171d3-b30c-4480-b88b-a18cb8dbd051 Co-authored-by: Ki-Seki <60967965+Ki-Seki@users.noreply.github.com> * fix: update dataset path and simplify loading logic for humaneval infilling * feat: enhance query formatting for code infilling with markdown code fences * feat: add eval script for humaneval infilling with multiple model configurations * upload eval results * feat: add code extraction method for fenced code blocks in CommonCodeInfillingEvaluator * update eval results --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Duguce <zhgyqc@163.com>
* feat: add throughput (tokens/s) and TTFT metrics for ppl and match evaluators Agent-Logs-Url: https://github.com/SculptAI/GIMBench/sessions/755d574f-33d3-4352-acfb-cf63ce27d892 Co-authored-by: Ki-Seki <60967965+Ki-Seki@users.noreply.github.com> * feat: move counter_tokenizer argument * feat: refactor TTFT measurement for OpenAI and vllm models * feat: add evaluation script for vllm models with various output types * feat: add timing metrics recording option to evaluators * feat: add evaluation script for vllm models with timing metrics * upload results * update eval script * add results * add results --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* upload eval results * upload
This comment was marked as outdated.
This comment was marked as outdated.
Co-authored-by: Copilot <copilot@github.com>
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.