Hi PromptSource team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets (Apache 2.0, a TestCase/Suite schema with a real JSON Schema validator, so a dataset can move between eval tools without a bespoke converter).
I read the real template format rather than guessing. promptsource/templates/ag_news/templates.yaml:
24e44a81-a18a-42dd-a71c-5b31b2d2cb39: !Template
answer_choices: World politics ||| Sports ||| Business ||| Science and technology
id: 24e44a81-a18a-42dd-a71c-5b31b2d2cb39
jinja: "What label best describes this news article?\n{{text}} ||| \n{{answer_choices[label] }}"
metadata: !TemplateMetadata
metrics:
- Accuracy
name: classify_question_firstand CONTRIBUTING.md confirms the contract: "a template must produce two strings: an input and a target," separated by |||, with answer_choices and metadata.metrics (e.g. Accuracy) as first-class fields.
That's a very close match to an EvalPort TestCase: the rendered input string → TestCase.input, the rendered target string → expected_output, answer_choices → the option set, and metadata.metrics → the EvalPort Grader type (exact_match for Accuracy, etc.). Applying a DatasetTemplates collection to its underlying HF dataset split would produce a ready-made EvalPort Suite.
Would it be of interest to have a small promptsource-openeval-adapter in the EvalPort repo that walks a DatasetTemplates/Template.apply() output into an EvalPort Suite, per dataset/subset, with attribution to P3 and this repo's license preserved? I'd do the work and validate against EvalPort's real schema — filing this as an issue first per CONTRIBUTING.md norms rather than sending a PR out of nowhere. No changes needed in this repo either way. Let me know if this is of interest, or not — no worries either way.
Hi PromptSource team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets (Apache 2.0, a
TestCase/Suiteschema with a real JSON Schema validator, so a dataset can move between eval tools without a bespoke converter).I read the real template format rather than guessing.
promptsource/templates/ag_news/templates.yaml:and CONTRIBUTING.md confirms the contract: "a template must produce two strings: an input and a target," separated by
|||, withanswer_choicesandmetadata.metrics(e.g.Accuracy) as first-class fields.That's a very close match to an EvalPort
TestCase: the rendered input string →TestCase.input, the rendered target string →expected_output,answer_choices→ the option set, andmetadata.metrics→ the EvalPortGradertype (exact_matchfor Accuracy, etc.). Applying aDatasetTemplatescollection to its underlying HF dataset split would produce a ready-made EvalPortSuite.Would it be of interest to have a small
promptsource-openeval-adapterin the EvalPort repo that walks aDatasetTemplates/Template.apply()output into an EvalPortSuite, per dataset/subset, with attribution to P3 and this repo's license preserved? I'd do the work and validate against EvalPort's real schema — filing this as an issue first per CONTRIBUTING.md norms rather than sending a PR out of nowhere. No changes needed in this repo either way. Let me know if this is of interest, or not — no worries either way.