Add data-query category: AL query-generation benchmark#740
Draft
onbuyuka wants to merge 11 commits into
Draft
Conversation
Adds a new execution-based `data-query` category that benchmarks models/agents at generating Business Central AL queries. Given a natural-language data question, the agent writes a single AL query object to query.al; evaluation compiles and runs both the generated query and a gold reference query against the container's Contoso demo data and compares the result sets. No MCP server and no LLM judge. - types.py: DATA_QUERY -> execution-based (ExecutionBasedEvaluationResult, summary, aggregate; resolution_rate/build_rate; ResolutionRate; requires_container; GitHub-BCBench) - DataQueryEntry: nl_prompt + gold_query + ordered; dataset/dataquery.jsonl (6 tasks) - DataQueryPipeline + result_sets_match (value-based, order-insensitive; unit-tested) - operations: wrap_query_as_api (unit-tested) + execute_al_query (wrap as API query, publish throwaway app, read OData) - ExecutionBasedEvaluationResult.create_result for compiled-but-wrong outcomes - config.yaml: data-query prompt (author query.al); al-query-authoring skill - Setup-ContainerAndRepository.ps1: skip repo clone for data-query (no repo), just provision the sandbox container; a stock Contoso artifact suffices - Wire data-query into the copilot/claude evaluation workflow category choices + docs Container round-trip in execute_al_query and the gold AL query bodies need validation on a runner (no local BC container); pure logic is unit-tested (592 tests pass). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
…atch) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
…d into container) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
Groenbech96
requested changes
Jul 13, 2026
Groenbech96
left a comment
Groenbech96
left a comment
Collaborator
There was a problem hiding this comment.
Looks good. Good stuff.
| [string] $commitSha = $entries[0].base_commit | ||
| # Some categories (e.g. data-query) generate code from scratch rather than editing an existing | ||
| # repository, so there is nothing to clone -- the agent just needs an empty working directory. | ||
| [string[]] $noCloneCategories = @('data-query') |
Collaborator
There was a problem hiding this comment.
Would it not in real life have the files available?
So maybe its not to bad to still clone, to simulate real environement.
| case EvaluationCategory.BUG_FIX | EvaluationCategory.TEST_GENERATION: | ||
| return True | ||
| case EvaluationCategory.DATA_QUERY: | ||
| # Data-query provisions its own container: it publishes the data-query seed app |
Collaborator
There was a problem hiding this comment.
lets remove. Verbose and odd placement.
| case EvaluationCategory.NL2AL: | ||
| return "windows-latest" | ||
| case EvaluationCategory.DATA_QUERY: | ||
| # The agent queries a live BC /mcp endpoint; the self-hosted runner is where a |
… (AL0124) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
A gold query failing to compile/run is a harness/dataset problem, not the agent's, so record it as a non-resolved result with a clear message instead of letting the uncaught BuildError crash the whole matrix job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
… sets The object name is irrelevant to a query's result set (we score by comparing data), but AL requires it be a valid <=30-char identifier and unique in the tenant. Two of the first real runs failed only on AL0305 (agent chose a long descriptive name), so normalize the name in wrap_query_as_api to keep the benchmark focused on query logic. Also give the generated and gold API queries distinct EntitySetName/EntityName so both can be published to the same tenant without colliding on the OData route once a generated query finally compiles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
… fetch Root cause of the 0/4 build rate: the agents were writing valid AL (e.g. a correct Vendor/Purch. Inv. Header query) but the compiler reported base tables as missing (AL0185, '26.0.0.0 could not be found in the database'). The custom Compile-AppInBcContainer -UpdateSymbols path did not load Base Application symbols reliably (intermittent across containers). Switch execute_al_query to the same Invoke-AppBuildAndPublish helper the passing categories use (explicit cleared .alpackages symbol folder, GenerateReportLayout No, ForceSync, dependencyPublishingOption ignore). Also fetch the query rows from *inside* the container (Invoke-ScriptInBcContainer -> http://localhost:7048/BC/api) so we no longer depend on host->container name resolution or published ports, which the runner does not set up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
…a fetch
Run 3 showed the compile+publish now works (Base App symbols resolve via the
proven helper), but the in-container OData fetch failed: PowerShell 7 refuses
Invoke-RestMethod -Credential over plain HTTP ('cannot protect plain text
secrets sent over unencrypted connections'). Build the Basic Authorization
header manually instead, which works on both Windows PowerShell 5.1 and
PowerShell 7. Add regression tests asserting the run template uses the proven
build helper, fetches from inside the container, and never passes -Credential
over HTTP.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
Run 4 proved the harness works end-to-end (build=2, real gold-vs-generated
result-set comparisons). The remaining resolved=0 was down to prompt ambiguity
and one buggy gold, not the harness:
- Tighten all prompts so a correct interpretation deterministically matches the
gold: specify the measure and whether it is net of VAT, the source (line vs
header, posted vs open), inner-join inclusion ('...that has at least one...'),
and grouping. E.g. the vendor prompt now pins line-level Amount net of VAT
(a model had reasonably summed header Amount Including VAT -> 5 vs 6 rows).
- Replace 'items on both open orders': its gold expressed a set intersection as
a join with no aggregate column, so an AL query returns one row per matching
(sales line x purchase line) pair instead of the distinct item set and cannot
be scored deterministically (13 vs 12 rows). Swap in a clean aggregate join
(open sales order count per customer).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: db34a2a0-7035-4361-b911-becb72f86e21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a new
data-queryevaluation category: an offline benchmark for AL query generation. Given a natural-language data question, the agent authors a single ALqueryobject (query.al) and it's scored deterministically — compile + run the generated query and a gold reference query against the container's Contoso demo data, pass if the result sets match. No MCP server, no LLM judge.This complements the AI Test Toolkit evals in the platform repo: those test the MCP server end-to-end; this benchmarks models/agents on query generation.
How it works
dataset/dataquery.jsonl) — each entry hasnl_prompt+gold_query(the reference AL query whose result set defines "correct") +environment_setup_version+ordered.evaluate/dataquery.py) — the agent writesquery.al; the harness compiles + runs the generated and gold queries and compares result sets.build= generated query compiled and ran;resolved(ResolutionRate) = result set matches gold.ordered.operations/bc_operations.py) —wrap_query_as_apiturns each query into an API query;execute_al_querypublishes a throwaway app to the container and reads its OData endpoint.requires_container = True, runnerGitHub-BCBench) — a stock BC sandbox artifact (Cronus/Contoso data) suffices; no special build needed.Setup-ContainerAndRepository.ps1skips the git clone for data-query (there's no repo) and just provisions the container.How to run
Actions → Evaluation with GitHub Copilot (or Claude Code) → Run workflow → category =
data-query. The self-hosted runner provisions the container; no local setup.Tested / not tested
result_sets_match,wrap_query_as_api(tests/test_dataquery_evaluation.py); full suite 592 pass; ruff + ty clean.execute_al_querycontainer round-trip (compile/publish/OData, incl. company-scoped API path) and the 6 gold AL query bodies. The first remote run is the shakeout.