build: fetch the test data instead of vendoring it as submodules

[andiwand]

🤖 Generated with Claude Code

Groundwork for shipping the Apple framework over Swift Package Manager, but worth landing on its own.

The four test-data repositories under test/data/ become a CMake fetch pinned in test/data.cmake and driven by cmake/test_data.cmake when ODR_TEST is on. Only conan-odr-index stays a submodule.

Why

SwiftPM runs git submodule update --init --recursive on a package repository at every consumer's checkout. A root Package.swift would therefore make everyone clone 4.5 GB of test data and then fail outright on the two private remotes. This repo's own history is 55.8 MiB; the 910 MB .git is almost entirely .git/modules/.

It also matches how the repo already treats external data — tools/pdf/cmap-resources, glyphlist.txt and afm/ are fetched into gitignored paths, and odr.js already comes down via FetchContent. The test data was the outlier.

Behaviour

• Configure time only clones what is missing and warns about drift. It never moves an existing checkout: during a reference-output regeneration the local revision is ahead of the pin on purpose, and silently resetting it would throw the work away.
• update_test_data moves checkouts onto the pins, refusing on a dirty tree.
• Checkouts stay in the source tree rather than the build tree: the path is baked into test_info.cpp, three build directories would otherwise hold three copies, and reference-output/* has to remain a working copy the regeneration workflow can commit from.
• Advancing a pin replaces git add-ing a submodule, and shows up as a readable diff.

Authentication stays git's business — the script shells out to plain git. The test job authenticates with gh auth setup-git instead of carrying the PAT as its checkout token, so the token never lands in a git config file. The build job passes ODR_FETCH_TEST_DATA=OFF because it compiles odr_test without running it.

Verification

All five paths exercised against a real repository: clone, silent no-op, drift warning that leaves the checkout alone, dirty-tree refusal, and clean update. odr_test builds and the data-reading suites pass.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[Copilot]

Pull request overview

This PR replaces the large test/data/* git submodules with a CMake-driven fetch mechanism (pinned via commit SHAs) to avoid forcing SwiftPM consumers to clone multi-GB test data and to handle private test-data repos without baking credentials into checkout.

Changes:

• Add test/data.cmake pin set and a new cmake/test_data.cmake fetch/update script for test inputs + reference output.
• Wire fetching into the test build via ODR_FETCH_TEST_DATA and provide an update_test_data target for pin-aligned updates.
• Update repo metadata/CI to remove test-data submodules, ignore fetched folders, and fetch data explicitly in the test job.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/data.cmake	Introduces the pinned test-data repo list (URL + revision per checkout).
test/CMakeLists.txt	Includes the fetch script during test configure (optional) and adds update_test_data target.
CMakeLists.txt	Adds ODR_FETCH_TEST_DATA option to control configure-time fetching.
cmake/test_data.cmake	Implements clone/update logic for pinned repositories under test/data/.
AGENTS.md	Updates contributor/build documentation to reflect fetched (not submodule) test data.
.gitmodules	Removes the four test-data submodules, leaving only conan-odr-index.
.gitignore	Ignores fetched test/data/input/odr-* and test/data/reference-output/odr-* checkouts.
.github/workflows/build_test.yml	Stops using submodules for test data; authenticates via gh auth setup-git and fetches pinned data in the test job; disables fetching in build jobs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[andiwand]

Force-pushed: the test job failed with fatal: could not read Username for 'https://github.com'.

gh auth setup-git installs a credential helper that shells back out to gh auth git-credential, which reads GH_TOKEN from the environment at the moment git asks it — not when setup-git ran. The token was set only on the auth step, so the fetch step had none and git fell through to prompting.

GH_TOKEN now sits on the job.

Also, per review: the option is renamed ODR_TEST_FETCH_DATA and defaults to OFF. Building odr_test does not need the data, and several gigabytes should not arrive because someone turned tests on. Opt in with -DODR_TEST_FETCH_DATA=ON. The build job no longer needs to pass the flag at all.