test(discovery): enforce manifest invariants in CI - #59
Closed
pengyuzhang wants to merge 5 commits into
Closed
Conversation
Turns the inventory in tests/README.md into data the harness can execute: every row becomes an entry with a stable id, the platforms it applies to, the block that describes how it arrives, and the fields the collector is expected to get right about it. TOML rather than YAML so the file keeps its comments - the manifest is reviewed like code, and the reasoning beside an entry is most of its value - and so the loader needs nothing but the standard library. Four entry shapes cover all 120 rows, distinguished by which block an entry carries: install, declare, create or state. The recipe family is derived from that block rather than declared beside it, so the two can never disagree about how an entry is executed. Validation happens at load: an unpinned package, a variant naming a base that does not apply on its platform, or an unknown install method fails immediately rather than twenty minutes into a VM run.
Three properties of the manifest that need no VM, and so belong in ordinary per-commit CI beside the fast tests rather than in the expensive instrument. Every catalog entry has a manifest row. This is the one that keeps the inventory honest over time: a catalog entry nothing verifies is a tool the collector claims to recognize and that nothing ever checks, so adding one should fail CI until somebody adds the row. Every id is unique and every series contiguous. A gap is almost always a deleted row rather than a deliberate hole, and a duplicate id silently makes one of the two rows unscoreable. Every canary referenced is declared, and every canary declared is planted. An undeclared canary is worse than a missing one: it is planted, never searched for, and the run then reports a clean redaction check it never made. The catalog is an input rather than a path this package goes looking for - the harness treats the collector as a black box - and a run given no catalog says the coverage check did not run rather than reporting it passed.
This was referenced Aug 22, 2026
lghupan
previously approved these changes
Aug 23, 2026
# Conflicts: # Discovery/tests/manifest.py # Discovery/tests/test_manifest.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #58.
Three checks that are properties of the manifest rather than of a machine, plus the workflow job that runs them. They take milliseconds and need no VM, so the expensive instrument never has to be the thing that discovers a typo.
The checks
Every catalog entry has a manifest row. The one that keeps the inventory honest as the catalog grows. A catalog entry with no row is a tool the collector claims to recognize and that nothing ever verifies — adding one should fail CI until somebody adds the row. This single check does more than any process rule.
Every id is unique and every series contiguous. A gap is almost always a deleted row rather than a deliberate hole; a duplicate silently makes one of the two rows unscoreable.
Every
{{canary:x}}has a declaredx, and every declared canary is planted. An undeclared canary is worse than a missing one — it gets planted, never searched for, and the run reports a clean redaction check it never made.The catalog is an input
The harness does not read the collector's tree, so
--catalogis passed in. A run given no catalog printsnot checkedrather than reporting a pass: a check that did not run is not a check that passed, and the difference matters when this gates a merge.Once
adr_discovery/catalog.jsonis on main, the CI step becomes--catalog adr_discovery/catalog.jsonand the coverage check starts gating.Unresolved vendor sources are reported, not failed
57 app-installer and vendor-binary descriptors still have no download URL. That blocks one entry each; it does not invalidate the manifest, so it prints and moves on.
Verification