test(discovery): read collector snapshots and derive the delta - #60
Closed
pengyuzhang wants to merge 1 commit into
Closed
test(discovery): read collector snapshots and derive the delta#60pengyuzhang wants to merge 1 commit into
pengyuzhang wants to merge 1 commit into
Conversation
A reader's view of the JSON the collector emits, and the set of assets that installation added. Scoring works on the delta rather than on the raw second snapshot, so residual baseline noise cancels out instead of being attributed to the manifest. A tool that changed channel between the two scans counts as added: the identity is the same but the asset id is not, and dropping it would score a tool that is demonstrably present as a miss. The harness derives the delta itself rather than importing the collector's own diff, for two reasons. The harness has to run from this directory alone, so scoring a recorded run needs nothing installed beside it. More importantly, a test that imports the thing it measures stops being able to catch a whole class of defect: a diff that dropped assets would drop them from the measurement too, and the run would score a clean sheet while quietly measuring less. The cost is that this file encodes an expectation about the snapshot format. That is deliberate - the format is the collector's published contract, and a test that fails when it changes silently is the correct outcome.
lghupan
approved these changes
Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #59.
A reader's view of the JSON the collector emits, and the computation of what installation added.
Scoring works on the delta
Not on the raw second snapshot. Residual baseline noise then cancels out instead of being attributed to the manifest. The baseline is separately asserted to be near-empty, so anything it reported is a false positive with nothing to blame — which fails the run before installation begins.
A tool that changed channel between scans counts as added: same identity, different asset id. Dropping it would score a tool that is demonstrably present as a miss.
A snapshot that repeats an
asset_idis refused rather than scored. It would silently halve the delta, and any number computed from that is wrong in the direction that flatters the collector.Why this does not import the collector
The plan reaches for
diff_snapshotsfromadr_discovery. This does not, for two reasons.The practical one: the harness must run from
Discovery/tests/alone, so scoring a recorded run needs nothing installed beside it.The load-bearing one: a test that imports the thing it measures stops being able to catch a whole class of defect. If the scorer computed "what arrived" with the collector's own diff, a diff that dropped assets would drop them from the measurement too — the run would score a clean sheet while quietly measuring less. Re-deriving means the two definitions can disagree, and a disagreement is exactly the finding worth having.
The cost is that
snapshot.pyencodes an expectation about the snapshot format. That is deliberate: the format is the collector's published contract, and a test that fails when it changes silently is the correct outcome.Note on
serialized()The canary check searches the original document, not a re-serialization of the parsed model. A credential that leaked into a field the harness does not model is exactly the one that would otherwise be missed.
Verification