feat: publish immutable golden-path evidence bundle#2
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c08ba32a97
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for result in evaluation["results"]: | ||
| _validate_contract(contract_root, "evaluation-result.schema.json", result) |
There was a problem hiding this comment.
When evaluation evidence contains returned lifecycle events, this loop validates only the top-level EvaluationResult records and never validates evidence[*].execution.events. The committed eval-evidence.json therefore passes despite its event objects omitting required canonical RunEvent fields such as kind, repo, actor, timestamp, and schemaVersion, violating the AGENTS.md requirement to preserve canonical cas-contracts v0.1 lifecycle event shapes and allowing CI to certify noncanonical events.
Useful? React with 👍 / 👎.
| if verification_result["outcome"] != "passed": | ||
| raise EvidenceVerificationError("canonical VerificationResult must pass") |
There was a problem hiding this comment.
When verification-result.json keeps its aggregate outcome as passed but any individual check has outcome: failed, the canonical schema still accepts the record and this condition lets the bundle verify successfully. Because verification-result.json is not covered by an artifact digest, even a direct edit introducing a failed check passes CI; reject failed checks or otherwise enforce consistency between the aggregate outcome and its checks.
Useful? React with 👍 / 👎.
| path = bundle_root / section["path"] | ||
| if section["sha256"] != _sha256(path): |
There was a problem hiding this comment.
When verifying a bundle supplied through the CLI, section paths are joined without the containment check used for artifact paths. For example, a descriptor can set platformWhatIf.path to ../../outside.ps1 and its matching digest, and verification succeeds while relying on an uncommitted external file, defeating the verifier's committed-bundle boundary; resolve each section path and reject it unless it remains under bundle_root.
Useful? React with 👍 / 👎.
| if evaluation.get("summary") != {"failed": 0, "passed": 1, "total": 1}: | ||
| raise EvidenceVerificationError("golden path evaluation did not pass exactly one case") |
There was a problem hiding this comment.
When an evaluation artifact reports the required passing summary but contains an EvaluationResult whose outcome is failed or inconclusive, the result remains schema-valid and verification still succeeds because only the hard-coded summary is checked. This allows the verifier to certify a golden-path pass that contradicts the canonical evaluation result; require exactly one result and confirm that its outcome is passed before accepting the summary.
Useful? React with 👍 / 👎.
2 participants
Publishes a canonical ArtifactManifest and VerificationResult for the verified local reference-product to evals golden path. Every included artifact and source SHA is digest-verified offline. Claims are bounded: no Azure deployment or immutable container digest is claimed. Validation: repository validation passed with 40 tests, 90.88% coverage, strict typing/linting, and evidence verification.