Skip to content

[DO NOT MERGE] canary: force DON2DON non-determinism to validate mixed-env - #23291

Draft
prashantkumar1982 wants to merge 2 commits into
cre/mixed-env-non-determinismfrom
cre/mixed-env-canary-test
Draft

[DO NOT MERGE] canary: force DON2DON non-determinism to validate mixed-env#23291
prashantkumar1982 wants to merge 2 commits into
cre/mixed-env-non-determinismfrom
cre/mixed-env-canary-test

Conversation

@prashantkumar1982

Copy link
Copy Markdown
Contributor

⚠️ DO NOT MERGE — canary test

Temporary canary stacked on top of #23290 (mixed-env topology) to prove the mixed-env non-determinism check actually fires in CI.

What it does: capability_executor.go unconditionally stamps Metadata.OrgID = "canary-nondeterminism" on executable-capability requests (ignoring PropagateOrgIDInRequestMetadata). PR-built nodes then emit a different request payload than develop-built nodes.

Expected result:

  • The capability DON logs received messages with the same id and different payloads.
  • The mixed-env TestMain scan fails the mixed-env matrix entries (Test_CRE_V2_Suite_Bucket_A / _B) with Non-Determinism introduced.
  • The non-mixed-env matrix entries are unaffected.

Base is the feature branch, so the diff is just the 6-line canary. Close and delete this branch once CI has demonstrated the catch.

@github-actions

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting cre/mixed-env-non-determinism

@github-actions

Copy link
Copy Markdown
Contributor

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@trunk-io

trunk-io Bot commented Jul 31, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic Badge

Failed Test Failure Summary Logs
Test_CRE_V2_Suite_Bucket_A/Proof_Of_Reserve_-_mixed-env The test failed because a specific feed did not update within the timeout period. Logs ↗︎
Test_CRE_V2_Suite_Bucket_B/Vault_DON_-_mixed-env/allowlist_auth_when_jwt_auth_disabled The test failed during execution without providing specific error details. Logs ↗︎
Test_CRE_V2_Suite_Bucket_B/Vault_DON_-_mixed-env/allowlist_auth_when_jwt_auth_disabled/allowlist_update_batch_at_limit The test failed during a vault secret creation process, likely due to an issue with allowlisting or encrypted secret handling. Logs ↗︎
Test_CRE_V2_Suite_Bucket_B/Vault_DON_-_mixed-env/allowlist_auth_when_jwt_auth_disabled/allowlist_create_batch_at_limit The test failed during a process involving creating secrets and allowlisting requests, but the specific cause of failure is not detailed in the log. Logs ↗︎

... and 4 more

View Full Report ↗︎Docs

@prashantkumar1982
prashantkumar1982 force-pushed the cre/mixed-env-canary-test branch from 93c8bb0 to 828f32a Compare July 31, 2026 18:51
@prashantkumar1982
prashantkumar1982 force-pushed the cre/mixed-env-non-determinism branch from 41c2e2f to b4be2da Compare July 31, 2026 19:08
@prashantkumar1982
prashantkumar1982 force-pushed the cre/mixed-env-canary-test branch from 828f32a to e8e7507 Compare July 31, 2026 19:12
@prashantkumar1982
prashantkumar1982 force-pushed the cre/mixed-env-non-determinism branch from b4be2da to 8b25b56 Compare July 31, 2026 20:11
@prashantkumar1982
prashantkumar1982 force-pushed the cre/mixed-env-canary-test branch 3 times, most recently from 9ba41a3 to 604f6de Compare August 3, 2026 16:19
@prashantkumar1982
prashantkumar1982 force-pushed the cre/mixed-env-non-determinism branch from 8b25b56 to 63aada7 Compare August 3, 2026 18:20
@prashantkumar1982
prashantkumar1982 force-pushed the cre/mixed-env-canary-test branch from 604f6de to ca6403c Compare August 3, 2026 18:29
prashantkumar1982 and others added 2 commits August 3, 2026 12:30
…d-env

Temporary canary to verify the mixed-env non-determinism check catches a real
divergence in CI, without breaking unit tests.

- capability_executor.go: when CRE_CANARY_NONDETERMINISM is set, unconditionally
  stamps Metadata.OrgID on executable-capability requests (ignoring
  PropagateOrgIDInRequestMetadata). PR-built nodes then emit a different request
  payload than develop-built nodes; the capability DON logs "received messages
  with the same id and different payloads", which the mixed-env TestMain scan
  turns into "Non-Determinism introduced".
- mixed-env-don.toml.tmpl: set CRE_CANARY_NONDETERMINISM on the workflow DON so
  the canary activates only in the mixed-env e2e run (unit tests unaffected).

Revert this commit once CI has demonstrated the catch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed-env

Second canary, at the OCR3 report-attestation layer. In vault/plugin.go
wrapReportWithKeyBundleInfo, when CRE_CANARY_NONDETERMINISM is set, perturb the
SIGNED report bytes on PR-built nodes so vault OCR3 nodes disagree during report
attestation (the state/outcome is already committed by then, so this is a clean
report-phase divergence). libocr logs "This is commonly caused by non-determinism
in the ReportingPlugin", which the mixed-env gate catches (Bucket_B). Also sets
CRE_CANARY_NONDETERMINISM on the capabilities DON.

An earlier version omitted ReportInfo.Id, but that lives in the unsigned .Info and
only broke report routing (a functional test failure), not attestation.

Revert with the rest of the canary once CI has demonstrated the catch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@prashantkumar1982
prashantkumar1982 force-pushed the cre/mixed-env-canary-test branch from ca6403c to 4a18325 Compare August 3, 2026 19:30
@cl-sonarqube-production

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant