[CRE] Mixed-env topology + automatic non-determinism detection - #23290
Open
prashantkumar1982 wants to merge 2 commits into
Open
[CRE] Mixed-env topology + automatic non-determinism detection#23290prashantkumar1982 wants to merge 2 commits into
prashantkumar1982 wants to merge 2 commits into
Conversation
product-security-plaid-production
Bot
requested review from
Tofel,
patrickhuie19,
tvc-robsondebraga and
vyzaldysanchez
July 31, 2026 16:53
prashantkumar1982
marked this pull request as draft
July 31, 2026 16:53
Contributor
|
✅ No conflicts with other open PRs targeting |
prashantkumar1982
force-pushed
the
cre/mixed-env-non-determinism
branch
from
July 31, 2026 19:08
41c2e2f to
b4be2da
Compare
Contributor
|
I see you updated files related to
|
prashantkumar1982
force-pushed
the
cre/mixed-env-non-determinism
branch
from
July 31, 2026 20:11
b4be2da to
8b25b56
Compare
Tofel
reviewed
Aug 3, 2026
| exit 1 | ||
| fi | ||
| echo "mixed-env non-determinism check: no markers found" | ||
|
|
Contributor
There was a problem hiding this comment.
@prashantkumar1982 what do you think about putting this mixed env in a dedicated workflow file? I am a bit weary about adding more things to this one. Soon it will be too complex to maintainable.
Adds a "mixed-env" Local CRE topology that runs each multi-node DON with 2 nodes built from the PR image and 2 from the develop image, and fails the run if the two code versions ever disagree — surfacing non-determinism and cross-version incompatibilities before merge. - configs/mixed-env-don.toml.tmpl + render-mixed-env.sh: 2-2 image split per DON via override_mode = "each"; rendered with envsubst (gitignored output). - test-helpers/nondeterminism_scan.go + smoke/cre TestMain: scan all node logs after the suite (mixed-env only) and fail with "Non-Determinism introduced". Markers: the libocr OCR3 signature-mismatch warning plus the DON2DON request/response divergence warnings. - cre-system-tests.yaml: run mixed-env for Bucket_A/B; render step resolves PR + develop-nightly images; empty chainlink-image so per-node TOML images are used; optional mixed_env_baseline_image input; exclude TestMain from discovery. - docs/mixed-env.md: developer-facing overview. Reuses the already-built per-PR image and the nightly develop image, so no extra image builds are added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
prashantkumar1982
force-pushed
the
cre/mixed-env-non-determinism
branch
from
August 3, 2026 18:20
8b25b56 to
63aada7
Compare
prashantkumar1982
marked this pull request as ready for review
August 3, 2026 18:48
Review feedback: keep cre-system-tests.yaml simple by extracting the mixed-env topology into a dedicated reusable workflow, mirroring how cre-regression-system-tests.yaml is structured. - New .github/workflows/cre-mixed-env-tests.yaml: owns the mixed-env matrix (Bucket_A, Bucket_B, EVM_Read_*), the render step, env start (empty CTF_CHAINLINK_IMAGE), and the non-determinism gate. No per-topology conditionals — every job is mixed-env, so it's simpler than the inline version. - integration-tests.yml: call it from a new run-core-cre-mixed-env-tests job, gated the same as the other CRE e2e workflows. - cre-system-tests.yaml: revert all mixed-env additions (matrix entries, render step, gate step, mixed_env_baseline_image input, chainlink-image conditional). Only the TestMain discovery exclusion remains (TestMain lives in the smoke pkg). - docs updated to point at the new workflow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[CRE] Mixed-env topology + automatic non-determinism detection
What & why
Adds a
mixed-envLocal CRE topology that runs each multi-node DON with 2 nodes on the PR image and 2 on the develop image, and fails the run if the two code versions disagree. This surfaces non-determinism and cross-version incompatibilities (e.g. things that would break a rolling upgrade) in CI, before merge — instead of after.A DON only works if its nodes compute identical results. If a change makes nodes produce different consensus reports or different cross-DON request/response payloads, the DON can't agree. Putting old (develop) and new (PR) nodes in the same DON forces any such divergence to show up as a failed check.
What it catches
While the normal smoke tests run, every node's logs are scanned for markers that mean two nodes disagreed. A single hit fails the run with
Non-Determinism introduced:This is commonly caused by non-determinism(libocr)received messages with the same id and different payloadsreceived multiple unique responses for the same request/response quorum unreachableThese lines are silent when all nodes run the same code.
Changes
configs/mixed-env-don.toml.tmpl(workflow + capabilities DONs atoverride_mode = "each", 2 PR + 2 develop; bootstrap/gateway on the baseline),configs/render-mixed-env.sh(envsubst renderer),.gitignorefor the rendered file.test-helpers/nondeterminism_scan.go(ScanContainersForNeedles) +smoke/cre/nondeterminism_check_test.go(TestMainscans node logs after the suite, mixed-env only).cre-system-tests.yaml): mixed-env added toTest_CRE_V2_Suite_Bucket_Aand_BviaPER_TEST_TOPOLOGIES_JSON;TestMainexcluded from test discovery; render step resolves PR + develop-nightly images; emptychainlink-imagefor mixed-env so per-node TOML images win; optionalmixed_env_baseline_imageinput.core/scripts/cre/environment/docs/mixed-env.md.Reuses the already-built per-PR image and the nightly develop image — no extra image builds.
Design notes
CTF_CHAINLINK_IMAGEis required for mixed-env (a non-empty value forces one image on all nodes) — handled in the workflow.