Skip to content

[P3] Stop double-running compatibility tests in full CI #213

Description

@alexeygrigorev

Problem

Push CI has two separate owners for compatibility verification, but the full Django job currently invokes the local aggregate target unconditionally:

Therefore, a full push with compatibility_mode=skip still runs compatibility once through Django, while rerun runs the same contract once in Django and once in quality. This is a CI scheduling and evidence-contract defect that adds avoidable minutes; it does not change product behavior.

Authority and current evidence

The current scheduled full-regression job intentionally calls local make test, so it is a separate full-regression contract and must remain explicit.

Scope

  1. Add a CI-only test-django-full Make target, or an equivalent explicit conditional target, that runs the existing full Django test recipe without the test-compatibility prerequisite. The preferred target keeps the local make test contract unchanged.
  2. Change the push workflow full Django branch to use that CI-only command. Keep the quality job as the sole compatibility owner when the verification plan says compatibility_mode=rerun; do not weaken the plan disposition.
  3. Update the versioned verification command and evidence contracts so the full Django command, ownership metadata where applicable, evidence allowlist, and recorded envelope all agree on the exact command. The current dynamic command mapping and command allowlist are here and here.
  4. Add or update CI contract tests and the selective-CI documentation to pin the target prerequisite, workflow mode conditions, plan command, evidence command, and local/scheduled distinction.

Non-goals

  • Do not change compatibility tests, compatibility fixtures, URL/link/SEO behavior, or the compatibility contract itself.
  • Do not remove compatibility from local make test; the scheduled full-regression job may continue to call it and must execute compatibility once.
  • Do not change the quality-contract-v1 primitive target list or its aggregate semantics.
  • Do not redesign classifier/profile selection, ownership closures, evidence reuse, or the four-hour scheduled backstop.
  • Do not change Django application code, public routes, templates, browser behavior, deployment behavior, or dependencies.

Dependencies and status

  • No blocking issue dependency is identified. The implementation depends on the current selective-CI plan/evidence contract and the existing compatibility_mode output consumed by push CI.
  • The scheduled full-regression behavior is an explicit compatibility boundary: its Django job calls make test and must stay separate from push CI.
  • Ready for engineering. If an equivalent conditional prerequisite is chosen instead of test-django-full, it must still expose an unambiguous command in the plan and evidence envelope.

Acceptance criteria

  • The CI-only full Django command runs the same full Django suite currently reached by make test, but does not depend on or execute test-compatibility. Local make test still depends on test-compatibility.
  • The push workflow full Django branch invokes the CI-only command and no longer invokes local make test.
  • With compatibility_mode=skip, the push Django and quality jobs execute no compatibility contract command, and the plan records the compatibility component as skipped.
  • With compatibility_mode=rerun, the push quality job executes the compatibility contract exactly once, the push Django job executes it zero times, and the aggregate evidence report contains one compatibility execution.
  • The scheduled full-regression path continues to run local make test and executes compatibility exactly once; existing local make test semantics are preserved.
  • The full-profile Django plan command, ownership metadata if applicable, evidence allowlist, recorded envelope, and workflow invocation agree on the exact CI-only command. The quality-contract primitive target list remains unchanged.
  • Contract tests pin the Make target prerequisite behavior and push workflow behavior for skip, rerun, and the full Django profile, including a regression assertion that the scheduled job remains on local make test.
  • make test-ci, the relevant verification plan/evidence/report checks, and the existing local/scheduled workflow contract checks pass with no product-page changes.

Implementation slices

  1. Make target: add the CI-only full Django target and preserve make test as the local compatibility-inclusive target.
  2. Plan and evidence contract: update the full Django command mapping, component command allowlist, ownership metadata if required by the selected design, and envelope/fixture expectations.
  3. Push workflow: switch only the full Django branch; retain quality compatibility gating on rerun and preserve all artifact/result recording.
  4. Tests and documentation: add focused Make/verification/workflow contract assertions for skip, rerun, full, local, and scheduled paths; document the two intentional commands and run the repository CI checks.

Verification and browser applicability

  • CI/Django/integration: verify target prerequisites and dry-run command expansion; generate and validate a current verification plan; exercise the workflow/evidence contracts; run make test-ci and the focused contract tests.
  • Repository/operations: inspect both push and scheduled workflow YAML, confirm the compatibility component is classified exactly once, and confirm no evidence command mismatch or duplicate execution.
  • Browser and screenshots: not_applicable. This is an internal CI/process change with no page-render, route, accessibility, or responsive behavior change, so no web.dtcdev.click browser session or screenshot is required.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Hygiene: pick up when a lane is freebugSomething isn't workinginfraArea: infratestingArea: testing

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions