feat: consolidated pip-compile -> uv migration (all 5 stacked PRs, rebased onto master) by irfanuddinahmad · Pull Request #38915 · openedx/openedx-platform

irfanuddinahmad · 2026-07-21T13:32:59Z

Summary

Consolidated view of the pip-compile -> uv migration tracked in openedx/public-engineering#543, requested by a reviewer as a single PR based on master instead of reviewing the 5-PR stack individually.

This PR contains the combined changes from:

feat: add pyproject.toml deps, dependency-groups, and uv.lock (1/5) #38835 -- populate pyproject.toml deps/dependency-groups + commit uv.lock
feat: cut over Makefile, tox.ini, and CI to uv (2/5) #38836 -- cut over Makefile, tox.ini, and CI to uv for the main app
feat: migrate codejail sandbox to its own uv project (3/5) #38837 -- migrate the codejail sandbox to its own standalone uv project
feat: migrate standalone scripts to their own uv projects (4/5) #38838 -- migrate the three standalone scripts to standalone uv projects
chore: remove now-unused pip-compile machinery (5/5) #38839 -- remove now-unused pip-compile machinery, finalize docs

The 5 stacked PRs remain open and can still be reviewed/merged individually if preferred -- this PR is an additional, rebased-onto-current-master view for reviewers who'd rather see the whole migration in one diff.

What changed

pyproject.toml (PEP 621/735): all runtime deps and dependency-groups (test, quality, doc, ci, assets, dev) populated; uv.lock committed for the root project.
Makefile/tox.ini/CI workflows cut over from pip-compile/pip-tools to uv.
The codejail sandbox (requirements/edx-sandbox) and the three standalone scripts (scripts/xblock, scripts/user_retirement, scripts/structures_pruning) each migrated to their own standalone uv projects (own pyproject.toml + uv.lock).
Removed now-fully-unused pip-compile machinery: requirements/constraints.txt, requirements/common_constraints.txt, requirements/pip-tools.{in,txt}, and the corresponding vestigial Makefile targets/variables.
requirements/edx/{base,assets,development}.txt, requirements/edx-sandbox/base.txt, and the three scripts' compat .txt files are kept as machine-generated uv export compatibility artifacts (for tooling, e.g. Tutor's Dockerfile, that still does pip install -r ... directly) rather than deleted.

What's intentionally NOT done here (tracked externally)

find_python_dependencies: support pyproject.toml / uv.lock, not just requirements/*.txt repo-tools#725: find_python_dependencies needs pyproject.toml support before check_python_dependencies.yml (disabled as part of this migration) can be re-enabled.
Tutor's Dockerfile installs from requirements/edx/{base,assets,development}.txt with plain pip. Those stay as uv export compatibility artifacts rather than being deleted, so no action is required on Tutor's side right now -- but Tutor maintainers should be aware these paths are now machine-generated, not hand-compiled.

Verification

Rebased all 8 commits from the 5-PR stack cleanly onto current master (as of this PR), resolving conflicts against ~110 commits of drift (mostly routine dependency-version bumps in files this migration deletes or replaces, plus a handful of CI-workflow steps master added for pip-caching that are no longer needed once uv's own caching takes over).
Re-ran make compile-requirements end-to-end against the rebased uv.lock for the root project and all 4 uv sub-projects -- regenerated compatibility export files are byte-identical to what the rebase produced, confirming consistency.
uv lock resolves cleanly (445 packages, root project); all 4 sub-projects (requirements/edx-sandbox, scripts/xblock, scripts/user_retirement, scripts/structures_pruning) sync cleanly with uv sync --frozen.
Root project's own uv sync could not be fully verified on this machine (missing libmysqlclient/pkg-config to build mysqlclient from source) -- relying on CI for full install + test verification, same as the original 5-PR stack already did.

🤖 Generated with Claude Code

…ration 1/5) Populates [project.dependencies] (from kernel.in + bundled.in), adds [project.optional-dependencies] for the legacy openstack storage backend, adds PEP 735 [dependency-groups] (coverage/testing/doc/assets/development/ semgrep/ci/dev, mirroring the current .in file composition), and [tool.edx_lint].uv_constraints + generated [tool.uv].constraint-dependencies for the ~20 repo-specific version pins, with a committed uv.lock. This is purely additive: the Makefile, tox.ini, and CI workflows are untouched and continue to use pip-compile/requirements/*.txt as the source of truth. Part of the pip-compile -> uv migration tracked in openedx/public-engineering#543 (1 of 5 PRs). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…ions Found via real CI runs: a fresh uv resolution picked social-auth-core 5.0.2 (previously locked at 4.9.1 via pip-compile), which changes the OAuth pipeline's post-login redirect behavior and breaks common/djangoapps/third_party_auth's integration test suite (AzureAD, Google, LinkedIn, Twitter full-pipeline specs all failed the same assertion in tests/specs/base.py's assert_logged_in_cookie_redirect). This migration is meant to be a tooling swap, not a dependency upgrade, so pin back to the 4.x line rather than bundle an investigation into social-auth-core 5.x's behavior change into this PR. Mirrors the existing social-auth-app-django<=5.4.1 constraint, pinned for a related, already-deferred migration in this same dependency family. Follow-up tracked at #38841. Verified: all 46 previously-failing third_party_auth tests pass with social-auth-core==4.9.1 restored via this constraint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Rewrites the Makefile's requirements targets, tox.ini, and ~13 CI workflows to use uv instead of pip-compile/pip-sync for the main app. Deletes requirements/edx/*.in and *.txt (superseded by pyproject.toml + uv.lock, added in PR 1 / #38835). requirements/constraints.txt, common_constraints.txt, and pip-tools.{in,txt} are intentionally kept for now: requirements/edx-sandbox and scripts/* still pip-compile against them and aren't migrated until PR 3/4. requirements/edx/{base,assets,development}.txt are regenerated as `uv export` compatibility artifacts (via the Makefile's compile-requirements target) since external tooling -- notably tutor's Dockerfile -- installs from those exact paths with plain pip, not uv. check_python_dependencies.yml is disabled (workflow_dispatch only, job gated with if: false) since find_python_dependencies can't scan pyproject.toml yet; tracked at openedx/repo-tools#725. User-confirmed before committing since this removes a CI safety net. Part of openedx/public-engineering#543 (2 of 5). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Found via real CI runs after opening the PR (unit tests, Quality Others, and the ReadTheDocs build all failed): 1. Makefile's test-requirements used `uv sync --only-group testing`, which EXCLUDES [project.dependencies] entirely (confirmed: `--only-group` replaces the dependency set rather than adding to it, unlike `--group`). This meant Django, XBlock, and the rest of the actual application were never installed for test runs -- ModuleNotFoundError: No module named 'xblock' on every unit test shard. Fixed to `--no-default-groups --group testing`, matching what tox-uv itself generates for the equivalent tox environment. 2. .readthedocs.yaml had the same bug in its doc-requirements.txt export (`--only-group doc`). Since [project.dependencies] were excluded from that export, the subsequent `pip install -e .` resolved the whole dependency tree completely unconstrained by uv.lock's [tool.uv].constraint-dependencies -- picking setuptools==82.0.1 (violates setuptools<82) and Django==6.0.6 (violates Django<6.0). The setuptools violation broke fs/pyfilesystem2's pkg_resources import, crashing the Sphinx build via Django app loading. Fixed to `--group doc` plus `--no-deps` on the `pip install -e .` step, so dependencies only ever come from the properly-constrained export. 3. scripts/xsslint_config.py's SKIP_DIRS didn't exclude .venv. Under the old pip-compile system, dependencies installed into the system Python outside the repo checkout, so this never mattered. Now that `uv sync` creates .venv/ inside the checkout, xsslint's directory walk (which defaults to scanning the whole cwd) swept up thousands of vendored third-party files, inflating violations from 64 (the accepted baseline) to 316. Verified all three: real pytest run (59 passed) plus full Django `manage.py check` for both LMS and CMS against the corrected test-requirements; a local simulation of the RTD build job sequence confirms Django==5.2.15/setuptools==81.0.0 (both constraint-compliant) and a successful Sphinx build. Audited every other `--only-group` usage introduced in this migration (assets.txt compat export, semgrep.yml) -- both are intentionally project-dependency-free, matching the original files' documented behavior, and their CI checks already passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

requirements/edx/{base,development}.txt were regenerated from the uv.lock that predated PR 1's social-auth-core<5.0.0 constraint, so they still referenced social-auth-core==5.0.2 -- caught by check-consistent-dependencies.yml's re-run of `make compile-requirements`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Gives requirements/edx-sandbox/ its own standalone pyproject.toml + uv.lock, independent of the main app's dependency graph (codejail intentionally runs untrusted code in a separate, isolated venv). [tool.edx_lint].uv_constraints holds only the subset of the root constraints relevant to this environment's deps (numpy, lxml, setuptools) -- uv/edx-lint have no cross-project constraint chaining equivalent to pip-compile's "-c ../constraints.txt", so root and sandbox constraints are now independently maintained (documented in requirements/edx-sandbox/README.rst). base.txt is regenerated as a `uv export` compatibility artifact (the README documents it as a supported, if unstable, direct pip-install target). releases/*.txt are untouched -- they're frozen historical snapshots, not part of any active compile loop; README now documents cutting future ones via `uv export` instead of pip-compile. Part of openedx/public-engineering#543 (3 of 5). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…on 4/5) Gives scripts/xblock, scripts/user_retirement, and scripts/structures_pruning each their own standalone pyproject.toml + uv.lock, mirroring the codejail sandbox pattern from PR 3. structures_pruning's local [tool.edx_lint].uv_constraints keeps the pymongo<4.4.1 pin it inherited via the old "-c ../../../requirements/constraints.txt" chain. These scripts are documented (in their own READMEs) to support git sparse-checkout usage -- cloning only e.g. scripts/user_retirement/ without the rest of edx-platform. A self-contained pyproject.toml is actually an improvement here over the old relative "-c ../../../requirements/constraints.txt" reference, which wouldn't even resolve in a sparse checkout that excludes the root requirements/ dir. Compatibility .txt exports are kept at their previously-documented paths (e.g. scripts/user_retirement/requirements/{base,testing}.txt) since each script's own README explicitly instructs `pip install -r` against those exact paths. Also fixes check-consistent-dependencies.yml's path filter and the two PR-creating workflows' add-paths, neither of which would have picked up changes to the new scripts/*/pyproject.toml or uv.lock files. Part of openedx/public-engineering#543 (4 of 5). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Deletes requirements/constraints.txt, common_constraints.txt, and pip-tools.{in,txt} -- these were kept alive through PR 2-4 because requirements/edx-sandbox and scripts/* still pip-compiled against them, but PR 4 was the last consumer, so they're now fully unused. Removes the correspondingly-vestigial Makefile machinery: the pre-requirements target, the COMMON_CONSTRAINTS_TXT curl-fetch-and-sed target, and the CUSTOM_COMPILE_COMMAND/COMPILE_OPTS variables that only existed to feed pip-compile invocations which no longer exist anywhere in this repo. Finalizes requirements/README.rst for the fully-migrated state and fixes a couple of remaining stale references (constraints.txt -> [tool.edx_lint].uv_constraints). This is the last of 5 PRs migrating openedx-platform from pip-compile to uv + PEP 621/735 pyproject.toml, tracked in openedx/public-engineering#543. Two follow-up items remain outside this repo's control: - openedx/repo-tools#725: find_python_dependencies needs pyproject.toml support before check_python_dependencies.yml can be re-enabled. - Tutor's Dockerfile installs from requirements/edx/{base,assets,development}.txt with plain pip; those are kept as `uv export` compatibility artifacts (see PR 2 / #38836) rather than deleted, so no action is required there, but tutor maintainers should be aware these are now generated files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

openedx-webhooks · 2026-07-21T13:33:05Z

Thanks for the pull request, @irfanuddinahmad!

This repository is currently maintained by @openedx/wg-maintenance-openedx-platform.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

CI's "Compile requirements" check re-runs `make compile-requirements` and fails if it produces any diff, to catch exactly this kind of inconsistency. The compat-export files in this PR were originally generated with uv 0.11.26; a newer uv (0.11.30, matching what astral-sh/setup-uv installs in CI) resolves grpcio/grpcio-status with an explicit `; platform_python_implementation != 'PyPy'` marker that 0.11.26 omitted. Regenerated with uv 0.11.30 to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The uv migration for scripts/user_retirement dropped the lxml pin that requirements/constraints.txt previously carried, letting the lockfile resolve to lxml 6.1.1. The pin exists to avoid a libxml2 version mismatch at runtime (#36695), and this script transitively depends on lxml via simple-salesforce -> zeep, so the same pin applies here as it does at the repo root.

…emaining setup-uv steps Faithfully translates .coveragerc into [tool.coverage.*] sections in pyproject.toml, matching the pattern used across the other uv-migration repos -- confirmed coverage.py picks up the new config identically (same source/omit/branch/concurrency/parallel/relative_files/paths values) once .coveragerc is removed. Also adds `enable-cache: true` to the 5 remaining astral-sh/setup-uv steps that weren't touched by the earlier migration commits (check-consistent-dependencies.yml, compile-python-requirements.yml, semgrep.yml, unit-tests.yml x2, upgrade-one-python-dependency.yml), for consistency with the other 7 workflows this migration already updated.

openedx/repo-tools#725 (find_python_dependencies needs to scan pyproject.toml/uv.lock) now has a proposed fix at openedx/repo-tools#735 -- linking it from the disabled-check comment so re-enabling this workflow is easy to track once it merges/releases.

salman2013 · 2026-07-22T05:59:44Z

@@ -1,5 +1,7 @@
 [tox]
-envlist = py{312} quality
+requires =


quality is in the envlist but there is no [testenv:quality] section.

salman2013 · 2026-07-22T06:07:29Z

          python-version: ${{ matrix.python-version }}

+      - name: Install uv
+        uses: astral-sh/setup-uv@v7


astral-sh/setup-uv@v7 uses a mutable tag. All other actions in this PR are SHA-pinned — pin this one too.

salman2013 · 2026-07-24T11:51:17Z

+            requirements
+            scripts/**/pyproject.toml
+            scripts/**/uv.lock
+            scripts/**/requirements*


Could you verify these paths? Previously, add-paths monitored the top-level requirements path. With this change, requirements* is only being added under scripts/. I just want to confirm this is the intended behavior.

salman2013 · 2026-07-24T11:52:27Z

-        id: cache-dependencies
-        uses: actions/cache@v6
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7


Can we pin this action to a commit SHA instead of @v7? We generally prefer immutable SHA-pinned GitHub Actions to improve supply chain security and reproducibility.

salman2013 · 2026-07-24T11:55:16Z

      - name: Install Required Python Dependencies
        run: |
          make base-requirements
+          echo "$PWD/.venv/bin" >> "$GITHUB_PATH"


Is adding .venv/bin to GITHUB_PATH required here? If subsequent steps are already using uv run, this may be unnecessary. Otherwise, it makes sense so that tools installed by uv sync are available on the PATH for later workflow steps.

salman2013 · 2026-07-24T11:55:38Z

-        id: cache-dependencies
-        uses: actions/cache@v6
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7


Can we pin this action to a commit SHA instead of @v7? We generally prefer immutable SHA-pinned GitHub Actions to improve supply chain security and reproducibility.

Irfan Ahmad and others added 8 commits July 21, 2026 18:03

irfanuddinahmad requested review from feanil and kdmccormick as code owners July 21, 2026 13:33

openedx-webhooks added open-source-contribution PR author is not from Axim or 2U core contributor PR author is a Core Contributor (who may or may not have write access to this repo). labels Jul 21, 2026

openedx-webhooks added this to Contributions Jul 21, 2026

github-project-automation Bot moved this to Needs Triage in Contributions Jul 21, 2026

Irfan Ahmad and others added 3 commits July 21, 2026 18:38

mphilbrick211 moved this from Needs Triage to Waiting on Author in Contributions Jul 21, 2026

irfanuddinahmad mentioned this pull request Jul 21, 2026

feat: teach find_python_dependencies to scan pyproject.toml and uv.lock openedx/repo-tools#735

Open

irfanuddinahmad changed the title ~~chore: consolidated pip-compile -> uv migration (all 5 stacked PRs, rebased onto master)~~ feat: consolidated pip-compile -> uv migration (all 5 stacked PRs, rebased onto master) Jul 21, 2026

blarghmatey mentioned this pull request Jul 23, 2026

fix: bridge pip-compile .txt and pyproject.toml/uv.lock for openedx Python builds mitodl/lehrer#128

Open

salman2013 reviewed Jul 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: consolidated pip-compile -> uv migration (all 5 stacked PRs, rebased onto master)#38915

feat: consolidated pip-compile -> uv migration (all 5 stacked PRs, rebased onto master)#38915
irfanuddinahmad wants to merge 12 commits into
masterfrom
irfanuddinahmad/uv-migration-consolidated

irfanuddinahmad commented Jul 21, 2026

Uh oh!

openedx-webhooks commented Jul 21, 2026

Uh oh!

salman2013 Jul 22, 2026

Uh oh!

salman2013 Jul 22, 2026

Uh oh!

salman2013 Jul 24, 2026

Uh oh!

salman2013 Jul 24, 2026

Uh oh!

salman2013 Jul 24, 2026

Uh oh!

salman2013 Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

irfanuddinahmad commented Jul 21, 2026

Summary

What changed

What's intentionally NOT done here (tracked externally)

Verification

Uh oh!

openedx-webhooks commented Jul 21, 2026

Uh oh!

salman2013 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

salman2013 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

salman2013 Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

salman2013 Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

salman2013 Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

salman2013 Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants