ci: run fork-PR unit/lint/type checks offline from a prewarmed cache (#831) (#845)

vikrantpuppala · web-flow · commit a238dc9be470 · 2026-07-08T11:27:05.000Z
Fork pull_request runs never receive a GitHub OIDC token, so .github/actions/setup-jfrog (which mints a JFrog PyPI token from that OIDC token) dies with "ACTIONS_ID_TOKEN_REQUEST_TOKEN: unbound variable" and every required check goes red on every fork PR — regardless of the diff (#831, reported against the trivial fork PR #671). Public PyPI is unreachable from the protected runners, so a fork run can obtain no package source at all. Adopt the Databricks SOP (mirrors databricks/dbt-databricks): a trusted warmer workflow that DOES get OIDC pre-builds the in-project .venv for every matrix leg and saves it to the Actions cache; fork PRs restore that .venv and run the tools directly from .venv/bin, fully offline, never touching JFrog. Everything still flows through JFrog (the warmer populates the cache from it); the fork just holds no credentials, so there is no pull_request_target / secret-exposure risk. - .github/workflows/warm-deps-cache.yml: new, sole writer of the forkvenv-* cache. Triggers on push:main (dep files), a daily schedule (beats GitHub's 7-day cache eviction), and workflow_dispatch with an optional pr_number that warms a fork's changed lockfiles (fetches ONLY poetry.lock/pyproject.toml, no source). Warms the full py x depset x extras matrix, priming mypy stubs too. - .github/actions/restore-deps: new composite action; restores the newest forkvenv-* entry for the leg via a restore-keys prefix and reports cache-hit. - .github/workflows/code-quality-checks.yml: each job tries restore-deps first and runs offline from .venv/bin on a hit; on a miss (all same-repo PRs, or a fork whose lockfiles changed pre-warm) it falls back to the original setup-poetry JFrog flow, unchanged. Same-repo PR behavior is unaffected. Scope: unit + lint + type checks (the fork-reachable required checks). e2e / kernel-e2e / integration / DBR-LTS stay merge-queue-gated (they need live warehouse secrets). Closes #831 Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
diff --git a/.github/actions/restore-deps/action.yml b/.github/actions/restore-deps/action.yml
@@ -0,0 +1,53 @@
+name: Restore Prewarmed Dependencies
+description: |
+  Restores a fully-built in-project .venv from the "forkvenv-*" cache written by
+  warm-deps-cache.yml, so FORK PRs run offline without touching JFrog (#831).
+
+  Outputs cache-hit. When true, the caller must SKIP setup-poetry (the .venv is
+  already complete) and only install Poetry itself. When false — a same-repo PR,
+  or a fork whose lockfiles changed and haven't been warmed — the caller falls
+  back to setup-poetry (JFrog OIDC), which works in every trusted context and on
+  cache-miss for same-repo PRs.
+
+inputs:
+  python-version:
+    description: Python version this leg targets (must match the warmed leg)
+    required: true
+  dependency-version:
+    description: 'Dependency set: "default" or "min"'
+    required: false
+    default: "default"
+  extras:
+    description: 'Extras installed in the warmed venv: "" (base), "pyarrow", or "kernel"'
+    required: false
+    default: ""
+
+outputs:
+  cache-hit:
+    description: "true if a warmed .venv was restored (install can be skipped)"
+    value: ${{ steps.restore.outputs.cache-matched-key != '' }}
+
+runs:
+  using: composite
+  steps:
+    - name: Restore warmed .venv
+      id: restore
+      uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
+      with:
+        path: .venv
+        # The warmer appends a timestamp to the key (caches are immutable), so
+        # there is no fixed exact key to hit. restore-keys does a PREFIX match
+        # and returns the most-recently-created entry for this lockfile +
+        # matrix leg — exactly the freshest warmed venv.
+        key: forkvenv-${{ runner.os }}-${{ inputs.python-version }}-${{ inputs.dependency-version }}-${{ inputs.extras || 'base' }}-${{ hashFiles('**/poetry.lock') }}-never
+        restore-keys: |
+          forkvenv-${{ runner.os }}-${{ inputs.python-version }}-${{ inputs.dependency-version }}-${{ inputs.extras || 'base' }}-${{ hashFiles('**/poetry.lock') }}-
+
+    - name: Report restore outcome
+      shell: bash
+      run: |
+        if [ -n "${{ steps.restore.outputs.cache-matched-key }}" ]; then
+          echo "Restored warmed .venv (${{ steps.restore.outputs.cache-matched-key }}) — install will be skipped."
+        else
+          echo "No warmed .venv for this lockfile/leg — caller falls back to setup-poetry (JFrog)."
+        fi
diff --git a/.github/workflows/code-quality-checks.yml b/.github/workflows/code-quality-checks.yml
@@ -2,6 +2,27 @@ name: Code Quality Checks
 
 on: [pull_request]
 
+# How fork PRs are made to work (issue #831)
+# ------------------------------------------
+# Every job below installs deps via .github/actions/setup-poetry ->
+# setup-jfrog, which needs a GitHub OIDC token to authenticate to the JFrog
+# PyPI proxy. GitHub withholds that token from fork `pull_request` runs, and
+# public PyPI is unreachable from the protected runners — so on a fork PR the
+# JFrog path cannot run at all (this is what made every fork check red).
+#
+# Each job therefore takes a cache-first path:
+#   1. restore-deps restores a fully-built .venv prewarmed by
+#      warm-deps-cache.yml (a trusted workflow that DOES have OIDC).
+#   2. On a cache HIT (the normal fork path) we run the tools directly from
+#      .venv/bin — NOT `poetry run` — because even bootstrapping Poetry
+#      (`pip install poetry`) would hit JFrog. The restored venv already
+#      contains pytest/black/mypy and the connector, so no install is needed.
+#   3. On a cache MISS (every same-repo PR, or a fork whose lockfiles changed
+#      before a maintainer re-warmed) we fall back to the original
+#      setup-poetry (JFrog OIDC) flow, unchanged. Same-repo PRs are unaffected.
+#
+# Each step branches on the restore step's output, `steps.deps.outputs.cache-hit`.
+
 permissions:
   contents: read
   id-token: write
@@ -26,28 +47,49 @@ jobs:
     steps:
       - name: Check out repository
         uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+
+      # --- Fork path: restore a prewarmed .venv and run offline -------------
+      - name: Restore prewarmed dependencies
+        id: deps
+        uses: ./.github/actions/restore-deps
+        with:
+          python-version: ${{ matrix.python-version }}
+          dependency-version: ${{ matrix.dependency-version }}
+      - name: Set up Python (cache hit — needed for the restored venv interpreter)
+        if: steps.deps.outputs.cache-hit == 'true'
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Run tests (offline, from prewarmed venv)
+        if: steps.deps.outputs.cache-hit == 'true'
+        run: .venv/bin/python -m pytest tests/unit -m "not realkernel"
+
+      # --- Fallback path: same-repo PRs / cache miss use JFrog (unchanged) --
       - name: Setup Poetry
+        if: steps.deps.outputs.cache-hit != 'true'
         uses: ./.github/actions/setup-poetry
         with:
           python-version: ${{ matrix.python-version }}
           cache-suffix: "${{ matrix.dependency-version }}-"
       - name: Install Python tools for custom versions
-        if: matrix.dependency-version != 'default'
+        if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
         run: poetry run pip install toml packaging
       - name: Generate requirements file
-        if: matrix.dependency-version != 'default'
+        if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
         run: |
           poetry run python scripts/dependency_manager.py ${{ matrix.dependency-version }} --output requirements-${{ matrix.dependency-version }}.txt
           echo "Generated requirements for ${{ matrix.dependency-version }} versions:"
           cat requirements-${{ matrix.dependency-version }}.txt
       - name: Override with custom dependency versions
-        if: matrix.dependency-version != 'default'
+        if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
         run: poetry run pip install -r requirements-${{ matrix.dependency-version }}.txt
       - name: Show installed versions
+        if: steps.deps.outputs.cache-hit != 'true'
         run: |
           echo "=== Dependency Version: ${{ matrix.dependency-version }} ==="
           poetry run pip list
       - name: Run tests
+        if: steps.deps.outputs.cache-hit != 'true'
         run: poetry run python -m pytest tests/unit -m "not realkernel"
 
   run-unit-tests-with-arrow:
@@ -69,11 +111,33 @@ jobs:
     steps:
       - name: Check out repository
         uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      # libkrb5 is a runtime system lib (not in the venv), so install it on
+      # both the fork and fallback paths.
       - name: Install Kerberos system dependencies
         run: |
           sudo apt-get update
           sudo apt-get install -y libkrb5-dev
+
+      # --- Fork path: restore prewarmed pyarrow venv, run offline ------------
+      - name: Restore prewarmed dependencies
+        id: deps
+        uses: ./.github/actions/restore-deps
+        with:
+          python-version: ${{ matrix.python-version }}
+          dependency-version: ${{ matrix.dependency-version }}
+          extras: pyarrow
+      - name: Set up Python (cache hit — needed for the restored venv interpreter)
+        if: steps.deps.outputs.cache-hit == 'true'
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Run tests (offline, from prewarmed venv)
+        if: steps.deps.outputs.cache-hit == 'true'
+        run: .venv/bin/python -m pytest tests/unit -m "not realkernel"
+
+      # --- Fallback path: same-repo PRs / cache miss use JFrog (unchanged) ---
       - name: Setup Poetry
+        if: steps.deps.outputs.cache-hit != 'true'
         uses: ./.github/actions/setup-poetry
         with:
           python-version: ${{ matrix.python-version }}
@@ -84,22 +148,24 @@ jobs:
           install-args: "--extras pyarrow"
           cache-suffix: "pyarrow-${{ matrix.dependency-version }}-"
       - name: Install Python tools for custom versions
-        if: matrix.dependency-version != 'default'
+        if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
         run: poetry run pip install toml packaging
       - name: Generate requirements file with pyarrow
-        if: matrix.dependency-version != 'default'
+        if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
         run: |
           poetry run python scripts/dependency_manager.py ${{ matrix.dependency-version }} --output requirements-${{ matrix.dependency-version }}-arrow.txt
           echo "Generated requirements for ${{ matrix.dependency-version }} versions with PyArrow:"
           cat requirements-${{ matrix.dependency-version }}-arrow.txt
       - name: Override with custom dependency versions
-        if: matrix.dependency-version != 'default'
+        if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
         run: poetry run pip install -r requirements-${{ matrix.dependency-version }}-arrow.txt
       - name: Show installed versions
+        if: steps.deps.outputs.cache-hit != 'true'
         run: |
           echo "=== Dependency Version: ${{ matrix.dependency-version }} with PyArrow ==="
           poetry run pip list
       - name: Run tests
+        if: steps.deps.outputs.cache-hit != 'true'
         run: poetry run python -m pytest tests/unit -m "not realkernel"
 
   run-unit-tests-with-kernel:
@@ -113,6 +179,8 @@ jobs:
         python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
 
     name: "Unit Tests + Kernel (Python ${{ matrix.python-version }})"
+    # base restore-deps entries omit 3.9-kernel (no kernel wheel there); this
+    # matrix already starts at 3.10, so every leg has a warmed counterpart.
 
     steps:
       - name: Check out repository
@@ -121,7 +189,35 @@ jobs:
         run: |
           sudo apt-get update
           sudo apt-get install -y libkrb5-dev
+
+      # --- Fork path: restore prewarmed kernel venv, run offline ------------
+      # NOTE: the [kernel] extra pulls databricks-sql-kernel, the one PRIVATE
+      # (JFrog-only) package. Baking it into the warmed venv is precisely what
+      # lets a fork exercise the kernel routing without any JFrog credential.
+      - name: Restore prewarmed dependencies
+        id: deps
+        uses: ./.github/actions/restore-deps
+        with:
+          python-version: ${{ matrix.python-version }}
+          extras: kernel
+      - name: Set up Python (cache hit — needed for the restored venv interpreter)
+        if: steps.deps.outputs.cache-hit == 'true'
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Assert the real kernel wheel is installed (offline)
+        if: steps.deps.outputs.cache-hit == 'true'
+        run: .venv/bin/python -c "import databricks_sql_kernel as k; assert k.__file__, 'kernel wheel missing __file__ — not the real wheel'; print('real kernel wheel:', k.__file__)"
+      - name: Unit tests, realkernel deselected (offline)
+        if: steps.deps.outputs.cache-hit == 'true'
+        run: .venv/bin/python -m pytest tests/unit -m "not realkernel"
+      - name: Drive use_kernel=True through the REAL wheel — routing (offline)
+        if: steps.deps.outputs.cache-hit == 'true'
+        run: .venv/bin/python -m pytest tests/unit/test_session.py -m realkernel -v
+
+      # --- Fallback path: same-repo PRs / cache miss use JFrog (unchanged) --
       - name: Setup Poetry
+        if: steps.deps.outputs.cache-hit != 'true'
         uses: ./.github/actions/setup-poetry
         with:
           python-version: ${{ matrix.python-version }}
@@ -132,19 +228,23 @@ jobs:
           install-args: "--extras kernel"
           cache-suffix: "kernel-"
       - name: Show installed versions
+        if: steps.deps.outputs.cache-hit != 'true'
         run: |
           echo "=== with databricks-sql-kernel ==="
           poetry run pip list
       - name: Assert the real kernel wheel is installed (not a stub)
+        if: steps.deps.outputs.cache-hit != 'true'
         run: |
           poetry run python -c "import databricks_sql_kernel as k; assert k.__file__, 'kernel wheel missing __file__ — not the real wheel'; print('real kernel wheel:', k.__file__)"
       - name: Unit tests (kernel wheel present, realkernel deselected)
+        if: steps.deps.outputs.cache-hit != 'true'
         # The bulk of tests/unit fakes databricks_sql_kernel in
         # sys.modules, so the real-wheel routing test is deselected here
         # and run on its own below (a shared session would shadow the
         # real wheel — both real-wheel tests fail loudly if that happens).
         run: poetry run python -m pytest tests/unit -m "not realkernel"
       - name: Drive use_kernel=True through the REAL wheel (routing)
+        if: steps.deps.outputs.cache-hit != 'true'
         # Separate invocation, explicit file path: never collects the
         # fake-module test file, so sys.modules stays unpolluted. This is
         # the no-network proof that sql.connect(use_kernel=True) actually
@@ -162,11 +262,26 @@ jobs:
     steps:
       - name: Check out repository
         uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - name: Restore prewarmed dependencies
+        id: deps
+        uses: ./.github/actions/restore-deps
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Set up Python (cache hit — needed for the restored venv interpreter)
+        if: steps.deps.outputs.cache-hit == 'true'
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Black (offline, from prewarmed venv)
+        if: steps.deps.outputs.cache-hit == 'true'
+        run: .venv/bin/black --check src
       - name: Setup Poetry
+        if: steps.deps.outputs.cache-hit != 'true'
         uses: ./.github/actions/setup-poetry
         with:
           python-version: ${{ matrix.python-version }}
       - name: Black
+        if: steps.deps.outputs.cache-hit != 'true'
         run: poetry run black --check src
 
   check-types:
@@ -179,11 +294,30 @@ jobs:
     steps:
       - name: Check out repository
         uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - name: Restore prewarmed dependencies
+        id: deps
+        uses: ./.github/actions/restore-deps
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Set up Python (cache hit — needed for the restored venv interpreter)
+        if: steps.deps.outputs.cache-hit == 'true'
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Mypy (offline, from prewarmed venv)
+        if: steps.deps.outputs.cache-hit == 'true'
+        # --install-types would reach the index; the warmed venv already has
+        # the stub packages, so run mypy without it on the offline path.
+        run: |
+          mkdir -p .mypy_cache
+          .venv/bin/mypy --non-interactive src
       - name: Setup Poetry
+        if: steps.deps.outputs.cache-hit != 'true'
         uses: ./.github/actions/setup-poetry
         with:
           python-version: ${{ matrix.python-version }}
       - name: Mypy
+        if: steps.deps.outputs.cache-hit != 'true'
         run: |
           mkdir .mypy_cache
           poetry run mypy --install-types --non-interactive src
diff --git a/.github/workflows/warm-deps-cache.yml b/.github/workflows/warm-deps-cache.yml
