Skip to content

ci: run fork-PR unit/lint/type checks offline from a prewarmed cache (#831) #845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vikrantpuppala merged 1 commit into from
Jul 8, 2026
+391 −6
Merged

ci: run fork-PR unit/lint/type checks offline from a prewarmed cache (#831) #845

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .github/actions/restore-deps/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: Restore Prewarmed Dependencies
description: |
Restores a fully-built in-project .venv from the "forkvenv-*" cache written by
warm-deps-cache.yml, so FORK PRs run offline without touching JFrog (#831).

Outputs cache-hit. When true, the caller must SKIP setup-poetry (the .venv is
already complete) and only install Poetry itself. When false — a same-repo PR,
or a fork whose lockfiles changed and haven't been warmed — the caller falls
back to setup-poetry (JFrog OIDC), which works in every trusted context and on
cache-miss for same-repo PRs.

inputs:
python-version:
description: Python version this leg targets (must match the warmed leg)
required: true
dependency-version:
description: 'Dependency set: "default" or "min"'
required: false
default: "default"
extras:
description: 'Extras installed in the warmed venv: "" (base), "pyarrow", or "kernel"'
required: false
default: ""

outputs:
cache-hit:
description: "true if a warmed .venv was restored (install can be skipped)"
value: ${{ steps.restore.outputs.cache-matched-key != '' }}

runs:
using: composite
steps:
- name: Restore warmed .venv
id: restore
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
with:
path: .venv
# The warmer appends a timestamp to the key (caches are immutable), so
# there is no fixed exact key to hit. restore-keys does a PREFIX match
# and returns the most-recently-created entry for this lockfile +
# matrix leg — exactly the freshest warmed venv.
key: forkvenv-${{ runner.os }}-${{ inputs.python-version }}-${{ inputs.dependency-version }}-${{ inputs.extras || 'base' }}-${{ hashFiles('**/poetry.lock') }}-never
restore-keys: |
forkvenv-${{ runner.os }}-${{ inputs.python-version }}-${{ inputs.dependency-version }}-${{ inputs.extras || 'base' }}-${{ hashFiles('**/poetry.lock') }}-

- name: Report restore outcome
shell: bash
run: |
if [ -n "${{ steps.restore.outputs.cache-matched-key }}" ]; then
echo "Restored warmed .venv (${{ steps.restore.outputs.cache-matched-key }}) — install will be skipped."
else
echo "No warmed .venv for this lockfile/leg — caller falls back to setup-poetry (JFrog)."
fi
146 changes: 140 additions & 6 deletions .github/workflows/code-quality-checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,27 @@ name: Code Quality Checks

on: [pull_request]

# How fork PRs are made to work (issue #831)
# ------------------------------------------
# Every job below installs deps via .github/actions/setup-poetry ->
# setup-jfrog, which needs a GitHub OIDC token to authenticate to the JFrog
# PyPI proxy. GitHub withholds that token from fork `pull_request` runs, and
# public PyPI is unreachable from the protected runners — so on a fork PR the
# JFrog path cannot run at all (this is what made every fork check red).
#
# Each job therefore takes a cache-first path:
# 1. restore-deps restores a fully-built .venv prewarmed by
# warm-deps-cache.yml (a trusted workflow that DOES have OIDC).
# 2. On a cache HIT (the normal fork path) we run the tools directly from
# .venv/bin — NOT `poetry run` — because even bootstrapping Poetry
# (`pip install poetry`) would hit JFrog. The restored venv already
# contains pytest/black/mypy and the connector, so no install is needed.
# 3. On a cache MISS (every same-repo PR, or a fork whose lockfiles changed
# before a maintainer re-warmed) we fall back to the original
# setup-poetry (JFrog OIDC) flow, unchanged. Same-repo PRs are unaffected.
#
# Each step branches on the restore step's output, `steps.deps.outputs.cache-hit`.

permissions:
contents: read
id-token: write
Expand All @@ -26,28 +47,49 @@ jobs:
steps:
- name: Check out repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4

# --- Fork path: restore a prewarmed .venv and run offline -------------
- name: Restore prewarmed dependencies
id: deps
uses: ./.github/actions/restore-deps
with:
python-version: ${{ matrix.python-version }}
dependency-version: ${{ matrix.dependency-version }}
- name: Set up Python (cache hit — needed for the restored venv interpreter)
if: steps.deps.outputs.cache-hit == 'true'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: ${{ matrix.python-version }}
- name: Run tests (offline, from prewarmed venv)
if: steps.deps.outputs.cache-hit == 'true'
run: .venv/bin/python -m pytest tests/unit -m "not realkernel"

# --- Fallback path: same-repo PRs / cache miss use JFrog (unchanged) --
- name: Setup Poetry
if: steps.deps.outputs.cache-hit != 'true'
uses: ./.github/actions/setup-poetry
with:
python-version: ${{ matrix.python-version }}
cache-suffix: "${{ matrix.dependency-version }}-"
- name: Install Python tools for custom versions
if: matrix.dependency-version != 'default'
if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
run: poetry run pip install toml packaging
- name: Generate requirements file
if: matrix.dependency-version != 'default'
if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
run: |
poetry run python scripts/dependency_manager.py ${{ matrix.dependency-version }} --output requirements-${{ matrix.dependency-version }}.txt
echo "Generated requirements for ${{ matrix.dependency-version }} versions:"
cat requirements-${{ matrix.dependency-version }}.txt
- name: Override with custom dependency versions
if: matrix.dependency-version != 'default'
if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
run: poetry run pip install -r requirements-${{ matrix.dependency-version }}.txt
- name: Show installed versions
if: steps.deps.outputs.cache-hit != 'true'
run: |
echo "=== Dependency Version: ${{ matrix.dependency-version }} ==="
poetry run pip list
- name: Run tests
if: steps.deps.outputs.cache-hit != 'true'
run: poetry run python -m pytest tests/unit -m "not realkernel"

run-unit-tests-with-arrow:
Expand All @@ -69,11 +111,33 @@ jobs:
steps:
- name: Check out repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
# libkrb5 is a runtime system lib (not in the venv), so install it on
# both the fork and fallback paths.
- name: Install Kerberos system dependencies
run: |
sudo apt-get update
sudo apt-get install -y libkrb5-dev

# --- Fork path: restore prewarmed pyarrow venv, run offline ------------
- name: Restore prewarmed dependencies
id: deps
uses: ./.github/actions/restore-deps
with:
python-version: ${{ matrix.python-version }}
dependency-version: ${{ matrix.dependency-version }}
extras: pyarrow
- name: Set up Python (cache hit — needed for the restored venv interpreter)
if: steps.deps.outputs.cache-hit == 'true'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: ${{ matrix.python-version }}
- name: Run tests (offline, from prewarmed venv)
if: steps.deps.outputs.cache-hit == 'true'
run: .venv/bin/python -m pytest tests/unit -m "not realkernel"

# --- Fallback path: same-repo PRs / cache miss use JFrog (unchanged) ---
- name: Setup Poetry
if: steps.deps.outputs.cache-hit != 'true'
uses: ./.github/actions/setup-poetry
with:
python-version: ${{ matrix.python-version }}
Expand All @@ -84,22 +148,24 @@ jobs:
install-args: "--extras pyarrow"
cache-suffix: "pyarrow-${{ matrix.dependency-version }}-"
- name: Install Python tools for custom versions
if: matrix.dependency-version != 'default'
if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
run: poetry run pip install toml packaging
- name: Generate requirements file with pyarrow
if: matrix.dependency-version != 'default'
if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
run: |
poetry run python scripts/dependency_manager.py ${{ matrix.dependency-version }} --output requirements-${{ matrix.dependency-version }}-arrow.txt
echo "Generated requirements for ${{ matrix.dependency-version }} versions with PyArrow:"
cat requirements-${{ matrix.dependency-version }}-arrow.txt
- name: Override with custom dependency versions
if: matrix.dependency-version != 'default'
if: steps.deps.outputs.cache-hit != 'true' && matrix.dependency-version != 'default'
run: poetry run pip install -r requirements-${{ matrix.dependency-version }}-arrow.txt
- name: Show installed versions
if: steps.deps.outputs.cache-hit != 'true'
run: |
echo "=== Dependency Version: ${{ matrix.dependency-version }} with PyArrow ==="
poetry run pip list
- name: Run tests
if: steps.deps.outputs.cache-hit != 'true'
run: poetry run python -m pytest tests/unit -m "not realkernel"

run-unit-tests-with-kernel:
Expand All @@ -113,6 +179,8 @@ jobs:
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]

name: "Unit Tests + Kernel (Python ${{ matrix.python-version }})"
# base restore-deps entries omit 3.9-kernel (no kernel wheel there); this
# matrix already starts at 3.10, so every leg has a warmed counterpart.

steps:
- name: Check out repository
Expand All @@ -121,7 +189,35 @@ jobs:
run: |
sudo apt-get update
sudo apt-get install -y libkrb5-dev

# --- Fork path: restore prewarmed kernel venv, run offline ------------
# NOTE: the [kernel] extra pulls databricks-sql-kernel, the one PRIVATE
# (JFrog-only) package. Baking it into the warmed venv is precisely what
# lets a fork exercise the kernel routing without any JFrog credential.
- name: Restore prewarmed dependencies
id: deps
uses: ./.github/actions/restore-deps
with:
python-version: ${{ matrix.python-version }}
extras: kernel
- name: Set up Python (cache hit — needed for the restored venv interpreter)
if: steps.deps.outputs.cache-hit == 'true'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: ${{ matrix.python-version }}
- name: Assert the real kernel wheel is installed (offline)
if: steps.deps.outputs.cache-hit == 'true'
run: .venv/bin/python -c "import databricks_sql_kernel as k; assert k.__file__, 'kernel wheel missing __file__ — not the real wheel'; print('real kernel wheel:', k.__file__)"
- name: Unit tests, realkernel deselected (offline)
if: steps.deps.outputs.cache-hit == 'true'
run: .venv/bin/python -m pytest tests/unit -m "not realkernel"
- name: Drive use_kernel=True through the REAL wheel — routing (offline)
if: steps.deps.outputs.cache-hit == 'true'
run: .venv/bin/python -m pytest tests/unit/test_session.py -m realkernel -v

# --- Fallback path: same-repo PRs / cache miss use JFrog (unchanged) --
- name: Setup Poetry
if: steps.deps.outputs.cache-hit != 'true'
uses: ./.github/actions/setup-poetry
with:
python-version: ${{ matrix.python-version }}
Expand All @@ -132,19 +228,23 @@ jobs:
install-args: "--extras kernel"
cache-suffix: "kernel-"
- name: Show installed versions
if: steps.deps.outputs.cache-hit != 'true'
run: |
echo "=== with databricks-sql-kernel ==="
poetry run pip list
- name: Assert the real kernel wheel is installed (not a stub)
if: steps.deps.outputs.cache-hit != 'true'
run: |
poetry run python -c "import databricks_sql_kernel as k; assert k.__file__, 'kernel wheel missing __file__ — not the real wheel'; print('real kernel wheel:', k.__file__)"
- name: Unit tests (kernel wheel present, realkernel deselected)
if: steps.deps.outputs.cache-hit != 'true'
# The bulk of tests/unit fakes databricks_sql_kernel in
# sys.modules, so the real-wheel routing test is deselected here
# and run on its own below (a shared session would shadow the
# real wheel — both real-wheel tests fail loudly if that happens).
run: poetry run python -m pytest tests/unit -m "not realkernel"
- name: Drive use_kernel=True through the REAL wheel (routing)
if: steps.deps.outputs.cache-hit != 'true'
# Separate invocation, explicit file path: never collects the
# fake-module test file, so sys.modules stays unpolluted. This is
# the no-network proof that sql.connect(use_kernel=True) actually
Expand All @@ -162,11 +262,26 @@ jobs:
steps:
- name: Check out repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Restore prewarmed dependencies
id: deps
uses: ./.github/actions/restore-deps
with:
python-version: ${{ matrix.python-version }}
- name: Set up Python (cache hit — needed for the restored venv interpreter)
if: steps.deps.outputs.cache-hit == 'true'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: ${{ matrix.python-version }}
- name: Black (offline, from prewarmed venv)
if: steps.deps.outputs.cache-hit == 'true'
run: .venv/bin/black --check src
- name: Setup Poetry
if: steps.deps.outputs.cache-hit != 'true'
uses: ./.github/actions/setup-poetry
with:
python-version: ${{ matrix.python-version }}
- name: Black
if: steps.deps.outputs.cache-hit != 'true'
run: poetry run black --check src

check-types:
Expand All @@ -179,11 +294,30 @@ jobs:
steps:
- name: Check out repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Restore prewarmed dependencies
id: deps
uses: ./.github/actions/restore-deps
with:
python-version: ${{ matrix.python-version }}
- name: Set up Python (cache hit — needed for the restored venv interpreter)
if: steps.deps.outputs.cache-hit == 'true'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
with:
python-version: ${{ matrix.python-version }}
- name: Mypy (offline, from prewarmed venv)
if: steps.deps.outputs.cache-hit == 'true'
# --install-types would reach the index; the warmed venv already has
# the stub packages, so run mypy without it on the offline path.
run: |
mkdir -p .mypy_cache
.venv/bin/mypy --non-interactive src
- name: Setup Poetry
if: steps.deps.outputs.cache-hit != 'true'
uses: ./.github/actions/setup-poetry
with:
python-version: ${{ matrix.python-version }}
- name: Mypy
if: steps.deps.outputs.cache-hit != 'true'
run: |
mkdir .mypy_cache
poetry run mypy --install-types --non-interactive src
Loading
Loading