Skip to content

Make Vertex AI evaluation optional to avoid litellm bloat#69749

Open
Vamsi-klu wants to merge 6 commits into
apache:mainfrom
Vamsi-klu:fix/google-evaluation-optional-69323
Open

Make Vertex AI evaluation optional to avoid litellm bloat#69749
Vamsi-klu wants to merge 6 commits into
apache:mainfrom
Vamsi-klu:fix/google-evaluation-optional-69323

Conversation

@Vamsi-klu

@Vamsi-klu Vamsi-klu commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Problem

apache-airflow-providers-google pulled google-cloud-aiplatform[evaluation] into the base provider install. That extra brings in evaluation-only dependencies such as litellm, scikit-learn, huggingface-hub, and tokenizers for users who may only need unrelated Google services.

The only code path that needs vertexai.preview.evaluation is Vertex AI model evaluation through GenerativeModelHook.get_eval_task(), GenerativeModelHook.run_evaluation(), and RunEvaluationOperator.

What Changed

  • Changed the base dependency to google-cloud-aiplatform>=1.155.0.
  • Added an evaluation provider extra that installs google-cloud-aiplatform[evaluation]>=1.155.0.
  • Added an import guard that raises AirflowOptionalProviderFeatureException with an install instruction when evaluation APIs are used without the extra.
  • Added a provider changelog note for the breaking dependency split.
  • Updated generated provider dependency docs and uv.lock.
  • Added a follow-up regression test for the missing-extra path and documented the evaluation extra in the Vertex AI operator docs.

Why The Follow-Up Commit Exists

The feedback was correct: the original PR proved the opt-in path but did not directly test the failure mode when the extra is missing, did not document the extra next to RunEvaluationOperator, and left the lockfile inconsistent with the dependency split.

Impact

  • Base Google provider installs no longer pull the evaluation dependency stack.
  • Users of RunEvaluationOperator or direct Vertex AI evaluation hook methods must install apache-airflow-providers-google[evaluation].
  • Users who do not use evaluation APIs should see no runtime behavior change.

Fixes: #69323

Testing

  • UV_CACHE_DIR=/tmp/uv-cache-69749 uv run --python 3.11 ruff format providers/google/tests/unit/google/cloud/hooks/vertex_ai/test_generative_model_optional_evaluation.py
  • UV_CACHE_DIR=/tmp/uv-cache-69749 uv run --python 3.11 ruff check --fix providers/google/tests/unit/google/cloud/hooks/vertex_ai/test_generative_model_optional_evaluation.py
  • AIRFLOW_HOME=/tmp/airflow-home-69749 UV_CACHE_DIR=/tmp/uv-cache-69749 uv run --python 3.11 --project providers/google pytest providers/google/tests/unit/google/cloud/hooks/vertex_ai/test_generative_model_optional_evaluation.py --with-db-init -xvs
  • Result: 2 passed, 1 warning
  • AIRFLOW_HOME=/tmp/airflow-home-69749 UV_CACHE_DIR=/tmp/uv-cache-69749 uv run --python 3.11 --project providers/google --extra evaluation pytest providers/google/tests/unit/google/cloud/hooks/vertex_ai/test_generative_model.py providers/google/tests/unit/google/cloud/operators/vertex_ai/test_generative_model.py -xvs
  • Result: 2 passed, 1 warning
  • Commit hooks passed for the follow-up commit.

Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines


Drafted-by: Codex (GPT-5); reviewed by @Vamsi-klu before posting

@boring-cyborg boring-cyborg Bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Jul 11, 2026
@Vamsi-klu

Copy link
Copy Markdown
Contributor Author

Reviewers: @shahar1 (google provider CODEOWNER). Breaking change: evaluation extra now optional to avoid litellm bloat. Changelog entry added, provider docs regenerated via update-providers-dependencies.

Fixes #69323


Drafted-by: Muse Spark 1.1; reviewed by @Vamsi-klu before posting

Vamsi-klu and others added 3 commits July 12, 2026 19:29
Base dependency on google-cloud-aiplatform[evaluation] forced litellm,
scikit-learn, tokenizers for all users. Move evaluation extra behind
optional provider extra 'evaluation'. Add lazy import guard with
AirflowOptionalProviderFeatureException in GenerativeModelHook.

Users needing RunEvaluationOperator should install
apache-airflow-providers-google[evaluation].

Fixes: apache#69323
Use TYPE_CHECKING guard and Any fallback to satisfy mypy when
evaluation extra not installed. Runtime guard still raises
AirflowOptionalProviderFeatureException before using None types.

Fixes mypy failure in CI for apache#69323
The provider now keeps evaluation dependencies out of the base install, so the missing-extra path needs regression coverage and operator docs that show users how to opt back in.
@Vamsi-klu Vamsi-klu force-pushed the fix/google-evaluation-optional-69323 branch from 8d5aab9 to b54b2d6 Compare July 12, 2026 19:56
@Vamsi-klu Vamsi-klu marked this pull request as ready for review July 13, 2026 03:36
@Vamsi-klu Vamsi-klu requested a review from shahar1 as a code owner July 13, 2026 03:36
@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

uv.lock on main just moved via #69924 ("Support excluding files from common.ai Agent Skills discovery"), commit 663a792 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Google provider base-depends on google-cloud-aiplatform[evaluation], force-installing litellm/scikit-learn for every user

1 participant