[FEATURE](pyspark) PySpark validation generated from the Pydantic schema

[Seth Fitzsimmons (sethfitz)]

Replacement PR description:

---

Summary

Adds a new runtime package (overture-schema-pyspark) plus a new output target in overture-schema-codegen that emits PySpark validation expressions and conformance tests from the same Pydantic models that define the schema. Ships an overture-validate CLI (to be merged with the overture-schema CLI later; that will be an opportunity to improve its ergonomics) for running the validation against Parquet on disk or in S3.

PySpark plugs in as a peer of the existing Markdown output target: same FeatureSpec extraction, same four-layer architecture (Discovery -> Extraction -> Output Layout -> Rendering), new pipeline module. See packages/overture-schema-codegen/docs/design.md for the full picture; the "PySpark Pipeline" section there covers the new stages in detail.

What's in the PR

packages/overture-schema-pyspark/ -- runtime. Public API in validate.py (validate_feature, explain_errors), schema comparison in schema_check.py, dataclasses in check.py, the overture-validate CLI in cli.py, and shared expression building blocks in expressions/{constraint_expressions,column_patterns,_schema_structs}.py. The per-feature expression modules under expressions/generated/overture/schema/<theme>/<feature>.py and per-feature conformance tests under tests/generated/overture/schema/<theme>/test_<feature>.py are emitted by codegen and confined to a generated/ boundary that make generate-pyspark wipes and recreates. _registry.py walks that tree at import time and exposes REGISTRY: dict[str, FeatureValidation] keyed by each module's ENTRY_POINT value (e.g. "overture.schema.transportation:Segment"), plus a parallel PARTITION_MAP derived from each module's PARTITIONS dict for partitioned features.

This has been tested to work with Spark versions 3.4.0 - 4.1.1 (and the lowest-direct CI check will verify that it continues to work with the lowest declared PySpark version, which is currently 3.4).

Adam Lastowka (@Rachmanin0xFF) the public API changed since the previous version you tested (in an attempt to simplify how it gets integrated with the CDP). If you point me to its current state, I can propose an update that uses the new API.

packages/overture-schema-codegen/src/overture/schema/codegen/pyspark/ -- new output target. Pipeline stages:

FeatureSpec (from extraction)
    |
constraint_dispatch.py    constraints -> ExpressionDescriptor / ModelConstraintDescriptor
    |
check_builder.py          FieldShape tree -> Check / ModelCheck IR
                          (check_ir.py; resolves array nesting, variant gating via
                          FieldPath = ScalarPath | ArrayPath and Guard sum types)
schema_builder.py         FieldShape tree -> SchemaField list (StructType source)
test_data/                FeatureSpec -> BASE_ROW_SPARSE / BASE_ROW_POPULATED, scaffold,
                          invalid_value
    |
renderer.py               Check IR -> per-feature expression module
test_renderer.py          Check IR -> per-feature conformance test module
    |
pipeline.py               orchestrates, returns GeneratedModule list

make generate-pyspark wipes both generated/ trees and recreates them; make check gates on regeneration being current.

What's covered

Every constraint Pydantic enforces today is dispatched to a PySpark expression, which allows Spark to push down predicates and take advantage of Catalyst to avoid unnecessary deserialization:

• Field constraints: Ge / Gt / Le / Lt / Interval, ArrayMinLen / ArrayMaxLen / ScalarMinLen / ScalarMaxLen (typed length constraints split by attachment layer), StrippedConstraint, PatternConstraint, UniqueItemsConstraint, GeometryTypeConstraint, JsonPointerConstraint.
• NewType overrides: LinearlyReferencedRange (length / bounds / order).
• Base-type overrides: HttpUrl (format + length), EmailStr, BBox (completeness, lat ordering, lat range).
• Model constraints: RequireAnyOfConstraint, RadioGroupConstraint, RequireIfConstraint, ForbidIfConstraint, MinFieldsSetConstraint. MinFieldsSetConstraint mirrors Pydantic's model_fields_set semantics: required fields are always set (the constructor enforces them) and count toward the threshold alongside any explicitly-set optional fields. NoExtraFieldsConstraint is intentionally skipped, as its behavior is implicit when using DataFrames.

Known semantic gaps / differences

Divergences from Pydantic:

• UniqueItemsConstraint uses Spark's array_distinct, which compares whole elements with structural equality on raw stored values. Pydantic compares normalized Python objects -- e.g., list[HttpUrl] is compared after URL normalization (such as ensuring that URLs like http://example.com contain a trailing /: http://example.com/). The PySpark check catches exact duplicates, not duplicates after normalization.
• require_any_of checks isNotNull as a proxy for Pydantic's model_fields_set. Parquet has no equivalent of "explicitly provided" because of the tabular nature of DataFrames; isNotNull is stricter (it rejects fields explicitly set to null).
• BBox's PySpark expressions include checking latitude ordering and range. Longitude is skipped due to undefined behavior around how we handle features that cross the antimeridian.

CLI

$ overture-validate --help
Usage: overture-validate [OPTIONS] FEATURE_TYPE PATH

  Validate Overture data at PATH and write annotated Parquet.

Options:
  -o, --output TEXT            Output path for validated Parquet.
  --head INTEGER               Error rows to display.  [default: 20]
  --conf TEXT                  Spark config key=value pairs.
  --count-only                 Report error count only; skip explain/unpivot.
  --skip-schema-check          Warn on schema mismatches instead of aborting.
  --skip-columns TEXT          Columns declared absent from data; skips their
                               checks.
  --ignore-extra-columns TEXT  Extra data columns to ignore in schema
                               comparison.
  --suppress TEXT              Suppress checks: FIELD (all checks) or
                               FIELD:CHECK (specific).
  --help                       Show this message and exit.

Output is one row per violation: feature ID, theme/type, failing field, check name, message, offending value.

Testing

overture-schema-codegen generates conformance tests alongside the PySpark expressions as a sanity-check by evaluating expected-good and expected-bad rows within a single DataFrame (setup/teardown overhead for DataFrame-per-test-case was much too high). Each feature gets two baseline rows -- BASE_ROW_SPARSE (required fields only) and BASE_ROW_POPULATED (every optional field filled in) -- and every scenario is exercised against both, so checks that depend on optional structure being present are covered as well as the minimal-row case.

make check generates tests and runs the full suite over everything, which use PySpark and increase test runtime (again; they run in just over a minute for me). testmon was introduced to improve the developer experience by skipping tests that weren't affected by recent edits.

Beyond the unit and conformance tests:

# Local Parquet
overture-validate segment samples/segment.parquet --count-only

# Real release prefix (expect bbox Float-vs-Double mismatch -> use --skip-schema-check)
# partitions (theme=places, type=place) will automatically be detected
# note that the s3a protocol must be used when AWS-provided Hadoop filesystem JARs are not available
overture-validate place s3a://overturemaps-us-west-2/release/<release>/ --skip-schema-check --head 50

Notes for review

• This PR is intentionally large because the generated tree is large. The interesting surface is smaller: pyspark/{constraint_dispatch,check_builder,check_ir,schema_builder,renderer,test_renderer,pipeline}.py plus the runtime in overture-schema-pyspark/src/overture/schema/pyspark/. Everything under generated/ is regenerable output -- review the codegen, but skim the output to understand the shape of what's produced.
• Bundles supporting changes (testmon for incremental test runs, VehicleSelectorBase extraction, Java 17 CI pin) rather than splitting them into separate PRs.
• This is a draft; expect force-pushes as cleanup continues, although the bulk of my review/refactoring is complete.

Closes #517.

[github-actions]

🗺️ Schema reference docs preview is live!

🌍 Preview	https://staging.overturemaps.org/schema/pr/518/schema/index.html
🕐 Updated	May 20, 2026 21:19 UTC
📝 Commit	4b55cc0
🔧 env SCHEMA_PREVIEW	true

Note

♻️ This preview updates automatically with each push to this PR.