[ENHANCEMENT] May schema release v1.17.0#528
Closed
Stephen Epps (stepps00) wants to merge 114 commits into
Closed
Conversation
Update multiple packages' pyproject.toml files to add a maintainers list, common keywords, and a [project.urls] section (Homepage, Source, Issues). Affects core, system, cli, annex, and all theme packages under packages/overture-schema-*. This centralizes package metadata to improve discoverability, attribution, and tooling integration.
pytest-subtests merged into pytest core as of pytest 9. Update test imports from pytest_subtests.SubTests to _pytest.subtests.Subtests. uv.lock was fully re-resolved, upgrading all transitive dependencies to current versions. Notable: pytest 8.4→9.0, mypy 1.18→1.19, pdoc 15→16, ruff 0.14.0→0.14.14.
* Widen `AdminLevel` from `uint8` to `int32` to match divisions data currently being generated. * Fix Pydantic rebinding self in model constraint validator
Geometry constraint tests: replace full powerset (127 subsets of 7 geometry types) with singletons + pairs + full set (29 subsets). GeometryTypeConstraint is a set-membership check — combinatorial testing beyond pairs exercises no additional code paths. Makefile: run pytest, doctest, ruff, and mypy concurrently via make -j after a single upfront uv-sync. Added -only variants (no uv-sync dep) for the parallel invocation; standalone targets preserved with their original uv-sync dependencies.
* Treat None as absent in model constraint validation
require_any_of, require_if, and forbid_if now treat None as "not
present" for constraint purposes. Fields must be both in
model_fields_set and non-null to satisfy (or violate) a constraint.
- require_any_of: None no longer satisfies "at least one must be set"
- require_if: None no longer satisfies "must be set when condition holds"
- forbid_if: None no longer violates "must not be set when condition holds"
JSON Schema generation updated to emit {"not": {"type": "null"}}
property constraints alongside "required" assertions, via a shared
required_non_null helper in _json_schema.py. Shared predicate
_field_has_non_null_value extracted onto OptionalFieldGroupConstraint
so the check lives in one place. Also fixes a stray backtick in the
require_if error message.
* Use Python None terminology and clarify constraint docs
Use Python terminology (None) instead of null in docstrings, identifiers,
and error messages. Rewrite error messages to describe what's checked
("set to a value other than None") rather than using jargon.
Eliminates duplicated validate() and __get_pydantic_json_schema__() across CountryCodeAlpha2, HexColor, LanguageTag, NoWhitespace, SnakeCase, PhoneNumber, RegionCode, and WikidataId constraints. Each is now a thin __init__-only wrapper calling super().__init__(). PatternConstraint gains optional keyword-only description, min_length, max_length parameters for JSON Schema annotations. StringConstraint gains _raise_validation_error() to deduplicate error construction across PatternConstraint, JsonPointerConstraint, and StrippedConstraint.
The previous pattern ^(\S.*)?\S$ required at least one non-whitespace
character, rejecting empty string. The validator itself accepts empty
string ("" == "".strip()), so the JSON schema was more restrictive
than the Python validation.
New pattern ^(\S(.*\S)?)?$ matches empty string (outer group optional),
single non-whitespace chars, and strings bookended by non-whitespace.
Updated all JSON schema baselines and inline expectations.
Replace `len(errors()) > 0` with error message assertions in hex color and no-whitespace invalid tests. The weak assertions only checked that validation failed, not that the correct error was raised.
Replace 16 individual valid/invalid test methods with two parametrized tests driven by PATTERN_CONSTRAINT_CASES. Covers all 8 PatternConstraint subclasses: LanguageTag, CountryCodeAlpha2, RegionCode, WikidataId, PhoneNumber, HexColor, NoWhitespace, SnakeCase. Moved SnakeCaseConstraint tests from TestErrorHandling (where they were misplaced) into the parametrized data. Non-PatternConstraint tests remain as standalone methods: base PatternConstraint (custom pattern), StrippedConstraint, and JsonPointerConstraint (empty-string special case).
The dict branch of the Pydantic validator in BBox.__get_pydantic_core_schema__ constructed a BBox from the dict but didn't return it, silently producing None. Add the missing return statement. The test parametrize case for dict input covers this path.
Bare triple-quoted strings after NewType assignments are expression statements that Python never attaches to the NewType object, leaving __doc__ as None. Convert each to an explicit __doc__ assignment so codegen and introspection tools can read them at runtime. Same pattern DocumentedEnum uses for enum member docs.
- Add -q, --tb=short to `make test` for compact output - Set verbosity_subtests=0 to suppress per-subtest progress characters (the u/,/- markers from pytest's built-in subtests support) Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
OvertureFeature validator error message had two continuation
lines missing the f-prefix, so {self.__class__.__name__} was
rendered literally. Also add missing space before "and".
Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Also fix "supserset" typo in docstring. Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Replace hardcoded discriminator_fields tuple ("type", "theme",
"subtype") in _process_union_member with the discriminator field
name extracted from the union's Annotated metadata.
introspect_union already extracted the discriminator field name
but didn't pass it through to member processing. Now it does,
so unions using any field name as discriminator work correctly.
For nested unions, parent discriminator values are extracted from
nested leaf models to preserve structural tuple classification.
Feature.field_discriminator now attaches _field_name to the
callable, and _extract_discriminator_name reads it. This handles
the Discriminator-wrapping-a-callable case that str(disc) got
wrong silently.
Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Make _extract_literal_value return str directly instead of object, eliminating implicit str() conversions at call sites. Add comment explaining nested union re-indexing under the parent discriminator. Remove redundant test covered by TestDiscriminatorDiscovery and debugging print() calls from TestStructuralTuples. Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
The field holds the entry point value in "module:Class" format, not a class name. The old name required callers to know this (codegen's cli.py had a comment explaining it, and assigned to a local `entry_point` variable to compensate). Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Empty package with build config, namespace packages, and py.typed marker. Declares click, jinja2, tomli, and overture-schema-core/system as dependencies. Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Type analyzer (analyze_type) handles all type unwrapping in a single iterative function: NewType → Annotated → Union → list → terminal classification. Constraints accumulate from Annotated metadata with source tracking via ConstraintSource. Data structures: TypeInfo (type representation), FieldSpec (model field), ModelSpec (model), EnumSpec, NewTypeSpec, PrimitiveSpec. Type registry maps type names to per-target string representations via TypeMapping. is_semantic_newtype() distinguishes meaningful NewTypes from pass-through aliases. Utilities: case_conversion (snake_case), docstring (cleaning and custom-docstring detection). Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Domain-specific extractors that consume analyze_type() and produce specs: - model_extraction: extract_model() for Pydantic models with MRO-aware field ordering, alias resolution, and recursive sub-model expansion via expand_model_tree() - enum_extraction: extract_enum() for DocumentedEnum classes - newtype_extraction: extract_newtype() for semantic NewTypes - primitive_extraction: extract_primitives() for numeric types with range and precision introspection - union_extraction: extract_union() with field merging across discriminated union variants Shared test fixtures in codegen_test_support.py. Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Generate prose from extracted constraint data: - field_constraint_description: describe field-level constraints (ranges, patterns, unique items, hex colors) as human-readable notes with NewType source attribution - model_constraint_description: describe model-level constraints (@require_any_of, @radio_group, @min_fields_set, @require_if, @forbid_if) as prose, with consolidation of same-field conditional constraints Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Determine what artifacts to generate and where they go:
- module_layout: compute output directories for entry points,
map Python module paths to filesystem output paths via
compute_output_dir
- path_assignment: build_placement_registry maps types to
output file paths. Feature models get {theme}/{slug}/,
shared types get types/{subsystem}/, theme-local types
nest under their feature or sit flat at theme level
- type_collection: discover supplementary types (enums,
NewTypes, sub-models) by walking expanded feature trees
- link_computation: relative_link() computes cross-page
links, LinkContext holds page path + registry for
resolving links during rendering
Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Embed JSON example features in [tool.overture-schema.examples] sections. Each example is a complete GeoJSON Feature matching the theme's Pydantic model, used by the codegen example_loader to render example tables in documentation. Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Jinja2 templates and rendering logic for documentation pages: - markdown_renderer: orchestrates page rendering for features, enums, NewTypes, primitives, and geometry. Recursively expands MODEL-kind fields inline with dot-notation. - markdown_type_format: type string formatting with link-aware rendering via LinkContext - example_loader: loads examples from theme pyproject.toml, validates against Pydantic models, flattens to dot-notation - reverse_references: computes "Used By" cross-references between types and the features that reference them Templates: feature, enum, newtype, primitives, geometry pages. Golden-file snapshot tests verify rendered output stability. Adds renderer-specific fixtures to conftest.py (cli_runner, primitives_markdown, geometry_markdown). Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Click-based CLI entry point (overture-codegen generate) that wires discovery → extraction → output layout → rendering: - Discovers models via discover_models() entry points - Filters themes, extracts specs, builds placement registry - Renders markdown pages with field tables, examples, cross- references, and sidebar metadata - Supports --theme filtering and --output-dir targeting Integration tests verify extraction against real Overture models (Building, Division, Segment, etc.) to catch schema drift. CLI tests verify end-to-end generation, output structure, and link integrity. Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Design doc covers the four-layer architecture, analyze_type(), domain-specific extractors, and extension points for new output targets. Walkthrough traces Segment through the full pipeline module-by-module in dependency order, with FeatureVersion as a secondary example for constraint provenance in the type analyzer. README describes the problem (Pydantic flattens domain vocabulary), the "unwrap once, render many" approach, CLI usage, architecture overview, and programmatic API. Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
TypeInfo.literal_value discarded multi-value Literals entirely (Literal["a", "b"] got None). Renamed to literal_values as a tuple of all args so consumers decide presentation. single_literal_value() preserves its contract: returns the value for single-arg Literals, None otherwise. Callers (example_loader, union_extraction) are unchanged. Multi-value Literals render as pipe-separated quoted values in markdown tables: `"a"` \| `"b"`. Signed-off-by: Seth Fitzsimmons <sethfitz@amazon.com>
Signed-off-by: John McCall <john@overturemaps.org>
Signed-off-by: John McCall <john@overturemaps.org>
Add id-token: write permissions to workflows (.github/workflows/publish-python-packages.yaml and reusable-check-python-package-versions.yaml) so reusable workflows can use OIDC for AWS CodeArtifact authentication. Also add contents: read to the publish job. Update schema-pr-preview.yml to remove the PREVIEW_PATH env var and inline the /schema/pr/${{ github.event.number }} path for S3 sync and CloudFront invalidation to ensure the preview is uploaded and invalidated at the correct location.
Signed-off-by: John McCall <john@overturemaps.org>
Signed-off-by: John McCall <john@overturemaps.org>
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 7.0.0 to 7.0.1. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@bbbca2d...043fb46) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: John McCall <john@overturemaps.org>
Bumps [marocchino/sticky-pull-request-comment](https://github.com/marocchino/sticky-pull-request-comment) from 3.0.3 to 3.0.4. - [Release notes](https://github.com/marocchino/sticky-pull-request-comment/releases) - [Commits](marocchino/sticky-pull-request-comment@d4d6b09...0ea0beb) --- updated-dependencies: - dependency-name: marocchino/sticky-pull-request-comment dependency-version: 3.0.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: John McCall <john@overturemaps.org>
Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 8.0.0 to 8.1.0. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@cec2083...0880764) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: 8.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps the actions group with 2 updates: [actions/github-script](https://github.com/actions/github-script) and [actions/setup-node](https://github.com/actions/setup-node). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@ed59741...3a2844b) Updates `actions/setup-node` from 6.3.0 to 6.4.0 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@53b8394...48b55a0) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-node dependency-version: 6.4.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com>
The check-python-code workflow now runs across the supported Python minor versions (3.10-3.14), and adds a lowest-direct cell that re- resolves every direct dependency to its declared floor before running make check. Failures in that cell mean a declared minimum is wrong -- either the code uses a feature newer than the floor admits, or the pin needs to be raised. Setting UV_RESOLUTION=lowest-direct causes the existing make uv-sync to re-resolve from the lockfile rather than follow it, so no separate lock or constraints file is needed. The declared floors lie about the actual minimums in several places. Bump them to what the code requires today: - pydantic >= 2.12: optionality.py imports pydantic.experimental.missing_sentinel, which only exists in 2.12+. feature.py imports ModelWrapValidatorHandler at the top level (re-exported in 2.10.4) and Tag (added in 2.5). - ruff >= 0.13: 0.12.x still fires the deprecated UP038 rule under select = ["UP"], which the workspace config selects. - deepdiff >= 8.6: first version shipping a py.typed marker, without which mypy errors on import-untyped. Per-package dev deps that were unpinned (ruff, mypy, pytest in overture-schema-cli and overture-schema-system; pyyaml, deepdiff in overture-schema) now match the workspace dev-group floors. uv warns about unpinned direct deps under lowest-direct resolution; pinning silences the warnings and makes the floor explicit. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Signed-off-by: jeffdefacto <jeffdefacto@gmail.com>
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
May: dev commits to staging
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
… staging # Conflicts: # .github/workflows/publish-python-packages.yaml # .github/workflows/reusable-check-python-package-versions.yaml # .github/workflows/schema-pr-preview-cleanup.yml # .github/workflows/schema-pr-preview.yml # Makefile # PYDANTIC_GUIDE.md # README.pydantic.md # packages/overture-schema-addresses-theme/pyproject.toml # packages/overture-schema-annex/pyproject.toml # packages/overture-schema-base-theme/pyproject.toml # packages/overture-schema-buildings-theme/pyproject.toml # packages/overture-schema-cli/src/overture/schema/cli/commands.py # packages/overture-schema-cli/tests/test_resolve_types.py # packages/overture-schema-codegen/README.md # packages/overture-schema-codegen/docs/design.md # packages/overture-schema-codegen/docs/walkthrough.md # packages/overture-schema-codegen/pyproject.toml # packages/overture-schema-codegen/src/overture/schema/codegen/cli.py # packages/overture-schema-codegen/src/overture/schema/codegen/extraction/field_constraints.py # packages/overture-schema-codegen/tests/codegen_test_support.py # packages/overture-schema-codegen/tests/conftest.py # packages/overture-schema-codegen/tests/golden/markdown/venue.md # packages/overture-schema-codegen/tests/test_cli.py # packages/overture-schema-codegen/tests/test_constraint_description.py # packages/overture-schema-codegen/tests/test_integration_real_models.py # packages/overture-schema-codegen/tests/test_markdown_renderer.py # packages/overture-schema-codegen/tests/test_module_layout.py # packages/overture-schema-codegen/tests/test_type_placement.py # packages/overture-schema-core/README.md # packages/overture-schema-core/pyproject.toml # packages/overture-schema-core/src/overture/schema/core/__about__.py # packages/overture-schema-core/src/overture/schema/core/discovery.py # packages/overture-schema-divisions-theme/pyproject.toml # packages/overture-schema-divisions-theme/src/overture/schema/divisions/division.py # packages/overture-schema-divisions-theme/src/overture/schema/divisions/division_area.py # packages/overture-schema-divisions-theme/src/overture/schema/divisions/division_boundary.py # packages/overture-schema-places-theme/pyproject.toml # packages/overture-schema-system/README.md # packages/overture-schema-system/src/overture/schema/system/field_constraint/string.py # packages/overture-schema-system/src/overture/schema/system/ref/ref.py # packages/overture-schema-transportation-theme/pyproject.toml # packages/overture-schema/pyproject.toml # pyproject.toml # uv.lock Signed-off-by: John McCall <john@overturemaps.org>
Stephen Epps (stepps00)
requested review from
John McCall (lowlydba),
Seth Fitzsimmons (sethfitz) and
Victor Schappert (vcschapp)
Stephen Epps (stepps00)
changed the title
🗺️ Schema reference docs preview is live!
Note ♻️ This preview updates automatically with each push to this PR. |
Stephen Epps (stepps00)
added
change type - minor 🤏
Contributor
Author
|
Closing in preference of a new pull request that only includes commits with version updates. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #529
For February schema release. Version:
v1.17.0This pull request includes all commits from
staging- some of which are currently on themainbranch, some of which are not. After merge, this should givestagingandmaina common history, so that in the future all the merges are clean and any commits map to actual net-new changes.Changes in these commits:
operating_statusfield optional for Places