Expose stage/release status in schema#5443
Open
janniklasrose wants to merge 8 commits into
Open
Conversation
Add per-enum-value plumbing on top of the field-level launch-stage support: - libs/jsonschema/extension.go: EnumMetadata + EnumValueMetadata for parsing x-databricks-enum-metadata from the OpenAPI spec; EnumDescriptionMap for x-databricks-enum-descriptions; EnumDescriptions []string for the output schema's parallel enumDescriptions array. - bundle/internal/annotation/descriptor.go: EnumLaunchStages and EnumDescriptions maps so the per-value data persists in annotations_openapi.yml. - bundle/internal/schema/parser.go: extractEnumLaunchStages and extractEnumDescriptions helpers; wired into extractAnnotations on both the root-type and per-property paths. - bundle/internal/schema/annotations.go: buildEnumDescriptions builds the parallel []string aligned 1:1 with Enum. enumDescriptionLabel formats one entry; previewTagShort returns the compact bracketed label ([PuPr], [Beta], [PrPr]) so VSCode can render per-value status next to each enum value in autocomplete dropdowns. - Unit tests cover buildEnumDescriptions (all paths, including non-string entries), extractEnumLaunchStages, extractEnumDescriptions, and assignAnnotation's per-value behavior. Schema regeneration is deferred until the DEVELOPMENT-value filter lands in the next commit, so we only regenerate once. Co-authored-by: Isaac
Genkit drops DEVELOPMENT-stage enum values when generating the SDK, so the SDK Go types have no constants for them. The CLI's annotation extractor didn't mirror that and copied the OpenAPI spec's enum list verbatim, causing values like vectorsearch.EndpointType.STANDARD_ON_ORION to land in bundle/schema/jsonschema.json on every regen. IDE schema validation would accept them; deploys would fail server-side. Fix: - bundle/internal/schema/parser.go: add filterDevelopmentEnumValues helper that drops any enum value whose x-databricks-enum-metadata marks it DEVELOPMENT. Wire it into extractAnnotations on both the root-type and per-property paths (Enum: filterDevelopmentEnumValues(...)). Also skip DEVELOPMENT entries in extractEnumLaunchStages and extractEnumDescriptions so the per-value metadata can't reintroduce the value names through a side channel. - bundle/internal/schema/annotations_test.go: add TestFilterDevelopmentEnumValues and extend the per-value extraction tests with DEVELOPMENT cases. - bundle/internal/schema/illegal-enum-in-schema.md: document why the filter exists (genkit's behavior is the source of truth) so future readers don't reintroduce the divergence. - bundle/internal/schema/sync-missing-annotations.md: track an orthogonal follow-up (stale PLACEHOLDER entries in annotations.yml) that surfaced while validating this work. - Regenerated bundle/internal/schema/annotations_openapi.yml, bundle/schema/jsonschema.json, bundle/schema/jsonschema_for_docs.json with the filter active. STANDARD_ON_ORION and any other DEVELOPMENT-stage enum values are gone. Co-authored-by: Isaac
The DEVELOPMENT case in previewTag was dead code: extractAnnotations in parser.go already returns early for DEVELOPMENT-stage types/properties, so a DEVELOPMENT value never reaches assignAnnotation. If it ever did (e.g. via a hand-edited annotation file), the previous behavior would leak a [Development] marker into the public schema — exactly what we filter out elsewhere. The companion previewTagShort already returned "" for DEVELOPMENT; this restores consistency between the two. Co-authored-by: Isaac
The legacy x-databricks-preview extension's PUBLIC value literally means
"Public Preview" per the upstream source at
universe/openapi/openapi/openapi.go:
PrivatePreviewType: "PRIVATE"
PublicPreviewType: "PUBLIC" // ← Public Preview, not GA
GAPreviewType: "GA"
func (pt PreviewType) IsPublicPreview() bool { return pt == PublicPreviewType }
The CLI's annotation extractor was stripping PUBLIC in normalizePreview
(treating it as a GA-equivalent default), losing the preview signal for
any field that only carried the legacy extension. In practice the modern
x-databricks-launch-stage is the primary source today, but PUBLIC could
silently shadow real preview status if a spec entry has only the legacy
field set.
Fix:
- bundle/internal/schema/parser.go: normalizePreview now drops only
empty/UNKNOWN/GA values; PUBLIC and PRIVATE are preserved.
- bundle/internal/schema/annotations.go: previewTag maps PUBLIC ->
[Public Preview] alongside the existing PRIVATE -> [Private Preview]
fallback. Launch stage still wins precedence when both are set, so a
field with both PUBLIC and PUBLIC_PREVIEW only gets one [Public Preview]
prefix.
- prefixWithPreviewTag is hardened against double-tagging: if the
description already starts with the tag, it's returned unchanged.
- Unit tests cover the table of (launchStage, preview) combinations,
the no-double-tag guarantee on the helper, and the assignAnnotation
scenarios that exercise the new fallback. Test-only at the helper
level so we don't assert against the same fact across multiple
generated files.
- Regenerated annotations_openapi.yml and both jsonschema files now
carry x-databricks-preview: PUBLIC entries for fields tagged that way
in the spec; their descriptions show [Public Preview] exactly once.
Co-authored-by: Isaac
ilyakuz-db
reviewed
Jun 5, 2026
| @@ -139,13 +139,51 @@ func assignAnnotation(s *jsonschema.Schema, a annotation.Descriptor) { | |||
| s.Deprecated = true | |||
| s.DeprecationMessage = a.DeprecationMessage | |||
| } | |||
| if a.Preview == "PRIVATE" { | |||
| s.DoNotSuggest = true | |||
| if a.Preview != "" { | |||
ilyakuz-db
approved these changes
Jun 5, 2026
Brings in SDK bump v0.132.0 -> v0.141.0 (OpenAPI sha a499dda -> c68a27fa). Conflicts were limited to three generated schema files: - bundle/internal/schema/annotations_openapi.yml: hand-merged as the union of main's new SDK content (fields, types, rewritten descriptions) and this branch's x-databricks-launch-stage / x-databricks-enum-launch-stages annotations. This file is produced by a non-standard genkit that emits launch-stage, so it cannot be regenerated from the universe master toolchain without wiping the 390 launch-stage annotations. Enum-value conflicts were resolved using the branch's own rule (see filterDevelopmentEnumValues in parser.go): genkit drops DEVELOPMENT-stage values from the SDK, so a value is dropped iff it has no Go SDK constant at v0.141.0. Dropped: CREATE_FEATURE, READ_FEATURE (catalog.Privilege); UPGRADING (database.DatabaseInstanceState); FILE (pipelines.FileIngestionOptionsFileFormat); FILE_PERMISSION, GROUP_MEMBERSHIP (GoogleDrive/Sharepoint EntityType); AVRO, PROTOBUF (pipelines.TransformerFormat). Kept GPU_XLARGE (present in the SDK). - bundle/schema/jsonschema.json, bundle/schema/jsonschema_for_docs.json: regenerated from the resolved annotations via `go run ./bundle/internal/schema` (no DATABRICKS_OPENAPI_SPEC, so the hand-merged annotations_openapi.yml is consumed, not overwritten). reference.md / resources.md are left at main's v0.141.0 output; the branch's docsgen launch-stage prefixes were not yet applied to the human docs (also true before this merge). Co-authored-by: Isaac
Follow-up to the origin/main merge, addressing review feedback on stale
generated artifacts and previewTag drift.
- bundle/docsgen/main.go: previewTag had drifted from the schema-side copy in
bundle/internal/schema/annotations.go — it still mapped DEVELOPMENT to
[Development] and lacked the legacy x-databricks-preview PUBLIC ->
[Public Preview] fallback. Aligned the two and noted the keep-in-sync
invariant.
- bundle/docsgen/output/{reference,resources}.md: regenerated. The docsgen
launch-stage prefixes added on this branch had never been applied to the
committed docs; they now match jsonschema.json (e.g. budget_policy_id is
tagged [Public Preview], no [Development] markers leak).
- python/databricks/bundles/**: regenerated via pydabs-codegen, which reads
bundle/schema/jsonschema.json. Picks up the launch-stage prefixes and drops
the DEVELOPMENT enum values filtered from the schema (catalog.Privilege
loses CREATE_FEATURE/READ_FEATURE/etc.; pipelines.TransformerFormat loses
AVRO/PROTOBUF), keeping the Python models consistent with the Go schema.
- bundle/internal/schema/parser.go: defer creating the per-type annotation
map until after the DEVELOPMENT early-return, so a DEVELOPMENT root type no
longer leaves an empty package entry in annotations_openapi.yml.
Co-authored-by: Isaac
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
janniklasrose
requested review from
andrewnester,
denik,
pietern and
shreyas-goenka
Contributor
Approval status: pending
|
Collaborator
|
Commit: a4089fd
24 interesting tests: 15 SKIP, 6 RECOVERED, 3 flaky
Top 24 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Expose launch stage/release status in the bundle schema
Why
Non-GA fields should be annotated as such when viewing a bundle in IDEs.
Tests
Example vector search endpoint:
Enum with one GA, one PuPr:
PuPr field without any annotation:
Side fix: DEVELOPMENT enum values no longer leak into the schema
While wiring up per-enum-value metadata, we noticed the schema on
mainwas emitting enum values that genkit drops from the SDK because they're tagged DEVELOPMENT in the OpenAPI spec (e.g.vectorsearch.EndpointType.STANDARD_ON_ORION— no Go constant exists, deploys with that value would fail server-side). The extractor was copying OpenAPI'senumlist verbatim, never consultingx-databricks-enum-metadata.Fixed in the same PR by filtering DEVELOPMENT values at extraction time so they never reach
annotations_openapi.ymlor the schema.