Move protobuf and flatbuffer definitions into their owning crates - #9550
Open
robert3005 wants to merge 1 commit into
Open
Move protobuf and flatbuffer definitions into their owning crates#9550robert3005 wants to merge 1 commit into
robert3005 wants to merge 1 commit into
Conversation
## Rationale for this change Closes #1906. The `.fbs` and `.proto` schemas all lived in `vortex-flatbuffers` and `vortex-proto`, away from the types they describe, with their generated Rust checked in. Build-time generation existed before #557 but was removed because `vortex-build` discovered include paths by walking workspace metadata with `cargo_metadata` and reaching outside the package directory, neither of which survives packaging, so publishing broke. Declaring the dependencies explicitly avoids that. ## What changes are included in this PR? Each schema moves to the crate that owns the types it describes, and is compiled into `OUT_DIR` by that crate's `build.rs`. Nothing generated is checked in: | Schema | Crate | Module | | -------------------------- | --------------- | ---------------------------- | | `array.fbs`, `dtype.fbs` | `vortex-array` | `vortex_array::flatbuffers` | | `dtype/scalar/expr.proto` | `vortex-array` | `vortex_array::proto` | | `layout.fbs` | `vortex-layout` | `vortex_layout::flatbuffers` | | `footer.fbs` | `vortex-file` | `vortex_file::flatbuffers` | | `message.fbs` | `vortex-ipc` | `vortex_ipc::flatbuffers` | `vortex-proto` and `vortex-flatbuffers` are both removed. The FlatBuffers read/write traits move into `vortex_array::flatbuffers` alongside the generated array and dtype bindings, which every crate that used them already depended on. The new `vortex-build` crate holds the shared build script helpers, and `xtask` is deleted since code generation was its only job. A crate whose schemas include another crate's names it explicitly: vortex_build::flatbuffers() .depends_on("vortex-array") .compile(&["vortex-serde/message.fbs"]); `depends_on` resolves the dependency's schema directory through Cargo's `links` metadata, so a path dependency in the workspace and a package unpacked from a registry behave identically. `flatc`'s `--include-prefix` points cross-crate includes at a small `deps` module each consuming crate provides by hand, so no schema is compiled twice. The issue suggests generating into `src/flatbuffers/` and git-ignoring it. Generating into `OUT_DIR` instead keeps the same property — no generated code in git — without a build script writing into its own package, which would invalidate the registry checksum for anyone building a published crate. `.proto` compilation uses `protox` rather than `protoc`, so protobuf codegen needs no external tooling; the generated output is byte-identical to what was checked in. `.fbs` compilation shells out to `flatc`, which must be on `PATH` or named by `FLATC`, so CI installs it as part of the shared Rust setup and the musl job pulls it from Alpine. The flatbuffer back-compat check now flattens each revision's per-crate schema directories into one tree before running `flatc --conform`. ## What APIs are changed? Are there any user-facing changes? `vortex_proto::{dtype, scalar, expr}` becomes `vortex_array::proto::*`, and both the flatbuffer traits and the generated modules move from `vortex_flatbuffers::*` to `vortex_array::flatbuffers` and the crates listed above. The `vortex` facade keeps `vortex::proto` and `vortex::flatbuffers` pointing at the same items, and `vortex_array:: dtype`'s `proto` and `flatbuffers` re-exports are unchanged. Building any Vortex crate from source now requires `flatc`, including for downstream consumers and docs.rs. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017zTmZ6ANxESomTnvkvk85t
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Instead of checking in the generated code in centralised crates and reexporting it we generate flatbuffer and protobuffer objects during build.
We remove the xtask module as it no longer has any tasks.
Fix #1906