Generate UUID sample data in the schema recommender#18973
Conversation
There was a problem hiding this comment.
Pull request overview
Adds end-to-end UUID support to the schema recommender’s sample-data pipeline by generating UUID values and ensuring the generated Avro schema represents UUID columns as logical-type UUID strings.
Changes:
- Add
UuidGeneratorand wireDataType.UUIDrouting inGeneratorFactory. - Update
AvroSchemaUtil.toAvroSchemaJsonObjectto emit UUID as AvrostringwithlogicalType: "uuid"(instead ofbytes) for recommender-generated Avro. - Add/extend tests covering UUID schema mapping and Avro round-trip behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| pinot-plugins/pinot-input-format/pinot-avro-base/src/test/java/org/apache/pinot/plugin/inputformat/avro/AvroSchemaUtilTest.java | Adds UUID-specific schema assertions for logicalType: "uuid" and updates prior expectations. |
| pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroSchemaUtil.java | Maps DataType.UUID to Avro string annotated with logicalType: "uuid" for recommender Avro schema generation. |
| pinot-controller/src/test/java/org/apache/pinot/controller/recommender/data/writer/AvroWriterTest.java | Adds an end-to-end UUID round-trip test through Avro writing/reading. |
| pinot-controller/src/test/java/org/apache/pinot/controller/recommender/data/generator/UuidGeneratorTest.java | New unit tests for UUID generation semantics (canonical formatting, cardinality cycling, MV behavior). |
| pinot-controller/src/main/java/org/apache/pinot/controller/recommender/data/generator/UuidGenerator.java | New generator producing UUID-formatted strings with bounded distinct values. |
| pinot-controller/src/main/java/org/apache/pinot/controller/recommender/data/generator/GeneratorFactory.java | Routes DataType.UUID to the new UuidGenerator. |
ca059ee to
dedab74
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #18973 +/- ##
============================================
+ Coverage 65.21% 65.24% +0.02%
Complexity 1405 1405
============================================
Files 3418 3419 +1
Lines 214651 214678 +27
Branches 33922 33924 +2
============================================
+ Hits 139989 140056 +67
+ Misses 63356 63318 -38
+ Partials 11306 11304 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
5722632 to
224ffd2
Compare
The recommender data generator had no UUID support, so recommending a schema with a UUID column would fail. Add UuidGenerator (canonical RFC-4122 strings, cardinality-bounded via a fixed high half + cycling low counter, mirroring StringGenerator) and route DataType.UUID to it in GeneratorFactory. Map UUID columns in the generated recommender Avro schema to a string with logicalType "uuid" (matching the original-type approach used for BOOLEAN and TIMESTAMP) instead of collapsing to raw bytes, so the generated sample data round-trips as canonical UUID strings. toAvroSchemaJsonObject is a recommender-only helper with a single caller; this does not touch ingestion or segment-conversion Avro mapping. Tests: UuidGenerator single/multi-value + cardinality-cycling + default; the UUID schema mapping; and an end-to-end AvroWriter round-trip for a UUID column.
224ffd2 to
fb6299c
Compare
yashmayya
left a comment
There was a problem hiding this comment.
LGTM. Nicely scoped change.
Verified the key risk — toAvroSchemaJsonObject has exactly one caller (AvroWriter), so the bytes→string(logicalType "uuid") remap stays recommender-only and doesn't touch the ingestion / segment-conversion Avro mapping. The uuid-on-string form is also the spec-correct Avro representation (the uuid logical type must back a string), and since UUID columns previously threw in the generator this only fixes a broken path rather than changing working behavior.
Counter cycling mirrors StringGenerator, the single-value FieldSpec from DataGenerator#buildSpec keeps the scalar Avro schema consistent, and the CSV/segment path in MemoryEstimator converts the canonical string back to bytes fine. Tests cover the round-trip well. 👍
|
Opened docs follow-up PR: pinot-contrib/pinot-docs#917 |
## What changed Added a schema-input note to the configuration recommendation engine docs that UUID columns are supported in the sample-data generation path. ## Why apache/pinot#18973 added UUID support for recommender sample-data generation, including canonical UUID string output and Avro `string` fields annotated with logical type `uuid`. ## Reader impact Users can now include `UUID` fields in the schema they send to `/tables/recommender` without the sample-data generation path failing. ## Source cross-check Verified against local `apache/pinot` merge commit `3790129aa15ec70d9870a5f61fa09b67d2a1fbc3`. ## Validation - `git diff --check` - Verified the edited page's existing relative link target `operate-pinot/tuning/realtime.md` resolves locally
Summary
The schema recommender's data generator had no support for
UUIDcolumns, so recommending a schema containing a UUID column would fail. This adds UUID sample-data generation. Split out from the UUID PR stack as an independent change per review.Changes
UuidGenerator(new): generates canonical RFC-4122 UUID strings. The number of distinct values is bounded by the requested cardinality — a fixed random high-order half combined with a low-order counter that cycles[1, cardinality], mirroringStringGenerator. (Values are not version-4; the version/variant bits are not set, which is irrelevant for sample data — they remain canonically-formatted, parseable UUID strings.)GeneratorFactory: routeDataType.UUIDtoUuidGenerator.AvroSchemaUtil.toAvroSchemaJsonObject: map a UUID column to an Avrostringannotated withlogicalType: "uuid"(consistent with the original-type approach already used forBOOLEANandTIMESTAMP) instead of collapsing to rawbytes, so generated sample data round-trips as canonical UUID strings. This is a recommender-only helper with a single caller (AvroWriter); ingestion and segment-conversion Avro mapping are untouched.Testing
UuidGeneratorTest: single-value canonical output, cardinality bounding + counter cycling, null-numberOfValuesPerEntrydefault, multi-value list.AvroSchemaUtilTest: UUID →logicalType:uuidstring mapping.AvroWriterTest: end-to-end round-trip — a UUID column generates, writes to Avro, and reads back as canonical UUID strings.