Skip to content

Generate UUID sample data in the schema recommender#18973

Merged
xiangfu0 merged 1 commit into
apache:masterfrom
xiangfu0:uuid-recommender-datagen
Jul 16, 2026
Merged

Generate UUID sample data in the schema recommender#18973
xiangfu0 merged 1 commit into
apache:masterfrom
xiangfu0:uuid-recommender-datagen

Conversation

@xiangfu0

Copy link
Copy Markdown
Contributor

Summary

The schema recommender's data generator had no support for UUID columns, so recommending a schema containing a UUID column would fail. This adds UUID sample-data generation. Split out from the UUID PR stack as an independent change per review.

Changes

  • UuidGenerator (new): generates canonical RFC-4122 UUID strings. The number of distinct values is bounded by the requested cardinality — a fixed random high-order half combined with a low-order counter that cycles [1, cardinality], mirroring StringGenerator. (Values are not version-4; the version/variant bits are not set, which is irrelevant for sample data — they remain canonically-formatted, parseable UUID strings.)
  • GeneratorFactory: route DataType.UUID to UuidGenerator.
  • AvroSchemaUtil.toAvroSchemaJsonObject: map a UUID column to an Avro string annotated with logicalType: "uuid" (consistent with the original-type approach already used for BOOLEAN and TIMESTAMP) instead of collapsing to raw bytes, so generated sample data round-trips as canonical UUID strings. This is a recommender-only helper with a single caller (AvroWriter); ingestion and segment-conversion Avro mapping are untouched.

Testing

  • UuidGeneratorTest: single-value canonical output, cardinality bounding + counter cycling, null-numberOfValuesPerEntry default, multi-value list.
  • AvroSchemaUtilTest: UUID → logicalType:uuid string mapping.
  • AvroWriterTest: end-to-end round-trip — a UUID column generates, writes to Avro, and reads back as canonical UUID strings.

@xiangfu0 xiangfu0 added extension-point Adds or modifies an extension/SPI point and removed extension-point Adds or modifies an extension/SPI point labels Jul 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds end-to-end UUID support to the schema recommender’s sample-data pipeline by generating UUID values and ensuring the generated Avro schema represents UUID columns as logical-type UUID strings.

Changes:

  • Add UuidGenerator and wire DataType.UUID routing in GeneratorFactory.
  • Update AvroSchemaUtil.toAvroSchemaJsonObject to emit UUID as Avro string with logicalType: "uuid" (instead of bytes) for recommender-generated Avro.
  • Add/extend tests covering UUID schema mapping and Avro round-trip behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
pinot-plugins/pinot-input-format/pinot-avro-base/src/test/java/org/apache/pinot/plugin/inputformat/avro/AvroSchemaUtilTest.java Adds UUID-specific schema assertions for logicalType: "uuid" and updates prior expectations.
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroSchemaUtil.java Maps DataType.UUID to Avro string annotated with logicalType: "uuid" for recommender Avro schema generation.
pinot-controller/src/test/java/org/apache/pinot/controller/recommender/data/writer/AvroWriterTest.java Adds an end-to-end UUID round-trip test through Avro writing/reading.
pinot-controller/src/test/java/org/apache/pinot/controller/recommender/data/generator/UuidGeneratorTest.java New unit tests for UUID generation semantics (canonical formatting, cardinality cycling, MV behavior).
pinot-controller/src/main/java/org/apache/pinot/controller/recommender/data/generator/UuidGenerator.java New generator producing UUID-formatted strings with bounded distinct values.
pinot-controller/src/main/java/org/apache/pinot/controller/recommender/data/generator/GeneratorFactory.java Routes DataType.UUID to the new UuidGenerator.

@xiangfu0
xiangfu0 force-pushed the uuid-recommender-datagen branch from ca059ee to dedab74 Compare July 11, 2026 06:41
@codecov-commenter

codecov-commenter commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.24%. Comparing base (fd772ee) to head (fb6299c).

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18973      +/-   ##
============================================
+ Coverage     65.21%   65.24%   +0.02%     
  Complexity     1405     1405              
============================================
  Files          3418     3419       +1     
  Lines        214651   214678      +27     
  Branches      33922    33924       +2     
============================================
+ Hits         139989   140056      +67     
+ Misses        63356    63318      -38     
+ Partials      11306    11304       -2     
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-21 65.24% <100.00%> (+0.02%) ⬆️
temurin 65.24% <100.00%> (+0.02%) ⬆️
unittests 65.23% <100.00%> (+0.02%) ⬆️
unittests1 56.86% <0.00%> (+0.01%) ⬆️
unittests2 37.76% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0
xiangfu0 force-pushed the uuid-recommender-datagen branch 5 times, most recently from 5722632 to 224ffd2 Compare July 15, 2026 08:02
The recommender data generator had no UUID support, so recommending a schema
with a UUID column would fail. Add UuidGenerator (canonical RFC-4122 strings,
cardinality-bounded via a fixed high half + cycling low counter, mirroring
StringGenerator) and route DataType.UUID to it in GeneratorFactory.

Map UUID columns in the generated recommender Avro schema to a string with
logicalType "uuid" (matching the original-type approach used for BOOLEAN and
TIMESTAMP) instead of collapsing to raw bytes, so the generated sample data
round-trips as canonical UUID strings. toAvroSchemaJsonObject is a
recommender-only helper with a single caller; this does not touch ingestion or
segment-conversion Avro mapping.

Tests: UuidGenerator single/multi-value + cardinality-cycling + default; the
UUID schema mapping; and an end-to-end AvroWriter round-trip for a UUID column.
@xiangfu0
xiangfu0 force-pushed the uuid-recommender-datagen branch from 224ffd2 to fb6299c Compare July 16, 2026 08:02

@yashmayya yashmayya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nicely scoped change.

Verified the key risk — toAvroSchemaJsonObject has exactly one caller (AvroWriter), so the bytes→string(logicalType "uuid") remap stays recommender-only and doesn't touch the ingestion / segment-conversion Avro mapping. The uuid-on-string form is also the spec-correct Avro representation (the uuid logical type must back a string), and since UUID columns previously threw in the generator this only fixes a broken path rather than changing working behavior.

Counter cycling mirrors StringGenerator, the single-value FieldSpec from DataGenerator#buildSpec keeps the scalar Avro schema consistent, and the CSV/segment path in MemoryEstimator converts the canonical string back to bytes fine. Tests cover the round-trip well. 👍

@xiangfu0
xiangfu0 merged commit 3790129 into apache:master Jul 16, 2026
11 checks passed
@xiangfu0
xiangfu0 deleted the uuid-recommender-datagen branch July 16, 2026 18:37
@xiangfu0

Copy link
Copy Markdown
Contributor Author

Opened docs follow-up PR: pinot-contrib/pinot-docs#917

xiangfu0 added a commit to pinot-contrib/pinot-docs that referenced this pull request Jul 16, 2026
xiangfu0 added a commit to pinot-contrib/pinot-docs that referenced this pull request Jul 16, 2026
## What changed
Added a schema-input note to the configuration recommendation engine
docs that UUID columns are supported in the sample-data generation path.

## Why
apache/pinot#18973 added UUID support for recommender sample-data
generation, including canonical UUID string output and Avro `string`
fields annotated with logical type `uuid`.

## Reader impact
Users can now include `UUID` fields in the schema they send to
`/tables/recommender` without the sample-data generation path failing.

## Source cross-check
Verified against local `apache/pinot` merge commit
`3790129aa15ec70d9870a5f61fa09b67d2a1fbc3`.

## Validation
- `git diff --check`
- Verified the edited page's existing relative link target
`operate-pinot/tuning/realtime.md` resolves locally
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants