Fix realtime segment size metadata

[xiangfu0]

Summary

• Finalize locally generated realtime segments with the latest index configuration before calculating segmentSizeBytes and creating the commit archive.
• Calculate the normal uncompressed size from the active segment directory so it matches the loaded immutable segment's getSegmentSizeBytes() value.
• Avoid an immediate duplicate preprocessing pass when the finalized files still match the latest configuration, while retaining the recheck in case the configuration changed again.
• Refresh stale segment.size.in.bytes metadata from the loaded immutable segment during deep-store recovery of an already-DONE segment.
• Document the field semantics and provide example metadata/table-size API queries.

This change does not add compressed-size metadata or change the segment completion protocol.

Why the old value was wrong

RealtimeSegmentDataManager retains the table configuration captured when consumption starts. At seal time, the realtime converter used that construction-time snapshot and immediately measured the generated directory. That value was sent to the controller and persisted as segment.size.in.bytes.

The segment was not necessarily in its final on-disk form yet. When the consuming segment was replaced, RealtimeTableDataManager fetched the latest table configuration and ImmutableSegmentLoader preprocessed the files. Preprocessing can remove indexes that were deleted from the table config or rebuild indexes whose definitions changed. For a long-running consuming segment, the preprocessed directory could therefore be substantially smaller than the directory measured before finalization.

The result was a stale pre-finalization byte count in ZK metadata, even though every loaded replica contained the smaller finalized immutable segment. In the regression fixture, the old path published 37,452 bytes while the finalized loaded segment occupied 35,174 bytes.

The deep-store recovery path had a second stale-value case. The selected server already returned ImmutableSegment#getSegmentSizeBytes(), but the controller only copied the full response for a COMMITTING segment. For an already-DONE segment it updated CRC fields and the download URL while preserving the old size, so recovery could not repair the confusing value.

Fix

Before publishing a locally generated segment, the server now:

1. Fetches the latest IndexLoadingConfig.
2. Preprocesses the generated segment with that configuration.
3. Measures the active V1/V2/V3 segment directory.
4. Creates the commit archive from those finalized files.

The local replacement path checks whether another preprocessing pass is actually needed. It skips the pass when the just-finalized files still match the current configuration, but preprocesses again if the configuration changed between build and replacement.

For deep-store recovery, the controller accepts a non-negative size returned by the selected server for DONE segments, just as it already accepts refreshed CRC metadata.

segment.size.in.bytes remains the logical uncompressed footprint of the finalized immutable segment. It is not the compressed archive length and is not filesystem-allocated block usage from du. It is one shared metadata snapshot; later reloads can still make individual replica footprints differ.

User guide and sample queries

pinot-tools/src/main/resources/examples/compression-stats/README.md now explains the field and includes curl/jq examples comparing:

• segment.size.in.bytes from the segment metadata API; and
• each loaded replica's diskSizeInBytes from /tables/{table}/size?verbose=true.

There is no sample table-config change because this fix introduces no configuration option.

Testing

• RealtimeSegmentDataManagerTest: 28 tests passed, including a regression that changes the inverted-index configuration while a segment is consuming and verifies the published size equals the finalized loaded size.
• LoaderTest#testIfNeedConvertSegmentFormat: verifies file-based preprocessing decisions before and after format conversion.
• PinotLLCRealtimeSegmentManagerTest#testUploadCommittedSegment: verifies DONE-segment recovery replaces stale size metadata with the loaded server value.
• spotless:apply, checkstyle:check, license:format, and license:check passed for pinot-segment-local, pinot-core, pinot-controller, and pinot-tools.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 60.00000% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.24%. Comparing base (068b458) to head (3490f3c).

Files with missing lines	Patch %	Lines
.../core/realtime/PinotLLCRealtimeSegmentManager.java	40.00%	4 Missing and 2 partials ⚠️
...a/manager/realtime/RealtimeSegmentDataManager.java	60.00%	4 Missing ⚠️
...ata/manager/realtime/RealtimeTableDataManager.java	0.00%	2 Missing ⚠️
...indexsegment/immutable/ImmutableSegmentLoader.java	84.61%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18992      +/-   ##
============================================
+ Coverage     65.22%   65.24%   +0.01%     
  Complexity     1405     1405              
============================================
  Files          3418     3418              
  Lines        214651   214674      +23     
  Branches      33922    33924       +2     
============================================
+ Hits         140006   140062      +56     
+ Misses        63356    63313      -43     
- Partials      11289    11299      +10     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-21	65.24% <60.00%> (+0.01%)	⬆️
temurin	65.24% <60.00%> (+0.01%)	⬆️
unittests	65.24% <60.00%> (+0.01%)	⬆️
unittests1	57.04% <68.00%> (+0.18%)	⬆️
unittests2	37.73% <42.85%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.