HIVE-29703: LLAP: Parquet footer larger than alloc.max fails to cache — RuntimeException: Trying to allocate N; max is M

[deniskuzZ]

What changes were proposed in this pull request?

• Allocate the remainder buffer with smallSize, not length.
• Use maxAlloc <= length so a footer of exactly maxAlloc is held in one full buffer.

Why are the changes needed?

When LLAP caches a Parquet file footer whose size exceeds hive.llap.io.allocator.alloc.max, the query fails with:

  java.lang.RuntimeException: Trying to allocate 35405479; max is 33554432

Does this PR introduce any user-facing change?

No

How was this patch tested?

mvn test -pl llap-server -Dtest='TestOrcMetadataCache#testStreamFooterEqualToMaxAlloc+testStreamFooterLargerThanMaxAlloc

[Copilot]

Pull request overview

Fixes an LLAP MetadataCache allocation bug when caching file footers via the InputStream path, specifically for footers at or above allocator.getMaxAllocation(), preventing failures like “Trying to allocate N; max is M”.

Changes:

• Adjust split logic to treat length == maxAlloc as a “full-buffer” case (avoid returning an empty wrapper).
• Allocate the remainder buffer using smallSize (remainder length) instead of the full length when footers are split.
• Refactor TestOrcMetadataCache to reuse a shared MetadataCache factory and add stream-based regression tests for == maxAlloc and > maxAlloc.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/MetadataCache.java	Fixes buffer-splitting and remainder allocation so stream footers at/over max alloc are cached without over-allocation or empty-wrapper behavior.
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java	Adds regression coverage for stream footer sizes equal to and larger than MAX_ALLOC, and reduces test duplication via helper constructors.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Aggarwal-Raghav]

Question? If there are any exceptions like IOException or AllocatorOutOfMemoryException in readIntoCacheBuffer()/ allocateMultiple() shouldn't there be a finally block to deallocate the previously allocated buffer memory?

In readIntoCacheBuffer() there is no finally block. In allocateMultiple() there is finally block but will deallocate the current buffer not previously allocated buffer.

[deniskuzZ]

need to check, but that's seems to be a diff issue

[Aggarwal-Raghav]

yeah, its outside the scope of current JIRA. If it's ok , can I pick it up as separate JIRA? Will come up with a UT for this?

[deniskuzZ]

oh, haven't noticed your latest comment and decided to fix it here: 3bf7817

[Aggarwal-Raghav]

No worries, resolution of issue is important.

[Aggarwal-Raghav]

LGTM +1

[sonarqubecloud]

Quality Gate passed

Issues
15 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud