[fix](http) Fall back to ranged GET when HEAD is rejected for HTTP file size probe - #66595
Open
zghong wants to merge 2 commits into
Open
[fix](http) Fall back to ranged GET when HEAD is rejected for HTTP file size probe#66595zghong wants to merge 2 commits into
zghong wants to merge 2 commits into
Conversation
Some servers reject HEAD requests, leaving the HTTP file size unknown. Fall back to a ranged GET (Range: bytes=0-0) and parse the total from the Content-Range header on 206, or Content-Length on 200, so downloads/reads still work. - BE: add HttpClient::get_content_range_total; recover size from Content-Range/Content-Length during range-support detection - FE: add ranged-GET fallback in HttpUtils file-size probing
Add BE/FE unit tests and a regression case simulating presigned URLs whose signature only allows GET. HEAD returns 403, so file-size detection must fall back to a GET-based probe instead of failing.
zghong
requested review from
Gabriel39,
gavinchou,
liaoxin01,
luwei16 and
morningman
as code owners
August 10, 2026 02:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #66527
Problem Summary:
When reading an HTTP resource (e.g. via the HTTP TVF), both BE (
HttpFileReader)and FE (
HttpUtils) first send an HTTP HEAD request to obtain the file size fromthe
Content-Lengthheader. Some resources reject HEAD requests -- most notablypresigned object-storage URLs whose signature covers the HTTP method, so a URL
signed for GET is rejected with 403 when accessed via HEAD, even though the same
URL works fine with GET. In that case the HEAD failure aborted opening the file,
making such presigned URLs unusable.
Fix: a HEAD failure (or a HEAD response without a usable size) no longer aborts
the open path. Instead it falls back to a ranged GET (
Range: bytes=0-0), whichmirrors the actual read path and recovers the total size from the
Content-Rangeheader on a 206 Partial Content response (format
bytes <start>-<end>/<total>,with
*treated as unknown), or fromContent-Lengthon a 200 response.HttpFileReader::opennow issues the HEAD probe without failing on error;if no size is obtained it reuses the existing ranged-GET Range-support probe to
recover the size. A new
HttpClient::get_content_range_totalparses the totalfrom the
Content-Rangeheader.HttpUtils.getHttpFileSizeis split intotryGetFileSizeWithHeadandtryGetFileSizeWithGetRange, attempting HEAD first and falling back to theranged GET, parsing
Content-Range/Content-Lengthaccordingly.Before the fix presigned GET-only URLs failed to open with a HEAD 403 error;
after the fix the file size is resolved via the ranged GET and the read succeeds.
Release note
None
Check List (For Author)
Test
regression-test/suites/external_table_p0/tvf/test_http_tvf.groovybe/test/io/fs/http_file_reader_test.cpp, FEHttpUtilsTest.javaBehavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)