Skip to content

[fix](http) Fall back to ranged GET when HEAD is rejected for HTTP file size probe - #66595

Open
zghong wants to merge 2 commits into
apache:masterfrom
zghong:fix/http-head-fail-back-get
Open

[fix](http) Fall back to ranged GET when HEAD is rejected for HTTP file size probe#66595
zghong wants to merge 2 commits into
apache:masterfrom
zghong:fix/http-head-fail-back-get

Conversation

@zghong

@zghong zghong commented Aug 10, 2026

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: close #66527

Problem Summary:

When reading an HTTP resource (e.g. via the HTTP TVF), both BE (HttpFileReader)
and FE (HttpUtils) first send an HTTP HEAD request to obtain the file size from
the Content-Length header. Some resources reject HEAD requests -- most notably
presigned object-storage URLs whose signature covers the HTTP method, so a URL
signed for GET is rejected with 403 when accessed via HEAD, even though the same
URL works fine with GET. In that case the HEAD failure aborted opening the file,
making such presigned URLs unusable.

Fix: a HEAD failure (or a HEAD response without a usable size) no longer aborts
the open path. Instead it falls back to a ranged GET (Range: bytes=0-0), which
mirrors the actual read path and recovers the total size from the Content-Range
header on a 206 Partial Content response (format bytes <start>-<end>/<total>,
with * treated as unknown), or from Content-Length on a 200 response.

  • BE: HttpFileReader::open now issues the HEAD probe without failing on error;
    if no size is obtained it reuses the existing ranged-GET Range-support probe to
    recover the size. A new HttpClient::get_content_range_total parses the total
    from the Content-Range header.
  • FE: HttpUtils.getHttpFileSize is split into tryGetFileSizeWithHead and
    tryGetFileSizeWithGetRange, attempting HEAD first and falling back to the
    ranged GET, parsing Content-Range/Content-Length accordingly.

Before the fix presigned GET-only URLs failed to open with a HEAD 403 error;
after the fix the file size is resolved via the ranged GET and the read succeeds.

Release note

None

Check List (For Author)

  • Test

    • Regression test: regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy
    • Unit Test: BE be/test/io/fs/http_file_reader_test.cpp, FE HttpUtilsTest.java
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

zghong added 2 commits August 9, 2026 17:30
Some servers reject HEAD requests, leaving the HTTP file size
unknown. Fall back to a ranged GET (Range: bytes=0-0) and parse
the total from the Content-Range header on 206, or Content-Length
on 200, so downloads/reads still work.

- BE: add HttpClient::get_content_range_total; recover size from
  Content-Range/Content-Length during range-support detection
- FE: add ranged-GET fallback in HttpUtils file-size probing
Add BE/FE unit tests and a regression case simulating presigned URLs
whose signature only allows GET. HEAD returns 403, so file-size
detection must fall back to a GET-based probe instead of failing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] HTTP func fails with Failed to get file size for URI

1 participant