SIGSEGV in S3CrtRequestHeadersCallback — use-after-free reading ContinueRequestHandler from destroyed request

[mxsrc]

Describe the bug

S3CrtClient stores a raw pointer to the caller's request
(CrtRequestCallbackUserData::originalRequest = &request, S3CrtClient.cpp
InitCommonCrtRequestOption), which leads to a segfault when passing in a stack-local request.

The crash was introduced by upstream PR #3231 "Cancel S3 CRT requests if should not continue". The dangling originalRequest raw pointer existed before that but was not dereferenced.

This is one of six locations where the dangling pointer is dereferenced, the others may be subject to a similar failure, though I did not observe that. This behavior is different in S3Client, which captures the passed request by value, avoiding this failure family.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The program should not crash with a segfault.

Current Behavior

The *Async submit call returns as soon as the request is queued with the CRT, so
a caller that does not keep its request object alive until completion (a common and
previously-safe pattern) has its request destroyed before the response arrives.
When the HTTP response headers are processed, the callback reads
GetContinueRequestHandler() out of the freed request and invokes the resulting
dangling std::function, crashing with SIGSEGV:

#0  0x0 in ?? ()
#1  Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback (response_status=200) at S3CrtClient.cpp:577
#2  s_s3_meta_request_default_request_finished           at aws-c-s3/source/s3_default_meta_request.c
#3  aws_s3_meta_request_finished_request                 at aws-c-s3/source/s3_meta_request.c
...
    aws_event_loop_thread                                at aws-c-io/source/linux/epoll_event_loop.c

Reproduction Steps

Using a small example:

#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentials.h>
#include <aws/core/utils/stream/PreallocatedStreamBuf.h>
#include <aws/s3-crt/S3CrtClient.h>
#include <aws/s3-crt/model/PutObjectRequest.h>
#include <aws/s3-crt/model/CreateBucketRequest.h>
#include <atomic>
#include <cstdlib>
#include <cstring>
#include <memory>
#include <thread>

static std::shared_ptr<Aws::S3Crt::S3CrtClient> client;
static std::atomic<int> done{0};

// The request is a LOCAL: it is destroyed the moment this function returns, long
// before the async HTTP 200 response is processed on an AwsEventLoop thread.
static void submit(uint8_t* buf, size_t len) {
    Aws::S3Crt::Model::PutObjectRequest req;
    req.SetBucket("bck");
    req.SetKey("1/1");
    req.SetContinueRequestHandler([](const Aws::Http::HttpRequest*) { return true; });
    req.SetBody(Aws::MakeShared<Aws::IOStream>(
        "", new Aws::Utils::Stream::PreallocatedStreamBuf(buf, len)));
    client->PutObjectAsync(req,
        [](const Aws::S3Crt::S3CrtClient*, const Aws::S3Crt::Model::PutObjectRequest&,
           const Aws::S3Crt::Model::PutObjectOutcome&,
           const std::shared_ptr<const Aws::Client::AsyncCallerContext>&) {
            done.fetch_add(1);
        }, nullptr);
    // <-- `req` destroyed here; userData->originalRequest now dangles.
}

int main() {
    const char* ep = std::getenv("MINIO_ENDPOINT");
    if (!ep) { std::puts("set MINIO_ENDPOINT"); return 0; }
    Aws::SDKOptions o; Aws::InitAPI(o);
    {
        Aws::S3Crt::S3CrtClientConfiguration cfg;
        cfg.scheme = Aws::Http::Scheme::HTTP;
        cfg.verifySSL = false;
        cfg.region = "us-east-1";
        cfg.useVirtualAddressing = false;        // path-style, for MinIO
        cfg.endpointOverride = ep;
        Aws::Auth::AWSCredentials creds(std::getenv("AWS_ACCESS_KEY_ID"),
                                        std::getenv("AWS_SECRET_ACCESS_KEY"));
        client = Aws::MakeShared<Aws::S3Crt::S3CrtClient>(
            "repro", creds, cfg,
            Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false);

        Aws::S3Crt::Model::CreateBucketRequest cb; cb.SetBucket("bck");
        client->CreateBucket(cb);                // ignore "already owned"

        static uint8_t payload[4096];
        std::memset(payload, 0xAB, sizeof(payload));
        for (int i = 0; i < 16; i++) submit(payload, sizeof(payload)); // crashes here (pre-fix)

        for (int s = 0; s < 500 && done.load() < 16; s++)
            std::this_thread::sleep_for(std::chrono::milliseconds(10));
        std::printf("completed %d/16\n", done.load());
        client.reset();
    }
    Aws::ShutdownAPI(o);
    return 0;
}

reliably crashes:

$ docker run -d -p 9000:9000 -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin quay.io/minio/minio server /data
$ g++ -std=c++17 -g  test.cpp -o repro   -laws-cpp-sdk-s3-crt -laws-cpp-sdk-core -pthread
$ MINIO_ENDPOINT=http://127.0.0.1:9000 AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin ./repro
Segmentation fault         (core dumped) MINIO_ENDPOINT=http://127.0.0.1:9000 AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin ./repro
$ MINIO_ENDPOINT=http://127.0.0.1:9000 AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin gdb ./repro
GNU gdb (GDB) 17.2
Copyright (C) 2025 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test2...
(gdb) r
...                                                                                                                                                    
[Thread debugging using libthread_db enabled]                                                                                                                                                                                                                                                                                                                                                                                                      
[New Thread 0x7ffff62ba6c0 (LWP 8981)]                                                                                                                                                                                                                       
[New Thread 0x7ffff5ab96c0 (LWP 8982)]
[New Thread 0x7ffff52b86c0 (LWP 8983)]
[New Thread 0x7ffff4ab76c0 (LWP 8984)]
[New Thread 0x7fffe7fff6c0 (LWP 8985)]
[New Thread 0x7fffe77fe6c0 (LWP 8986)]
[New Thread 0x7fffe6ffd6c0 (LWP 8995)]
[New Thread 0x7fffe67fc6c0 (LWP 8996)]
[New Thread 0x7fffe5ffb6c0 (LWP 8997)]
[New Thread 0x7fffe57fa6c0 (LWP 8998)]
[New Thread 0x7fffe4ff96c0 (LWP 8999)]
[New Thread 0x7fffc3fff6c0 (LWP 9000)]
[New Thread 0x7fffc37fe6c0 (LWP 9001)]

Thread 6 "AwsEventLoop5" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe7fff6c0 (LWP 8985)]
thread 6

Downloading 201.34 K source file /usr/src/debug/aws-sdk-cpp/aws-sdk-cpp-1.11.792/generated/src/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp
0x00007ffff7d545b7 in Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback (meta_request=<optimized out>, headers=0x7fffdc014b70, response_status=<optimized out>, user_data=<optimized out>)                                                                
    at /usr/src/debug/aws-sdk-cpp/aws-sdk-cpp-1.11.792/generated/src/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp:573
573       auto& shouldContinueFn = userData->originalRequest->GetContinueRequestHandler();
(gdb) thread 6
[Switching to thread 6 (Thread 0x7fffe7fff6c0 (LWP 8985))]
#0  0x00007ffff7d545b7 in Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback (meta_request=<optimized out>, headers=0x7fffdc014b70, response_status=<optimized out>, user_data=<optimized out>)
    at /usr/src/debug/aws-sdk-cpp/aws-sdk-cpp-1.11.792/generated/src/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp:573
573       auto& shouldContinueFn = userData->originalRequest->GetContinueRequestHandler();
(gdb) 
[Switching to thread 6 (Thread 0x7fffe7fff6c0 (LWP 8985))]
#0  0x00007ffff7d545b7 in Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback (meta_request=<optimized out>, headers=0x7fffdc014b70, response_status=<optimized out>, user_data=<optimized out>)
    at /usr/src/debug/aws-sdk-cpp/aws-sdk-cpp-1.11.792/generated/src/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp:573
573       auto& shouldContinueFn = userData->originalRequest->GetContinueRequestHandler();
(gdb) bt
#0  0x00007ffff7d545b7 in Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback (meta_request=<optimized out>, headers=0x7fffdc014b70, response_status=<optimized out>, user_data=<optimized out>)
    at /usr/src/debug/aws-sdk-cpp/aws-sdk-cpp-1.11.792/generated/src/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp:573
#1  0x00007ffff797f39f in s_s3_meta_request_default_request_finished (meta_request=0x555555924610, request=0x7fffe8013f60, error_code=<optimized out>) at /usr/src/debug/aws-c-s3/aws-c-s3-0.12.2/source/s3_default_meta_request.c:417
#2  0x00007ffff79791b2 in aws_s3_client_notify_connection_finished (client=0x5555559789c0, connection=0x7fffe8014770, error_code=0, finish_code=<optimized out>) at /usr/src/debug/aws-c-s3/aws-c-s3-0.12.2/source/s3_client.c:2640
#3  0x00007ffff75b0864 in s_stream_complete (stream=0x7fffdc0120e0, error_code=<optimized out>) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:741
#4  0x00007ffff75b2240 in s_decoder_on_done (user_data=0x7fffdc011ba0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:1504
#5  0x00007ffff75b7c3a in s_mark_done (decoder=0x7fffdc011e60) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_decoder.c:199
#6  s_linestate_header (decoder=0x7fffdc011e60, input=...) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_decoder.c:372
#7  0x00007ffff75b3da5 in s_state_getline (decoder=0x7fffdc011e60, input=0x7fffe7ffd2a0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_decoder.c:128
#8  0x00007ffff75b5cc5 in aws_h1_decode (decoder=0x7fffdc011e60, data=data@entry=0x7fffe7ffd2a0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_decoder.c:742
#9  0x00007ffff75b7116 in s_try_process_next_stream_read_message (connection=0x7fffdc011ba0, out_stop_processing=<synthetic pointer>) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:2057
#10 aws_h1_connection_try_process_read_messages (connection=0x7fffdc011ba0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:1923
#11 0x00007ffff75acb89 in aws_h1_connection_try_process_read_messages (connection=0x7fffdc011ba0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:1896
#12 s_handler_process_read_message (handler=<optimized out>, slot=<optimized out>, message=0x7fffdc00d5e0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:1889
#13 0x00007ffff75790e6 in s_do_read (socket_handler=0x7fffe8014bd0) at /usr/src/debug/aws-c-io/aws-c-io-0.26.3/source/socket_channel_handler.c:166
#14 0x00007ffff7577853 in s_on_socket_io_event (event_loop=<optimized out>, handle=<optimized out>, events=3, user_data=0x7fffdc000c10) at /usr/src/debug/aws-c-io/aws-c-io-0.26.3/source/posix/socket.c:1880
#15 0x00007ffff756ef1f in aws_event_loop_thread (args=<optimized out>) at /usr/src/debug/aws-c-io/aws-c-io-0.26.3/source/linux/epoll_event_loop.c:686
#16 0x00007ffff790523a in thread_fn (arg=0x555555682f60) at /usr/src/debug/aws-c-common/aws-c-common-0.12.6/source/posix/thread.c:177
#17 0x00007ffff7297739 in ?? () from /usr/lib/libc.so.6
#18 0x00007ffff731bedc in ?? () from /usr/lib/libc.so.6

Possible Solution

A hotfix that copies the ContinueRequestHandler into the CrtCallbackRequestUserData works around this issue. The root cause, the dangling originalRequest pointer remains though, potentially leading to other issues when dereferencing it. A proper solution should copy the request into the user data, similar to how S3Client captures it by value.

Additional Information/Context

No response

AWS CPP SDK version used

1.11.792 and 1.11.855

Compiler and Version used

gcc 16.1.1

Operating System and version

Archlinux

[sbaluja]

Hey @mxsrc, thanks for the detailed writeup and repro. This is actually the same thing that came up in #3510, and it's working as intended. The async methods take the request by const ref:

void PutObjectAsync(const PutObjectRequest& request, ...) const;

so the request is a borrow from the caller's scope, as you noted it must be kept it alive until the completion handler fires, same as the client object.

The async methods developer guide calls this out:

The resources for an asynchronous operation must exist until the operation finishes. For example, the client and request objects must exist until the application receives notification that the operation completed.

[mxsrc]

Hi @sbaluja , thanks for the quick reply and the pointer to #3510, I can fix my client keeping this in mind.

I'd still push back on the design of the function contract though, a few reasons:

• S3Client::PutObjectAsync and S3CrtClient::PutObjectAsync share the exact same signature (const PutObjectRequest&), but one copies the request and the other keeps a raw pointer to it. Nothing in the type system signals that the CRT variant needs a stricter lifetime than its sibling.

• I looked for this before filing and overlooked the paragraph that describes this. The constraint isn't in the header or the Doxygen comment, only on a separate guide page.

• This is now the third independent report of the same pattern: this issue, SIGSEGV when any S3CRT async request object gets destructed after queuing request #3510, and S3CrtClient::GetObjectAsync Doesn't copy handler #1655 (handler retained as a raw pointer instead of copied, same root cause). Imho this indicates users intuitively misunderstand the API.

• Against convention. const T& conventionally means "borrowed for this call only," per the rationale behind Core Guidelines (F.7, F.15, Pro.Lifetime). Retaining it past return without reflecting that in the signature is exactly the surprising case that convention exists to prevent.

• It's cheap to fix, S3Client already follows this pattern and a change in semantics is backwards compatible. The same has already been applied for the handler parameter in S3CrtClient::GetObjectAsync Doesn't copy handler #1655, which was changed to be copied instead of retained by pointer. Extending that to the request object is a small, well-precedented change. As it stands, every caller has to manually reason about keeping request objects alive past the initial submit.