Skip to content

[12.x] Memoize credentials in SqsConnector#59867

Merged
taylorotwell merged 1 commit into
laravel:12.xfrom
kieranbrown:sqs-memoize-credential-provider-12x
Apr 26, 2026
Merged

[12.x] Memoize credentials in SqsConnector#59867
taylorotwell merged 1 commit into
laravel:12.xfrom
kieranbrown:sqs-memoize-credential-provider-12x

Conversation

@kieranbrown
Copy link
Copy Markdown
Member

Summary

Backport of #59866 to 12.x. SqsConnector.php is identical between 12.x and 13.x, so this is a straight cherry-pick.

When a queue config sets credentials.provider = ecs (or instance) — as Laravel Cloud does for managed queue workers via Illuminate\Foundation\Cloud::configureManagedQueues()SqsConnector::resolveCredentialProvider() returned a raw Aws\Credentials\EcsCredentialProvider / Aws\Credentials\InstanceProfileProvider instance.

The AWS SDK's Aws\ClientResolver::_apply_credentials() short-circuits any callable passed as credentials (the SDK only auto-wraps in memoize() for its own internal defaultProvider() path), and the signer middleware (Aws\Middleware::signer) invokes the credentials provider on every signed request. Combined with the fact that EcsCredentialProvider / InstanceProfileProvider issue a fresh HTTP GET to the EKS Pod Identity Agent / EC2 metadata endpoint on every __invoke(), the result is that every single SQS API call triggers an HTTP fetch to the credentials endpoint, saturating the agent's built-in rate limiter under steady-state load.

What this PR does

Wraps the resolved provider in Aws\Credentials\CredentialProvider::memoize() so credentials are cached in-process for the lifetime of the worker, with the SDK's standard 60-second pre-expiry refresh window. This matches what the SDK's own CredentialProvider::defaultProvider() does for the auto-discovered chain (which also wraps ecsCredentials() in memoize()).

$resolved = match ($provider) {
    'ecs' => CredentialProvider::ecsCredentials($options),
    'instance' => CredentialProvider::instanceProfile($options),
    default => throw new InvalidArgumentException(
        \"Invalid credential provider [{$provider}].\"
    ),
};

return CredentialProvider::memoize($resolved);

Benefit to end users

  • Queue workers using ECS / EKS Pod Identity credentials make one credentials fetch per worker per ~6 hours instead of one per signed SQS request.
  • Drastically reduces request volume to the Pod Identity Agent, eliminating 429 throttling under steady-state load.
  • Same fix applies to the instance (EC2 IMDS) branch.
  • No config change required.

Backwards compatibility

  • No public API change. Return type of resolveCredentialProvider() remains callable|null.
  • CredentialProvider::memoize() is part of the AWS SDK's public, documented API.
  • Per-process scope: each worker has its own cache (PHP scopes static per Closure instance). No cross-worker state.

Test plan

  • tests/Queue/ suite passes locally (177 tests, 480 assertions).

Generated by AI tools (Claude), and reviewed by Kieran Brown.

When a queue config sets `credentials.provider = ecs` (or `instance`),
SqsConnector::resolveCredentialProvider returned a raw EcsCredentialProvider
or InstanceProfileProvider. The AWS SDK's ClientResolver short-circuits any
callable passed as `credentials` (no automatic memoize wrap), and the signer
middleware invokes the provider on every signed request — so every SQS API
call triggered a fresh HTTP fetch to the EKS Pod Identity Agent / EC2
metadata endpoint.

Wrap the resolved provider in CredentialProvider::memoize so credentials are
cached in-process for the lifetime of the worker, with the SDK's standard
60-second pre-expiry refresh window. This matches what the SDK's own
defaultProvider() does and stops queue workers from saturating the Pod
Identity Agent's rate limiter under steady-state polling.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@taylorotwell taylorotwell merged commit 94dafaa into laravel:12.x Apr 26, 2026
72 checks passed
jonagoldman pushed a commit to deplox/laravel-framework that referenced this pull request Apr 30, 2026
When a queue config sets `credentials.provider = ecs` (or `instance`),
SqsConnector::resolveCredentialProvider returned a raw EcsCredentialProvider
or InstanceProfileProvider. The AWS SDK's ClientResolver short-circuits any
callable passed as `credentials` (no automatic memoize wrap), and the signer
middleware invokes the provider on every signed request — so every SQS API
call triggered a fresh HTTP fetch to the EKS Pod Identity Agent / EC2
metadata endpoint.

Wrap the resolved provider in CredentialProvider::memoize so credentials are
cached in-process for the lifetime of the worker, with the SDK's standard
60-second pre-expiry refresh window. This matches what the SDK's own
defaultProvider() does and stops queue workers from saturating the Pod
Identity Agent's rate limiter under steady-state polling.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
timacdonald added a commit to timacdonald/framework that referenced this pull request May 14, 2026
taylorotwell added a commit that referenced this pull request May 14, 2026
* [13.x] Cloud queue metrics (#60074)

* Cloud queue metrics

* Reset static state before running tests

* Add binding test

* formatting

---------

Co-authored-by: Taylor Otwell <taylor@laravel.com>

* Support `after_commit` for queue metrics (#60078)

* Use config for queue suffix and prefix (#60094)

* Improve queue metric tests

* Back port #58341

* Back port #59310

* Back port #59370

* Lint

* Remove restart and pause functionality

* Remove unsupported tests

* Remove unsupported test

* Remove unsupported test

* Ensure queue is chopped correctly

* Ensure to configure managed queues

* Back port #59754

* Back port #59867

* Fix issue using custom aws credential providers

Includes the fix from #60000 — allows non-string credential providers
(e.g. callables from CredentialProvider::ini()) to be passed through
to the AWS SDK instead of being treated as invalid.

---------

Co-authored-by: Taylor Otwell <taylor@laravel.com>
Co-authored-by: Kieran Brown <kswb96@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants