Skip to content

fix(messagequeue): bound a subscription to the caller's context - #628

Open
behinddwalls wants to merge 2 commits into
mainfrom
preetam/subscriber-ctx-lifetime
Open

fix(messagequeue): bound a subscription to the caller's context#628
behinddwalls wants to merge 2 commits into
mainfrom
preetam/subscriber-ctx-lifetime

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

The Subscriber interface documents that the delivery channel closes when the subscriber is closed or the context is cancelled. The MySQL subscriber — the only implementation — honoured just the first half: Subscribe's ctx appeared solely in the signature and was never read, and the subscription instead ran on an independent context.Background() that only Close could cancel.

That inert parameter is what let the package's own test leak. TestSubscriber_Subscribe called Subscribe with no paired Close, leaving a managePartitions supervisor per topic running on a one-second discovery ticker. Under the race detector the suite runs slowly enough for a tick to land after the subtest has returned, calling GetLeasedPartitions against a finished gomock controller while zaptest logs to a completed *testing.T. Without -race the suite finishes inside a second and the tick never fires, so the failure surfaces only under --@rules_go//go/config:race — which CI does not run.

What?

Subscribe derives the subscription context from the caller's ctx. Close still cancels that context directly, so a caller whose ctx never completes is unaffected.

Honouring ctx opens a hazard that could not previously exist: cancellation ends the supervisor and closes deliveryCh, but leaves the entry in the subscriptions map, so the next Subscribe on that key would hand a new caller an already-closed channel. Each subscription now carries a done channel, closed by the supervisor on its way out; Subscribe treats an entry whose done is closed as absent and replaces it. That signal has to be a channel rather than a lock-guarded flag, because Close holds subMu across cancelFunc and wg.Wait for every subscription — acquiring the same mutex on the shutdown path would deadlock against the very Close that triggered it.

The consumer now hands Subscribe a detached context. Every service starts its consumers with a SIGTERM-cancelled context, and feeding that straight through would close the delivery channel the moment shutdown began, cutting the consume loop's drain short. The consume loop already ran detached for exactly that reason, so the two now share one context. Production shutdown behaviour is unchanged.

TestSubscriber_Subscribe now closes its subscriber, and two new tests cover the semantics: cancelling the context closes the delivery channel, and a stale entry is replaced rather than reused.

make test-race runs the unit suite under the detector. It needs --build_tests_only — the //... pattern otherwise pulls in the cross-compiled *_linux binaries, which are built without cgo, and race instrumentation requires cgo.

Test Plan

Reproduced first: three of five runs of the filtered test failed with WARNING: DATA RACE, the reader being the managePartitions goroutine reaching GetLeasedPartitions while the subtest goroutine had already finished.

✅ 10/10 runs clean under --@rules_go//go/config:race for both //platform/extension/messagequeue/mysql and //platform/consumer; confirmed with -test.v that the new tests actually ran rather than silently matching nothing.

make test-race — 109/109 targets pass repo-wide, so no other package carried a latent leak.

make test (109/109), gazelle produced no BUILD changes, license headers clean, mocks unchanged.

✅ Integration, uncached, covering the real subscribe/close and shutdown-ordering paths: //test/integration/submitqueue/core/consumer and //test/integration/extension/messagequeue/mysql.

Checked that the new guard is meaningful: with the stale-entry eviction removed, TestSubscriber_SubscribeReplacesStaleSubscription fails on both assertions and passes again once restored.

behinddwalls and others added 2 commits August 21, 2026 10:46
## Summary

### Why?

The `Subscriber` interface documents that the delivery channel closes when the subscriber is closed *or the context is cancelled*. The MySQL subscriber — the only implementation — honoured just the first half: `Subscribe`'s `ctx` appeared solely in the signature and was never read, and the subscription instead ran on an independent `context.Background()` that only `Close` could cancel.

That inert parameter is what let the package's own test leak. `TestSubscriber_Subscribe` called `Subscribe` with no paired `Close`, leaving a `managePartitions` supervisor per topic running on a one-second discovery ticker. Under the race detector the suite runs slowly enough for a tick to land after the subtest has returned, calling `GetLeasedPartitions` against a finished gomock controller while `zaptest` logs to a completed `*testing.T`. Without `-race` the suite finishes inside a second and the tick never fires, so the failure surfaces only under `--@rules_go//go/config:race` — which CI does not run.

### What?

`Subscribe` derives the subscription context from the caller's `ctx`. `Close` still cancels that context directly, so a caller whose `ctx` never completes is unaffected.

Honouring `ctx` opens a hazard that could not previously exist: cancellation ends the supervisor and closes `deliveryCh`, but leaves the entry in the subscriptions map, so the next `Subscribe` on that key would hand a new caller an already-closed channel. Each subscription now carries a `done` channel, closed by the supervisor on its way out; `Subscribe` treats an entry whose `done` is closed as absent and replaces it. That signal has to be a channel rather than a lock-guarded flag, because `Close` holds `subMu` across `cancelFunc` and `wg.Wait` for every subscription — acquiring the same mutex on the shutdown path would deadlock against the very `Close` that triggered it.

The consumer now hands `Subscribe` a detached context. Every service starts its consumers with a SIGTERM-cancelled context, and feeding that straight through would close the delivery channel the moment shutdown began, cutting the consume loop's drain short. The consume loop already ran detached for exactly that reason, so the two now share one context. Production shutdown behaviour is unchanged.

`TestSubscriber_Subscribe` now closes its subscriber, and two new tests cover the semantics: cancelling the context closes the delivery channel, and a stale entry is replaced rather than reused.

`make test-race` runs the unit suite under the detector. It needs `--build_tests_only` — the `//...` pattern otherwise pulls in the cross-compiled `*_linux` binaries, which are built without cgo, and race instrumentation requires cgo.

## Test Plan

Reproduced first: three of five runs of the filtered test failed with `WARNING: DATA RACE`, the reader being the `managePartitions` goroutine reaching `GetLeasedPartitions` while the subtest goroutine had already finished.

✅ 10/10 runs clean under `--@rules_go//go/config:race` for both `//platform/extension/messagequeue/mysql` and `//platform/consumer`; confirmed with `-test.v` that the new tests actually ran rather than silently matching nothing.

✅ `make test-race` — 109/109 targets pass repo-wide, so no other package carried a latent leak.

✅ `make test` (109/109), gazelle produced no BUILD changes, license headers clean, mocks unchanged.

✅ Integration, uncached, covering the real subscribe/close and shutdown-ordering paths: `//test/integration/submitqueue/core/consumer` and `//test/integration/extension/messagequeue/mysql`.

Checked that the new guard is meaningful: with the stale-entry eviction removed, `TestSubscriber_SubscribeReplacesStaleSubscription` fails on both assertions and passes again once restored.
@behinddwalls
behinddwalls marked this pull request as ready for review August 21, 2026 17:51
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners August 21, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant