From 233d3a3b6817f62a63c5a9cb3529d304a4211d19 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Sat, 4 Jul 2026 13:26:05 +0530 Subject: [PATCH 1/4] fix(uts): correct objects proxy specs and harden proxy rule/timeout guidance Fixes found while evaluating the derived ably-java ObjectsFaultsTest against the sandbox through uts-proxy: - objects_faults.md: restructure the RTO20e test to the RTO20e1 sequence (mutation in flight while SYNCING, then channel enters FAILED -> 92008). The previous steps mutated after FAILED, which per RTO26b fails with 90001 and never reaches publishAndApply - confirmed against objects-features.md, ably-js source and both ably-js/ably-java behaviour. - objects_faults.md: rule match.action values must be strings ("20"), not JSON numbers (uts-proxy rejects numbers with HTTP 400); bound every await with a timeout; extend the action-number table (ACK, ERROR); align the Common Cleanup with multi-client tests. - connection_resume.md: RTN15g rule used invalid action type "refuse" (silently passes through) - corrected to "refuse_connection". - helpers/proxy.md: document the match.action string format, the v0.3.0 action-name resolution limit (names beyond AUTH need numeric strings), and add a wall-clock timeout convention. - docs: add wall-clock timeout guidance for integration specs (writing-derived-tests.md new section, integration-testing.md and writing-test-specs.md pointers) - virtual-time test frameworks (e.g. kotlinx-coroutines runTest) fire spec timeouts instantly unless waits run on the real clock. - PLAN.md: update the objects_faults.md summary row to the corrected RTO20e scenario. --- uts/docs/integration-testing.md | 6 ++ uts/docs/writing-derived-tests.md | 21 ++++++ uts/docs/writing-test-specs.md | 4 + uts/objects/PLAN.md | 2 +- .../integration/proxy/objects_faults.md | 74 +++++++++++++++---- uts/realtime/integration/helpers/proxy.md | 11 +++ .../integration/proxy/connection_resume.md | 2 +- 7 files changed, 103 insertions(+), 17 deletions(-) diff --git a/uts/docs/integration-testing.md b/uts/docs/integration-testing.md index 74464f5cc..24e90064e 100644 --- a/uts/docs/integration-testing.md +++ b/uts/docs/integration-testing.md @@ -227,6 +227,12 @@ Guidelines: The goal is: every await in the test is bounded, and the suite timeout is generous enough that it only fires if something truly unexpected happens. When a test fails, the error should say *what* timed out, not just "suite timeout exceeded." +**All timeouts are wall-clock time.** Every `WITH timeout`, `poll_until` and `WAIT` in an +integration spec measures real elapsed time — the test is waiting on a real server over a real +network. Derived tests in frameworks that virtualise time by default must run these waits +against the real clock; see the *Integration timeouts are wall-clock* section in +`writing-derived-tests.md` for the failure mode and recommended helpers. + ### Avoiding Flaky Tests - Use polling with timeouts instead of fixed waits (see `README.md` polling conventions) diff --git a/uts/docs/writing-derived-tests.md b/uts/docs/writing-derived-tests.md index 68a22b93c..dd5378115 100644 --- a/uts/docs/writing-derived-tests.md +++ b/uts/docs/writing-derived-tests.md @@ -268,6 +268,27 @@ const timer = setTimeout(() => reject(new Error('Timed out')), 5000); connection.once('connected', () => { clearTimeout(timer); resolve(); }); ``` +### Integration timeouts are wall-clock (beware virtual-time frameworks) + +The rule above is inverted for **integration and proxy tests**: every `WITH timeout`, +`poll_until` and `WAIT` in an integration spec is **wall-clock (real) time**, because the test +is waiting on a real server or proxy over a real network. + +This is a trap in test frameworks that virtualise time by default. For example, +kotlinx-coroutines' `runTest` runs the test body on a virtual clock: a bare `withTimeout(15s)` +wrapping a real network await measures *virtual* time, which fast-forwards the moment the test +coroutine idles — the timeout fires almost instantly, long before the server can respond, with +a misleading "Timed out after 15s" error. The same applies to a bare `delay()`, which skips +instead of waiting. + +Derived integration tests in such frameworks must run their waits against the real clock — +e.g. dispatch onto a real-thread dispatcher before applying the timeout +(`withContext(Dispatchers.Default.limitedParallelism(1)) { withTimeout(...) { ... } }` in +Kotlin), or use the framework's escape hatch for real time. Define shared helpers +(`awaitState`, `pollUntil`, `withRealTimeout`, ...) that encapsulate this once, and use them for +every wait in integration test bodies. Unit tests are unaffected — there, fake/virtual timers +remain the preferred mechanism. + ### Cleanup with afterEach Always restore mocks in `afterEach`, not just at the end of each test. If a test throws before its cleanup code, the next test inherits dirty state. Use the SDK's mock restoration mechanism (e.g. `restoreAll()`) in an `afterEach` hook. diff --git a/uts/docs/writing-test-specs.md b/uts/docs/writing-test-specs.md index 4abce6575..bf920b6f1 100644 --- a/uts/docs/writing-test-specs.md +++ b/uts/docs/writing-test-specs.md @@ -587,6 +587,10 @@ This means implementations should: - Otherwise wait for state change events with timeout - Fail if timeout expires +In **integration and proxy** specs these timeouts are **wall-clock time** (the test waits on a +real server — see the *Timeout Strategy* section in `docs/integration-testing.md`); in unit specs they may be +realised with fake/virtual timers. + ## Timer Mocking Tests verifying timeout behavior should use timer mocking where practical to avoid slow tests. diff --git a/uts/objects/PLAN.md b/uts/objects/PLAN.md index a95734829..eb2002d34 100644 --- a/uts/objects/PLAN.md +++ b/uts/objects/PLAN.md @@ -61,7 +61,7 @@ All new test files go in `specification/uts/objects/`. ### Proxy Integration Tests | File | Spec Points | ~Tests | |------|-------------|--------| -| `integration/proxy/objects_faults.md` | RTO5a2, RTO7, RTO8, RTO17, RTO20e (sync interruption, mutation buffering during re-sync, server-initiated detach, publish failure on FAILED channel, publish during delayed sync) | ~5 | +| `integration/proxy/objects_faults.md` | RTO5a2, RTO7, RTO8, RTO17, RTO20e (sync interruption, mutation buffering during re-sync, server-initiated detach, in-flight publish failing when channel enters FAILED during sync wait, publish during delayed sync) | ~5 | **Totals: ~20 files, ~310 tests** diff --git a/uts/objects/integration/proxy/objects_faults.md b/uts/objects/integration/proxy/objects_faults.md index adfb709e8..511631773 100644 --- a/uts/objects/integration/proxy/objects_faults.md +++ b/uts/objects/integration/proxy/objects_faults.md @@ -1,6 +1,6 @@ # Objects Proxy Integration Tests -Spec points: `RTO5a2`, `RTO7`, `RTO8`, `RTO17`, `RTO20e` +Spec points: `RTO5a2`, `RTO7`, `RTO8`, `RTO17`, `RTO20e`, `RTO20e1` ## Test Type @@ -13,7 +13,9 @@ See `realtime/integration/helpers/proxy.md` for the full proxy infrastructure sp ## Corresponding Unit Tests - `objects/unit/objects_pool.md` — RTO5a2 (new sync discards old), RTO7/RTO8 (buffering during SYNCING) -- `objects/unit/realtime_object.md` — RTO17 (sync state events), RTO20e (publishAndApply waits for SYNCED/fails on FAILED) +- `objects/unit/realtime_object.md` — RTO17 (sync state events), RTO20e (publishAndApply waits + for SYNCED), RTO20e1 (in-flight operation fails with 92008 when the channel leaves the + attached state during the sync wait) ## Sandbox Setup @@ -39,10 +41,11 @@ AFTER ALL TESTS: ```pseudo AFTER EACH TEST: - IF client IS NOT null AND client.connection.state IN [connected, connecting, disconnected]: - client.connection.close() - AWAIT_STATE client.connection.state == ConnectionState.closed - WITH timeout: 10 seconds + FOR EACH client created by the test (client / client_a / client_b): + IF client.connection.state IN [connected, connecting, disconnected]: + client.connection.close() + AWAIT_STATE client.connection.state == ConnectionState.closed + WITH timeout: 10 seconds IF session IS NOT null: session.close() ``` @@ -51,11 +54,16 @@ AFTER EACH TEST: | Name | Number | |------|--------| +| ACK | 1 | +| ERROR | 9 | | ATTACHED | 11 | | DETACHED | 13 | | OBJECT | 19 | | OBJECT_SYNC | 20 | +> Rule `match.action` values are **strings**; objects actions must use numeric strings +> (e.g. `"20"`) — see the *Match Conditions* section in `realtime/integration/helpers/proxy.md`. + --- ## RTO5a2, RTO17 - Sync interrupted by disconnect, re-syncs on reconnect @@ -82,7 +90,7 @@ session = create_proxy_session( endpoint: "nonprod:sandbox", rules: [{ - "match": { "type": "ws_frame_to_client", "action": 20 }, + "match": { "type": "ws_frame_to_client", "action": "20" }, "action": { "type": "disconnect" }, "times": 1, "comment": "RTO5a2: Disconnect after first OBJECT_SYNC to interrupt sync" @@ -156,6 +164,7 @@ AWAIT_STATE client_a.connection.state == CONNECTED channel_a = client_a.channels.get(channel_name, { modes: ["OBJECT_SUBSCRIBE", "OBJECT_PUBLISH"] }) root_a = AWAIT channel_a.object.get() + WITH timeout: 15 seconds // Set initial data AWAIT root_a.set("key1", "initial") @@ -305,11 +314,19 @@ ASSERT root.get("before_detach").value() == "hello" | Spec | Requirement | |------|-------------| -| RTO20e | publishAndApply waits for SYNCED; fails with 92008 if channel enters DETACHED/SUSPENDED/FAILED | +| RTO20e | publishAndApply waits for the sync state to transition to SYNCED before applying locally | +| RTO20e1 | If the channel enters DETACHED/SUSPENDED/FAILED while waiting, the operation fails with 92008, statusCode 400, cause = channel errorReason | + +The client syncs a channel, is forced back into SYNCING, and issues a mutation +*while* SYNCING — the publish and its ACK complete against the real server, then +publishAndApply parks in the RTO20e wait for SYNCED. The proxy then injects a +channel ERROR so the channel enters FAILED whilst the operation is waiting; the +pending mutation must fail with error 92008 (RTO20e1). -Client sets up a channel with objects, then the proxy injects a channel ERROR -to transition to FAILED. A PathObject mutation (which uses publishAndApply -internally) should fail with error 92008. +> Note: the mutation must be in flight *before* the channel fails. A mutation issued on a +> channel already in DETACHED/FAILED/SUSPENDED fails the RTO26b write precondition with 90001 +> and never reaches publishAndApply — that is different behaviour, not this test. The unit-tier +> test `objects/unit/RTO20e1/fails-on-channel-failed-0` uses the same sequence. ### Setup @@ -344,7 +361,31 @@ AWAIT_STATE client.connection.state == CONNECTED root = AWAIT channel.object.get() WITH timeout: 15 seconds -// Inject channel ERROR to transition to FAILED +// Force the objects back into SYNCING: inject an ATTACHED (action 11) carrying the +// HAS_OBJECTS flag (bit 7, i.e. flags: 128). RTO4c starts a new sync sequence on every +// ATTACHED protocol message; the server never sent this ATTACHED, so no OBJECT_SYNC +// follows and the objects remain SYNCING. The channel itself stays ATTACHED. +session.trigger_action({ + type: "inject_to_client", + message: { action: 11, channel: channel_name, flags: 128 } +}) + +// Mutate WHILE SYNCING: the channel is ATTACHED so the write preconditions (RTO26) +// pass and the publish + ACK complete against the real server; publishAndApply then +// waits for a SYNCED that will never arrive (RTO20e). Do not await yet. +pending = root.set("key", "value") + +// Ensure the operation is in the RTO20e sync-wait, not still publishing: wait until +// the proxy log shows the server's ACK (action 1) for the OBJECT publish, then allow +// a brief real-time yield for the client to move the ACKed operation into the wait. +// (There is no observable client state between "ACK processed" and "parked in the +// sync-wait" to poll on, so a small fixed yield is required — the deriving SDK may +// substitute an equivalent scheduler yield.) +poll_until(session.get_log() CONTAINS event WHERE type == "ws_frame" + AND direction == "server_to_client" AND message.action == 1, timeout: 10s) +WAIT 500ms // real (wall-clock) time + +// The channel enters FAILED whilst the operation waits for SYNCED (RTO20e1) session.trigger_action({ type: "inject_to_client", message: { @@ -357,14 +398,16 @@ session.trigger_action({ AWAIT_STATE channel.state == ChannelState.failed WITH timeout: 15 seconds -// Attempt a mutation — should fail since channel is FAILED -AWAIT root.set("key", "value") FAILS WITH error +AWAIT pending FAILS WITH error + WITH timeout: 15 seconds ``` ### Assertions ```pseudo ASSERT error.code == 92008 +ASSERT error.statusCode == 400 +// RTO20e1: cause is set to RealtimeChannel.errorReason — the injected channel ERROR ASSERT error.cause IS NOT null ASSERT error.cause.code == 90000 ``` @@ -398,6 +441,7 @@ AWAIT_STATE client_a.connection.state == CONNECTED channel_a = client_a.channels.get(channel_name, { modes: ["OBJECT_SUBSCRIBE", "OBJECT_PUBLISH"] }) root_a = AWAIT channel_a.object.get() + WITH timeout: 15 seconds // Set up initial data AWAIT root_a.set("existing", "before") @@ -407,7 +451,7 @@ session = create_proxy_session( endpoint: "nonprod:sandbox", rules: [{ - "match": { "type": "ws_frame_to_client", "action": 20 }, + "match": { "type": "ws_frame_to_client", "action": "20" }, "action": { "type": "delay", "delayMs": 3000 }, "times": 1, "comment": "Delay first OBJECT_SYNC to keep B in SYNCING state" diff --git a/uts/realtime/integration/helpers/proxy.md b/uts/realtime/integration/helpers/proxy.md index 786929604..282befe23 100644 --- a/uts/realtime/integration/helpers/proxy.md +++ b/uts/realtime/integration/helpers/proxy.md @@ -112,6 +112,14 @@ Rules are evaluated in order. First matching rule wins. Unmatched traffic passes **`count`**: 1-based occurrence counter. `count: 2` matches only the 2nd occurrence. +**`action`** (frame matches): must be a **string** — either a protocol action *name* +(e.g. `"ATTACHED"`) or a *numeric string* (e.g. `"20"`). A JSON number is rejected at +session creation with HTTP 400 (`cannot unmarshal number into Go struct field +MatchConfig.match.action of type string`). Note: uts-proxy v0.3.0 resolves action +*names* only up to `AUTH` (17) — an unresolvable name (e.g. `"OBJECT_SYNC"`) makes the +rule silently never match, so use numeric strings for `OBJECT` (`"19"`), +`OBJECT_SYNC` (`"20"`) and `ANNOTATION` (`"21"`) until the proxy's name table is extended. + ### Actions ```json @@ -233,3 +241,6 @@ ClientOptions( 7. Timeouts are generous (10-30s) since real network is involved 8. Each test file provisions a sandbox app in `BEFORE ALL TESTS` and cleans up in `AFTER ALL TESTS` 9. Each test creates its own proxy session and cleans it up after +10. All `WITH timeout` / `poll_until` / `WAIT` durations are **wall-clock (real) time** — see + the *Integration timeouts are wall-clock* section in `docs/writing-derived-tests.md` for + the virtual-time trap in derived tests diff --git a/uts/realtime/integration/proxy/connection_resume.md b/uts/realtime/integration/proxy/connection_resume.md index 1621137a1..9dcdb8dab 100644 --- a/uts/realtime/integration/proxy/connection_resume.md +++ b/uts/realtime/integration/proxy/connection_resume.md @@ -815,7 +815,7 @@ session = create_proxy_session( }, { "match": { "type": "ws_connect", "count": 2 }, - "action": { "type": "refuse" }, + "action": { "type": "refuse_connection" }, "times": 1, "comment": "RTN15g: Refuse 2nd ws_connect — keeps client disconnected until TTL expires and SUSPENDED fires" } From 3dbee55c44b1e5a2e723550a4fcb48e99f62e52b Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Sat, 4 Jul 2026 13:43:13 +0530 Subject: [PATCH 2/4] docs(uts): move proxy infrastructure spec to uts/docs/proxy.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit proxy.md is cross-module infrastructure documentation, not a realtime helper: it is consumed by the realtime (7 proxy specs), rest (rest_fallback.md) and objects (objects_faults.md) modules plus both process docs, unlike the genuinely module-local helper specs (mock_websocket.md, mock_http.md, standard_test_pool.md) which are each used only by their own module. Its old location forced cross-module reach-ins and implied it was realtime-specific. - git mv uts/realtime/integration/helpers/proxy.md -> uts/docs/proxy.md, leaving a one-line stub at the old path so external deep links keep resolving - update all 19 references across 12 files, normalising the pre-existing stale `uts/test/` prefix carried by the realtime proxy specs' references and replacing two mentions of the long-gone in-repo `uts/test/proxy/` directory with a link to the ably/uts-proxy repo - update the directory-tree diagrams in uts/README.md, docs/integration-testing.md and docs/writing-test-specs.md - adjust proxy.md's own cross-reference to sibling style (writing-derived-tests.md) No content changes to the spec itself beyond that one internal reference. Note: ~80 unit/proxy spec files still carry stale `uts/test/...` legacy paths unrelated to proxy.md — left for a separate cleanup PR. --- uts/README.md | 5 ++--- uts/docs/integration-testing.md | 4 +--- uts/{realtime/integration/helpers => docs}/proxy.md | 2 +- uts/docs/writing-test-specs.md | 8 +++----- uts/objects/integration/proxy/objects_faults.md | 4 ++-- uts/realtime/integration/proxy/auth_reauth.md | 2 +- uts/realtime/integration/proxy/channel_faults.md | 2 +- .../integration/proxy/connection_open_failures.md | 2 +- uts/realtime/integration/proxy/connection_resume.md | 2 +- uts/realtime/integration/proxy/heartbeat.md | 2 +- uts/realtime/integration/proxy/presence_reentry.md | 2 +- uts/realtime/integration/proxy/rest_faults.md | 2 +- uts/rest/integration/proxy/rest_fallback.md | 2 +- 13 files changed, 17 insertions(+), 22 deletions(-) rename uts/{realtime/integration/helpers => docs}/proxy.md (98%) diff --git a/uts/README.md b/uts/README.md index 50440d3f7..e28e697fa 100644 --- a/uts/README.md +++ b/uts/README.md @@ -28,14 +28,13 @@ uts/ │ │ ├── connection/ # RTN — connection management │ │ └── presence/ # RTP — realtime presence │ └── integration/ # Realtime integration tests -│ ├── helpers/ -│ │ └── proxy.md # Proxy infrastructure spec │ ├── proxy/ # Proxy-based fault injection tests │ └── *.md # Direct sandbox tests ├── docs/ # Guides and reference │ ├── writing-test-specs.md # How to write UTS specs │ ├── writing-derived-tests.md # How to translate specs into SDK tests │ ├── integration-testing.md # Integration testing policy +│ ├── proxy.md # Proxy infrastructure spec (cross-module) │ └── completion-status.md # Spec coverage matrix └── README.md # This file ``` @@ -101,4 +100,4 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo The programmable proxy for integration testing lives in a separate repository: [ably/uts-proxy](https://github.com/ably/uts-proxy). It sits between the SDK and the Ably sandbox, transparently forwarding traffic while allowing rule-based fault injection. -See `realtime/integration/helpers/proxy.md` for the proxy infrastructure specification used by test specs in this repository. +See `docs/proxy.md` for the proxy infrastructure specification used by test specs in this repository. diff --git a/uts/docs/integration-testing.md b/uts/docs/integration-testing.md index 24e90064e..c88002876 100644 --- a/uts/docs/integration-testing.md +++ b/uts/docs/integration-testing.md @@ -99,8 +99,6 @@ realtime/ presence/ presence_lifecycle_test.md ... - helpers/ - proxy.md # Proxy infrastructure spec proxy/ # Proxy-based tests (sandbox + proxy) connection_open_failures.md connection_resume.md @@ -151,7 +149,7 @@ AFTER ALL TESTS: ### Proxy Setup (integration-proxy only) -Proxy tests additionally set up a proxy session per test or group of tests. See `realtime/integration/helpers/proxy.md` for the proxy infrastructure API. +Proxy tests additionally set up a proxy session per test or group of tests. See `docs/proxy.md` for the proxy infrastructure API. ```pseudo BEFORE EACH TEST: diff --git a/uts/realtime/integration/helpers/proxy.md b/uts/docs/proxy.md similarity index 98% rename from uts/realtime/integration/helpers/proxy.md rename to uts/docs/proxy.md index 282befe23..c52f57cdb 100644 --- a/uts/realtime/integration/helpers/proxy.md +++ b/uts/docs/proxy.md @@ -242,5 +242,5 @@ ClientOptions( 8. Each test file provisions a sandbox app in `BEFORE ALL TESTS` and cleans up in `AFTER ALL TESTS` 9. Each test creates its own proxy session and cleans it up after 10. All `WITH timeout` / `poll_until` / `WAIT` durations are **wall-clock (real) time** — see - the *Integration timeouts are wall-clock* section in `docs/writing-derived-tests.md` for + the *Integration timeouts are wall-clock* section in `writing-derived-tests.md` for the virtual-time trap in derived tests diff --git a/uts/docs/writing-test-specs.md b/uts/docs/writing-test-specs.md index bf920b6f1..715898584 100644 --- a/uts/docs/writing-test-specs.md +++ b/uts/docs/writing-test-specs.md @@ -21,7 +21,7 @@ This guide provides comprehensive guidance for writing portable test specificati - Run against Ably Sandbox through a programmable proxy ([ably/uts-proxy](https://github.com/ably/uts-proxy)) - Proxy transparently forwards traffic but can inject faults via rules - Use for testing fault behaviour: connection failures, token renewal under errors, heartbeat starvation, channel error injection -- See `realtime/integration/helpers/proxy.md` for the full proxy infrastructure spec +- See `docs/proxy.md` for the full proxy infrastructure spec ## Test IDs @@ -296,7 +296,7 @@ mock_ws.active_connection.simulate_disconnect() ## Proxy Integration Tests -For detailed proxy infrastructure documentation, see `realtime/integration/helpers/proxy.md`. +For detailed proxy infrastructure documentation, see `docs/proxy.md`. ### When to Use Proxy Tests @@ -317,7 +317,7 @@ Spec points: `RTN14a`, `RTN14b`, ... Proxy integration test against Ably Sandbox endpoint ## Proxy Infrastructure -See `realtime/integration/helpers/proxy.md` for proxy infrastructure specification. +See `docs/proxy.md` for proxy infrastructure specification. ## Corresponding Unit Tests - `realtime/unit/connection/connection_failures_test.md` — RTN15a, RTN15b @@ -1029,8 +1029,6 @@ realtime/ connection_open_failures_test.md ... integration/ - helpers/ - proxy.md # Proxy infrastructure spec proxy/ connection_open_failures.md # RTN14 tests via proxy connection_resume.md # RTN15 tests via proxy diff --git a/uts/objects/integration/proxy/objects_faults.md b/uts/objects/integration/proxy/objects_faults.md index 511631773..7ccbfee35 100644 --- a/uts/objects/integration/proxy/objects_faults.md +++ b/uts/objects/integration/proxy/objects_faults.md @@ -8,7 +8,7 @@ Proxy integration test against Ably Sandbox endpoint ## Proxy Infrastructure -See `realtime/integration/helpers/proxy.md` for the full proxy infrastructure specification. +See `docs/proxy.md` for the full proxy infrastructure specification. ## Corresponding Unit Tests @@ -62,7 +62,7 @@ AFTER EACH TEST: | OBJECT_SYNC | 20 | > Rule `match.action` values are **strings**; objects actions must use numeric strings -> (e.g. `"20"`) — see the *Match Conditions* section in `realtime/integration/helpers/proxy.md`. +> (e.g. `"20"`) — see the *Match Conditions* section in `docs/proxy.md`. --- diff --git a/uts/realtime/integration/proxy/auth_reauth.md b/uts/realtime/integration/proxy/auth_reauth.md index edb4d9894..e490e587d 100644 --- a/uts/realtime/integration/proxy/auth_reauth.md +++ b/uts/realtime/integration/proxy/auth_reauth.md @@ -6,7 +6,7 @@ Spec points: `RTN22`, `RTC8a` Proxy integration test against Ably Sandbox endpoint. -Uses the programmable proxy (`uts/test/proxy/`) to inject transport-level faults while the SDK communicates with the real Ably backend. See `uts/test/realtime/integration/helpers/proxy.md` for proxy infrastructure details. +Uses the programmable proxy ([ably/uts-proxy](https://github.com/ably/uts-proxy)) to inject transport-level faults while the SDK communicates with the real Ably backend. See `uts/docs/proxy.md` for proxy infrastructure details. Corresponding unit tests: - `uts/test/realtime/unit/connection/server_initiated_reauth_test.md` (RTN22, RTN22a) diff --git a/uts/realtime/integration/proxy/channel_faults.md b/uts/realtime/integration/proxy/channel_faults.md index 35b2da0fd..02063fa63 100644 --- a/uts/realtime/integration/proxy/channel_faults.md +++ b/uts/realtime/integration/proxy/channel_faults.md @@ -8,7 +8,7 @@ Proxy integration test against Ably Sandbox endpoint ## Proxy Infrastructure -See `uts/test/realtime/integration/helpers/proxy.md` for the full proxy infrastructure specification. +See `uts/docs/proxy.md` for the full proxy infrastructure specification. ## Corresponding Unit Tests diff --git a/uts/realtime/integration/proxy/connection_open_failures.md b/uts/realtime/integration/proxy/connection_open_failures.md index 0c79673db..d2ca0ee7a 100644 --- a/uts/realtime/integration/proxy/connection_open_failures.md +++ b/uts/realtime/integration/proxy/connection_open_failures.md @@ -8,7 +8,7 @@ Proxy integration test against Ably Sandbox endpoint ## Proxy Infrastructure -See `uts/test/realtime/integration/helpers/proxy.md` for the full proxy infrastructure specification. +See `uts/docs/proxy.md` for the full proxy infrastructure specification. ## Related Unit Tests diff --git a/uts/realtime/integration/proxy/connection_resume.md b/uts/realtime/integration/proxy/connection_resume.md index 9dcdb8dab..a72d10e55 100644 --- a/uts/realtime/integration/proxy/connection_resume.md +++ b/uts/realtime/integration/proxy/connection_resume.md @@ -6,7 +6,7 @@ Spec points: `RTN15a`, `RTN15b`, `RTN15c6`, `RTN15c7`, `RTN15g`, `RTN15g2`, `RTN Proxy integration test against Ably Sandbox endpoint. -Uses the programmable proxy (`uts/test/proxy/`) to inject transport-level faults while the SDK communicates with the real Ably backend. See `uts/test/realtime/integration/helpers/proxy.md` for proxy infrastructure details. +Uses the programmable proxy ([ably/uts-proxy](https://github.com/ably/uts-proxy)) to inject transport-level faults while the SDK communicates with the real Ably backend. See `uts/docs/proxy.md` for proxy infrastructure details. Corresponding unit tests: `uts/test/realtime/unit/connection/connection_failures_test.md`, `uts/test/realtime/unit/connection/connection_recovery_test.md` diff --git a/uts/realtime/integration/proxy/heartbeat.md b/uts/realtime/integration/proxy/heartbeat.md index e213436b7..8c4701a79 100644 --- a/uts/realtime/integration/proxy/heartbeat.md +++ b/uts/realtime/integration/proxy/heartbeat.md @@ -8,7 +8,7 @@ Proxy integration test against Ably Sandbox endpoint ## Proxy Infrastructure -See `uts/test/realtime/integration/helpers/proxy.md` for the full proxy infrastructure specification. +See `uts/docs/proxy.md` for the full proxy infrastructure specification. ## Related Unit Tests diff --git a/uts/realtime/integration/proxy/presence_reentry.md b/uts/realtime/integration/proxy/presence_reentry.md index 0e3d6bd91..10e7dcb6d 100644 --- a/uts/realtime/integration/proxy/presence_reentry.md +++ b/uts/realtime/integration/proxy/presence_reentry.md @@ -8,7 +8,7 @@ Proxy integration test against Ably Sandbox endpoint ## Proxy Infrastructure -See `uts/test/realtime/integration/helpers/proxy.md` for the full proxy infrastructure specification. +See `uts/docs/proxy.md` for the full proxy infrastructure specification. ## Corresponding Unit Tests diff --git a/uts/realtime/integration/proxy/rest_faults.md b/uts/realtime/integration/proxy/rest_faults.md index e93ce3cf9..a1d6ea8cc 100644 --- a/uts/realtime/integration/proxy/rest_faults.md +++ b/uts/realtime/integration/proxy/rest_faults.md @@ -8,7 +8,7 @@ Proxy integration test against Ably Sandbox endpoint ## Proxy Infrastructure -See `uts/test/realtime/integration/helpers/proxy.md` for the full proxy infrastructure specification. +See `uts/docs/proxy.md` for the full proxy infrastructure specification. ## Corresponding Unit Tests diff --git a/uts/rest/integration/proxy/rest_fallback.md b/uts/rest/integration/proxy/rest_fallback.md index 51ffc5261..d774120ee 100644 --- a/uts/rest/integration/proxy/rest_fallback.md +++ b/uts/rest/integration/proxy/rest_fallback.md @@ -8,7 +8,7 @@ Proxy integration test against Ably Sandbox endpoint ## Proxy Infrastructure -See `uts/realtime/integration/helpers/proxy.md` for the full proxy infrastructure specification. +See `uts/docs/proxy.md` for the full proxy infrastructure specification. ## Corresponding Unit Tests From 9cf3cfab0568240e86232a9b24ae5826d7e34aa0 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Mon, 6 Jul 2026 23:21:34 +0530 Subject: [PATCH 3/4] Added missing objects_gc_test.md for garbage collecting tombstoned objects --- uts/objects/PLAN.md | 4 +- uts/objects/integration/objects_gc_test.md | 166 +++++++++++++++++++ uts/objects/integration/objects_sync_test.md | 2 +- 3 files changed, 169 insertions(+), 3 deletions(-) create mode 100644 uts/objects/integration/objects_gc_test.md diff --git a/uts/objects/PLAN.md b/uts/objects/PLAN.md index eb2002d34..8deb546a1 100644 --- a/uts/objects/PLAN.md +++ b/uts/objects/PLAN.md @@ -56,7 +56,7 @@ All new test files go in `specification/uts/objects/`. | `integration/objects_lifecycle_test.md` | RTO23, RTPO15, RTPO17 (create objects, mutate via PathObject, read back, REST provisioning) | ~6 | | `integration/objects_sync_test.md` | RTO4, RTO5, RTO17 (attach, sync sequence, re-attach) | ~4 | | ~~`integration/objects_batch_test.md`~~ | ~~Batch API not in current spec revision~~ | — | -| `integration/objects_gc_test.md` | RTO10, RTLM19 (behavioral GC verification with ADVANCE_TIME) | ~2 | +| `integration/objects_gc_test.md` | RTO10, RTLM19 (observable tombstone semantics: undefined/null reads, re-create with new objectId; the timer-based GC sweep is unit-tier) | ~2 | ### Proxy Integration Tests | File | Spec Points | ~Tests | @@ -380,6 +380,6 @@ onMessageFromClient: (msg) => { | Batch API deferred | Not included in current spec revision (a397e34); may be added in a future spec update | | LiveObject/LiveMap/LiveCounter marked internal but still unit-tested | Direct testing of CRDT logic is essential; public API tests can't cover all edge cases | | Test IDs use `objects/unit/` prefix | Matches directory structure, not nested under `realtime/` | -| Behavioral GC testing via ADVANCE_TIME | Verify GC through observable consequences (value becomes null, object recreatable) rather than internal pool state inspection | +| Behavioral GC testing tiered by clock control | Unit tier verifies the timer-based sweep with fake timers + ADVANCE_TIME; the integration tier verifies observable consequences only (value becomes undefined/null, object recreatable) since integration tests run on wall-clock time | | Table-driven tests for input validation | Use FOR loops over scenario arrays (like ably-js forScenarios) to test all invalid/valid type combinations | | Bytes data type coverage | Standard test pool includes "avatar" bytes entry; compact/compactJson/value tests verify base64 encoding | diff --git a/uts/objects/integration/objects_gc_test.md b/uts/objects/integration/objects_gc_test.md new file mode 100644 index 000000000..8bda00a4f --- /dev/null +++ b/uts/objects/integration/objects_gc_test.md @@ -0,0 +1,166 @@ +# Objects GC Integration Tests + +Spec points: `RTO10`, `RTLM19`, `RTLM5d2h`, `RTLM7` + +## Test Type +Integration test against Ably sandbox + +## Protocol Variants +json, msgpack + +Each test in this file runs once per protocol variant. The `PROTOCOL` variable +is set to `"json"` or `"msgpack"` for the current run. Client options should set +`useBinaryProtocol: PROTOCOL == "msgpack"`. + +## Purpose + +Behavioral verification of tombstone semantics end-to-end against the real +server: removing a map entry tombstones it (RTLM7), tombstoned entries read +back as undefined/null (RTLM5d2h), and the same key is recreatable — the +server assigns a fresh objectId to the replacement object, which is safe +because tombstoned state is retained for the GC grace period (RTO10). + +## Scope + +The timer-based GC sweep itself (RTO10a–RTO10c, RTLM19a) is verified at the +**unit tier** (`objects/unit/realtime_object.md`, RTO10 tests), where the +clock and the sweep interval are controllable. It is intentionally **not** +exercised here: integration tests run on wall-clock time against a real +server (see *Integration timeouts are wall-clock* in +`docs/writing-derived-tests.md`), and the sweep cadence (RTO10a, ~5 minutes) +combined with the server-provided grace period (RTO10b, default 24 hours per +RTO10b3) is not observable within test timeouts. Do not use `ADVANCE_TIME` +in this file. + +> Note: assertions of the form `value() == null` denote the spec's +> undefined/null absent value (RTLM5d2h); a deriving SDK asserts its +> language's mapping (e.g. `undefined` in JavaScript). + +## Sandbox Setup + +Tests run against the Ably Sandbox at `https://sandbox.realtime.ably-nonprod.net`. + +### App Provisioning + +```pseudo +BEFORE ALL TESTS: + response = POST https://sandbox.realtime.ably-nonprod.net/apps + WITH body from ably-common/test-resources/test-app-setup.json + + app_config = parse_json(response.body) + api_key = app_config.keys[0].key_str + app_id = app_config.app_id + +AFTER ALL TESTS: + DELETE https://sandbox.realtime.ably-nonprod.net/apps/{app_id} + WITH Authorization: Basic {api_key} +``` + +### Notes +- Each test uses a unique channel name + +--- + +## RTO10 - Tombstoned object is recreatable with new objectId + +**Test ID**: `objects/integration/RTO10/tombstoned-object-gc-recreate-0` + +**Spec requirement:** After an object is tombstoned (removed from its parent +map), a new object can be created at the same map key. The new object gets a +different server-assigned objectId, confirming the old object is gone while +its tombstone is retained for the grace period (RTO10). + +### Setup +```pseudo +channel_name = "objects-gc-object-" + random_id() + +client = Realtime(options: { key: api_key, endpoint: "nonprod:sandbox", autoConnect: false, useBinaryProtocol: PROTOCOL == "msgpack" }) +client.connect() +AWAIT_STATE client.connection.state == CONNECTED + WITH timeout: 15 seconds + +channel = client.channels.get(channel_name, { modes: ["OBJECT_SUBSCRIBE", "OBJECT_PUBLISH"] }) +root = AWAIT channel.object.get() + WITH timeout: 15 seconds +``` + +### Test Steps +```pseudo +// Create a counter +AWAIT root.set("counter", LiveCounter.create(42)) +poll_until(root.get("counter").value() == 42, timeout: 10s) + +counter_id = root.get("counter").instance().id + +// Remove it (tombstones the entry and the object, RTLM7) +AWAIT root.remove("counter") + +// RTLM5d2h: tombstoned entries read back as undefined/null +poll_until(root.get("counter").value() == null, timeout: 10s) + +// Create a new counter at the same key +AWAIT root.set("counter", LiveCounter.create(99)) +poll_until(root.get("counter").value() == 99, timeout: 10s) +``` + +### Assertions +```pseudo +ASSERT root.get("counter").value() == 99 +ASSERT root.get("counter").instance().id != counter_id +``` + +### Teardown +```pseudo +client.close() +``` + +--- + +## RTLM19 - Tombstoned map entry is re-settable + +**Test ID**: `objects/integration/RTLM19/tombstoned-entry-gc-reset-0` + +**Spec requirement:** After a map entry is tombstoned (removed, RTLM7), the +entry can be re-set. The subsequent MAP_SET succeeds because the server +assigns a newer serial than the removal's, while the tombstoned entry is +retained for the grace period pending GC (RTLM19). + +### Setup +```pseudo +channel_name = "objects-gc-entry-" + random_id() + +client = Realtime(options: { key: api_key, endpoint: "nonprod:sandbox", autoConnect: false, useBinaryProtocol: PROTOCOL == "msgpack" }) +client.connect() +AWAIT_STATE client.connection.state == CONNECTED + WITH timeout: 15 seconds + +channel = client.channels.get(channel_name, { modes: ["OBJECT_SUBSCRIBE", "OBJECT_PUBLISH"] }) +root = AWAIT channel.object.get() + WITH timeout: 15 seconds +``` + +### Test Steps +```pseudo +// Set then remove a key +AWAIT root.set("ephemeral", "temporary") +poll_until(root.get("ephemeral").value() == "temporary", timeout: 10s) + +AWAIT root.remove("ephemeral") + +// RTLM5d2h: tombstoned entries read back as undefined/null +poll_until(root.get("ephemeral").value() == null, timeout: 10s) + +// Set the same key again +AWAIT root.set("ephemeral", "revived") +poll_until(root.get("ephemeral").value() == "revived", timeout: 10s) +``` + +### Assertions +```pseudo +ASSERT root.get("ephemeral").value() == "revived" +``` + +### Teardown +```pseudo +client.close() +``` diff --git a/uts/objects/integration/objects_sync_test.md b/uts/objects/integration/objects_sync_test.md index eb7513676..ca715fb8e 100644 --- a/uts/objects/integration/objects_sync_test.md +++ b/uts/objects/integration/objects_sync_test.md @@ -172,7 +172,7 @@ client.close() --- -## RTO4 - Attach without OBJECT_SUBSCRIBE still resolves get() with empty pool +## RTO4 - Attach without OBJECT_PUBLISH still resolves get() with empty pool **Test ID**: `objects/integration/RTO4/attach-subscribe-only-0` From f5a8a74ce07e329631d9e1dce61c2fe1ac88aa63 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Wed, 8 Jul 2026 18:11:33 +0530 Subject: [PATCH 4/4] Fixed relative paths pointing to `proxy.md` --- uts/README.md | 2 +- uts/docs/integration-testing.md | 2 +- uts/docs/writing-test-specs.md | 6 +++--- uts/objects/integration/proxy/objects_faults.md | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/uts/README.md b/uts/README.md index e28e697fa..506dfb96e 100644 --- a/uts/README.md +++ b/uts/README.md @@ -100,4 +100,4 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo The programmable proxy for integration testing lives in a separate repository: [ably/uts-proxy](https://github.com/ably/uts-proxy). It sits between the SDK and the Ably sandbox, transparently forwarding traffic while allowing rule-based fault injection. -See `docs/proxy.md` for the proxy infrastructure specification used by test specs in this repository. +See `uts/docs/proxy.md` for the proxy infrastructure specification used by test specs in this repository. diff --git a/uts/docs/integration-testing.md b/uts/docs/integration-testing.md index c88002876..352d66136 100644 --- a/uts/docs/integration-testing.md +++ b/uts/docs/integration-testing.md @@ -149,7 +149,7 @@ AFTER ALL TESTS: ### Proxy Setup (integration-proxy only) -Proxy tests additionally set up a proxy session per test or group of tests. See `docs/proxy.md` for the proxy infrastructure API. +Proxy tests additionally set up a proxy session per test or group of tests. See `uts/docs/proxy.md` for the proxy infrastructure API. ```pseudo BEFORE EACH TEST: diff --git a/uts/docs/writing-test-specs.md b/uts/docs/writing-test-specs.md index 715898584..8c1018266 100644 --- a/uts/docs/writing-test-specs.md +++ b/uts/docs/writing-test-specs.md @@ -21,7 +21,7 @@ This guide provides comprehensive guidance for writing portable test specificati - Run against Ably Sandbox through a programmable proxy ([ably/uts-proxy](https://github.com/ably/uts-proxy)) - Proxy transparently forwards traffic but can inject faults via rules - Use for testing fault behaviour: connection failures, token renewal under errors, heartbeat starvation, channel error injection -- See `docs/proxy.md` for the full proxy infrastructure spec +- See `uts/docs/proxy.md` for the full proxy infrastructure spec ## Test IDs @@ -296,7 +296,7 @@ mock_ws.active_connection.simulate_disconnect() ## Proxy Integration Tests -For detailed proxy infrastructure documentation, see `docs/proxy.md`. +For detailed proxy infrastructure documentation, see `uts/docs/proxy.md`. ### When to Use Proxy Tests @@ -317,7 +317,7 @@ Spec points: `RTN14a`, `RTN14b`, ... Proxy integration test against Ably Sandbox endpoint ## Proxy Infrastructure -See `docs/proxy.md` for proxy infrastructure specification. +See `uts/docs/proxy.md` for proxy infrastructure specification. ## Corresponding Unit Tests - `realtime/unit/connection/connection_failures_test.md` — RTN15a, RTN15b diff --git a/uts/objects/integration/proxy/objects_faults.md b/uts/objects/integration/proxy/objects_faults.md index 7ccbfee35..668d7a2fc 100644 --- a/uts/objects/integration/proxy/objects_faults.md +++ b/uts/objects/integration/proxy/objects_faults.md @@ -8,7 +8,7 @@ Proxy integration test against Ably Sandbox endpoint ## Proxy Infrastructure -See `docs/proxy.md` for the full proxy infrastructure specification. +See `uts/docs/proxy.md` for the full proxy infrastructure specification. ## Corresponding Unit Tests @@ -62,7 +62,7 @@ AFTER EACH TEST: | OBJECT_SYNC | 20 | > Rule `match.action` values are **strings**; objects actions must use numeric strings -> (e.g. `"20"`) — see the *Match Conditions* section in `docs/proxy.md`. +> (e.g. `"20"`) — see the *Match Conditions* section in `uts/docs/proxy.md`. ---