Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
2ab1ec7
Add ably-common submodule
paddybyers Mar 30, 2026
b43ccb3
Add UTS (Universal Test Spec) suite for REST API
paddybyers Apr 30, 2026
df337ce
Add UTS specs for Realtime connection lifecycle
paddybyers Mar 30, 2026
0bfff12
Fix connection test specs to close WebSocket transport when necessary
paddybyers Mar 30, 2026
000d86f
Refactor realtime test specs: extract mock WebSocket helper and add s…
paddybyers Mar 30, 2026
ec93091
Refactor test specs to use EXPECT_THROW instead of TRY/CATCH
paddybyers Mar 30, 2026
ebb5774
Use unique channel names in test specs to prevent cross-test interfer…
paddybyers Mar 30, 2026
cfd7b0f
Rewrite heartbeat test specs and extend mock WebSocket helper
paddybyers Mar 30, 2026
3b386e0
Fix RTN15a (immediate reconnection) test approach and update skill
paddybyers Mar 30, 2026
fa26163
Fix minor issues in channel attach and state events test specs
paddybyers Mar 30, 2026
7bcd995
Add test specs for connection state transitions and channel properties
paddybyers Mar 30, 2026
a8541da
Add test specs for realtime channel subscribe (RTL7/RTL8)
paddybyers Mar 30, 2026
31b573c
Add test specs for realtime channel publish (RTL6)
paddybyers Mar 30, 2026
dac3be8
Add realtime entries for stats() and time() referencing existing REST…
paddybyers Mar 30, 2026
bc4352c
Add test specs for Realtime.request() (RTN18)
paddybyers Mar 30, 2026
7e99171
Extend realtime publish tests: queued messages and state transitions
paddybyers Mar 30, 2026
c3979d5
Add test specs for RSN1-4 (connection recovery) and RTL10 (realtime h…
paddybyers Mar 30, 2026
466f9e2
Add test specs for client logging (LOG1-LOG3)
paddybyers Mar 30, 2026
11693a5
Add remaining auth test specs
paddybyers Mar 30, 2026
2b9d71f
Add test specs for realtime presence (RTP)
paddybyers Mar 30, 2026
929a334
Fix path component encoding in test specs to use encode_uri_component()
paddybyers Mar 30, 2026
229d280
Update presence test specs and add integration tests
paddybyers Mar 30, 2026
ddc6be0
Update write-test-spec skill to emphasise keeping specs in sync
paddybyers Mar 30, 2026
ab5a5d1
Add test specs for batch presence (RSP4)
paddybyers Mar 30, 2026
553b33d
Add test specs for token revocation (RSA10)
paddybyers Mar 30, 2026
baa7e41
Add test specs for RTL12 (channel UPDATE event handling)
paddybyers Mar 30, 2026
70daee5
Add test specs for VCDIFF delta message encoding (RTL18/RTL19)
paddybyers Mar 30, 2026
34da36b
Add test specs for channel attributes, whenState, timeouts, and auto-…
paddybyers Mar 30, 2026
6ef8e66
Add test specs for mutable messages (RTL22/RTL23)
paddybyers Mar 30, 2026
0bc3b57
Add test specs for push admin (RSH1/RSH7)
paddybyers Mar 30, 2026
37ab2eb
Fix presence test specs: server echoes, wildcard clientId, RTL13b, RT…
paddybyers Mar 30, 2026
ff22962
Fix integration test specs based on sandbox behavior
paddybyers Mar 30, 2026
f3887f8
Add CLOSE_CLIENT(client) to all UTS specs that create Realtime clients
paddybyers Apr 30, 2026
f425aac
UTS spec corrections from ably-js audit
paddybyers May 2, 2026
fa8339b
RSC10b: Clarify that non-token 401 errors MUST NOT trigger renewal
paddybyers Mar 30, 2026
716bb40
Add realtime integration test specs and fix REST integration specs
paddybyers May 2, 2026
34f855e
Add test proxy framework for integration testing
paddybyers Mar 30, 2026
b97cb9a
Fix RSL2b3 history time range test to use server timestamps
paddybyers May 3, 2026
9667dc4
Fix mislabeled spec points in proxy integration test specs
paddybyers May 3, 2026
40c284d
Add tier 1+2 proxy integration test specs and proxy writing guidance
paddybyers May 3, 2026
96c223c
Reorganise UTS docs: move guides to docs/, rewrite README
paddybyers May 3, 2026
cfa31de
Add UTS specs for missing test coverage
paddybyers May 3, 2026
b93ea12
Add RSC15f test: expired fallback not resurrected by late in-flight s…
paddybyers May 3, 2026
f5e3ec8
Add RTN15a proxy test variant for TCP close without WebSocket close f…
paddybyers May 4, 2026
733c20c
Remove proxy source code; now lives at ably/uts-proxy
paddybyers May 4, 2026
b799ddf
UTS: updated completion status
paddybyers May 4, 2026
2bc783a
Fix RSA4c3 test: auth failure while CONNECTED should not set errorReason
paddybyers May 5, 2026
77f1154
Fix RTN24 tests: connectionId is not inside connectionDetails
paddybyers May 5, 2026
f33ab7b
Fix TM3/TM4 UTS tests: correct spec point interpretations
paddybyers May 5, 2026
420263a
Fix batch operation response format in UTS specs (RSC24, RSA17c)
paddybyers May 5, 2026
3404b23
Add UTS spec for REST fallback proxy integration tests (RSC15l2, RSC1…
paddybyers May 5, 2026
522499e
Use endpoint instead of deprecated restHost in proxy fallback spec
paddybyers May 6, 2026
97517b8
Add comprehensive REST proxy integration tests for HTTP error handling
paddybyers May 6, 2026
ecdaff7
Add unique Test IDs to all 1120 tests across 126 UTS spec files
paddybyers May 6, 2026
dcc770c
Update RSL1k4 UTS spec: use http_replace_response proxy action
paddybyers May 7, 2026
37fc021
Fix endpoint values in UTS test specs
paddybyers May 7, 2026
9a51430
Update sandbox provisioning URLs to nonprod hostname
paddybyers May 7, 2026
48da79d
Add protocol variant convention to UTS integration specs (G1)
paddybyers May 8, 2026
2e12074
Add RSL6a3 UTS spec for msgpack interoperability fixtures
paddybyers May 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "submodules/ably-common"]
path = submodules/ably-common
url = https://github.com/ably/ably-common.git
1 change: 1 addition & 0 deletions submodules/ably-common
Submodule ably-common added at bf2908
104 changes: 104 additions & 0 deletions uts/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Universal Test Specifications (UTS)

Portable test specifications for Ably client library implementations. Each spec defines what to test in language-neutral pseudocode, which is then translated into runnable tests for each SDK.

## Directory Structure

```
uts/
├── rest/
│ ├── unit/ # REST unit tests (mocked HTTP)
│ │ ├── helpers/
│ │ │ └── mock_http.md # Mock HTTP infrastructure spec
│ │ ├── auth/ # RSA — authentication
│ │ ├── channel/ # RSL — channel operations
│ │ ├── encoding/ # RSL4/RSL6 — message encoding
│ │ ├── presence/ # RSP — REST presence
│ │ ├── push/ # RSH — push admin
│ │ └── types/ # T* — type definitions
│ └── integration/ # REST integration tests (Ably sandbox)
├── realtime/
│ ├── unit/ # Realtime unit tests (mocked WebSocket)
│ │ ├── helpers/
│ │ │ ├── mock_websocket.md # Mock WebSocket infrastructure spec
│ │ │ └── mock_vcdiff.md # Mock VCDiff decoder spec
│ │ ├── auth/ # RSA/RTC8 — realtime auth
│ │ ├── channels/ # RTL/RTS — channels and messages
│ │ ├── client/ # RTC — realtime client
│ │ ├── connection/ # RTN — connection management
│ │ └── presence/ # RTP — realtime presence
│ └── integration/ # Realtime integration tests
│ ├── helpers/
│ │ └── proxy.md # Proxy infrastructure spec
│ ├── proxy/ # Proxy-based fault injection tests
│ └── *.md # Direct sandbox tests
├── docs/ # Guides and reference
│ ├── writing-test-specs.md # How to write UTS specs
│ ├── writing-derived-tests.md # How to translate specs into SDK tests
│ ├── integration-testing.md # Integration testing policy
│ └── completion-status.md # Spec coverage matrix
└── README.md # This file
```

## Spec File Counts

| Category | Count | Description |
|----------|-------|-------------|
| REST unit | 40 | Mocked HTTP client tests |
| REST integration | 11 | Ably sandbox tests |
| Realtime unit | 54 | Mocked WebSocket tests |
| Realtime integration (direct) | 13 | Direct sandbox tests |
| Realtime integration (proxy) | 7 | Fault injection via Go proxy |
| Helper specs | 4 | Mock infrastructure definitions |
| **Total** | **129** | |

## Three Test Tiers

**Unit tests** use mocked transports (MockHttpClient, MockWebSocket) to verify client-side logic: state machines, request formation, response parsing, timer behaviour, error handling. They are fast and deterministic.

**Integration tests** (direct) run against the Ably sandbox to verify that the SDK interoperates correctly with the real service. No fault injection — these test happy-path behaviour and real error responses.

**Integration tests** (proxy) run against the Ably sandbox through a programmable Go proxy that can inject faults: connection drops, suppressed frames, replaced responses, HTTP error injection. These test behaviour that can't be verified without controlling the network path.

## Pseudocode Conventions

Test specs use a consistent pseudocode syntax:

```pseudo
# Setup
mock_http = MockHttpClient(
onConnectionAttempt: (conn) => conn.respond_with_success(),
onRequest: (req) => req.respond_with(200, {"result": "ok"})
)
install_mock(mock_http)
client = Rest(options: ClientOptions(key: "appId.keyId:keySecret"))

# Test
result = AWAIT client.operation()

# Assertions
ASSERT result.field == "ok"
ASSERT request.url.path == "/channels/" + encode_uri_component(name) + "/messages"

# Error testing
AWAIT client.badOperation() FAILS WITH error
ASSERT error.code == 40160

# State transitions
AWAIT_STATE client.connection.state == ConnectionState.connected
```

See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudocode reference, mock patterns, and conventions.

## Guides

- **[Writing Test Specs](docs/writing-test-specs.md)** — How to author UTS specs: mock patterns, pseudocode conventions, proxy test structure, common mistakes
- **[Writing Derived Tests](docs/writing-derived-tests.md)** — How to translate UTS specs into SDK-specific tests, diagnose failures, and record deviations
- **[Integration Testing Policy](docs/integration-testing.md)** — When to write integration vs unit tests, proxy test design principles, test structure conventions
- **[Completion Status](docs/completion-status.md)** — Coverage matrix tracking which spec items have UTS test specs

## Go Test Proxy

The programmable proxy for integration testing lives in a separate repository: [ably/uts-proxy](https://github.com/ably/uts-proxy). It sits between the SDK and the Ably sandbox, transparently forwarding traffic while allowing rule-based fault injection.

See `realtime/integration/helpers/proxy.md` for the proxy infrastructure specification used by test specs in this repository.
Loading
Loading