Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 66 additions & 17 deletions architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -398,11 +398,43 @@ provider occurs before it constructs the durable effect. Replay, including the
replayed prefix of a partial run, bypasses the coordinator, execution and live
publication. Callback-based durable effects keep their existing path.

The shared Workspace durable-operation wrapper selects a contextual Workspace
coordinator explicitly. Its default fails before execution or publication, so
installing no provider cannot leak a mutation outside its transaction. Selecting
Workspace coordination for one operation does not enlist unrelated durable
operations in the same scope.
The shared Workspace durable-operation wrapper reads a contextual provider
selection explicitly. That replaceable API carries only a non-operational
selection identity: execution, publication, failure activation and durable
publication identity never pass through that middleware. Provider selection
creates a one-use route and a separate opaque credential. The live call keeps
that credential out of its contextual invocation request and places the exact
executor, publisher and failure activator behind an execution-owned capability
that accepts only the credential. A provider installed by any loaded package
copy terminally consumes the route and invokes that capability directly. It
does not send inspection, execution, publication, failure or completion through
the contextual continuation. Enclosing middleware at either priority can
therefore route or refuse the initial request but cannot observe or acknowledge
an operational phase. No module registry or replaceable structural value
resolves provider or invocation authority.

The execution-owned capability accepts execution and publication once and
records the exact `Result` only after the provider returns it from the completed
publication. The live operation ignores the contextual call's response as
completion evidence and resumes from that recorded result alone. A short
circuit or altered response therefore activates fail-stop before resume, while
an exception or substituted response after authoritative completion cannot
replace the published result. Foreign or substituted selectors and reused,
completed or stale calls fail before transaction work. Installing no provider
cannot leak execution or publication, and selecting Workspace coordination for
one operation does not enlist unrelated durable operations in the same scope.

The Deno adapter retains the canonical module's non-operational identity for
each WorkflowRun journal. The existing secret guard privately associates only
its own wrapper, including nested official wrappers, with that same identity.
Before it opens a transaction, the adapter requires both the exact proof
executor and the publication identity from the consumed invocation to belong to
the selected WorkflowRun. An in-memory stream, another run's journal, a copied
property, or an unproven wrapper is refused before mutation or publication.
Inside the transaction, the provider's publication operation calls the
execution-owned publisher directly and returns only after that publisher records
the exact Result. The transaction body cannot return or commit on a contextual
publication acknowledgement.

Every Workspace-local expansion publishes one effect through one effect
transaction:
Expand All @@ -421,9 +453,14 @@ staging, switching and committing, operate on the same transactional Workspace
rather than invoking an untracked native Git side effect.

Successful effect coordination finishes the mutation scope, including child
cleanup, before capturing the resulting root. The provider-level coordinator
that performs this ordering and journal publication is not installed at this
layer.
cleanup, before capturing the resulting root. The Deno provider installs that
ordering for its adapter-private Workspace proof operation: the mutation
savepoint, root publication, filtered routed Yield, and caller-owned transaction
commit form one boundary. The proof filesystem uses the pinned synchronous DOFS
entry points for its string and byte-array surface, so cancellation leaves no
eager promise or stream pull able to reach the connection after transaction
authority ends. Public filesystem components and workflow lifecycle commands do
not yet select that operation.

An external provider cannot join that transaction. Prompt, Git push and pull
request effects derive a stable identity from the run and expansion, ask the
Expand Down Expand Up @@ -482,8 +519,9 @@ materialize one privately through the authoritative connection. Capture runs
inside the caller-owned transaction. Restoration runs in a nested savepoint,
clears the authoritative resolution and blob caches, and resnapshots to the
selected identity before release. Private Workspace transaction bodies finish
their child teardown before final live/current validation; a later effect
coordinator finishes its mutation scope before it invokes capture.
their child teardown before final live/current validation. The Workspace effect
coordinator also finishes its mutation savepoint scope before it invokes
capture.

Separately, every unsuccessful caller-owned transaction attempts top-level
SQLite rollback and then
Expand All @@ -496,10 +534,21 @@ after SQLite has restored the prior frontier.
Retained roots, manifests and blobs remain indefinitely. Cloudflare garbage
collection is not in the production closure and is never invoked. The provider
exposes no public Workspace mutation effect, history selection or fork
operation at this layer. Provider-neutral durable coordination and explicit
filtered-journal routing are present, while the Deno Workspace coordinator that
combines mutation, root publication and journal publication atomically is
absent.
operation at this layer. Its adapter-private coordinator combines one mutation,
immutable-root publication and one filtered journal result atomically for the
provider-level proof; declarative `<File>` and workflow start/resume do not
reach it yet.

The coordinator treats only errors produced through its private filesystem
adapter's documented path and mutation refusals as journalable operation
failures. It rolls their savepoint back, keeps the prior current root, and
commits one failed protocol Yield. Connection, savepoint, content, capture,
routing, filtering, insertion, teardown and commit failures roll back the outer
transaction and activate the durable run's first-failure fence. Cancellation is
control flow: it publishes no failed result. The durable coordinator contract
supplies a provider-neutral failure activator so a provider can retain that
first infrastructure failure by identity; the default coordinator does not use
it and keeps its existing behavior.

The Deno journal adapter routes an append ordinarily when no destination is
bound. A publication may instead bind one exact transaction destination for its
Expand Down Expand Up @@ -892,10 +941,10 @@ Status is measured against main.
| `Expansion` / `getExpansion()` | describes the current logical element expansion | built on main |
| `useWorkflow()` / `getWorkflowRun()` | associates one document execution with a workflow run | built on main |
| `Git.revParse()` | verifies and resolves one Git revision expression contextually | built on main |
| workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; Workspace effect publication is unbuilt |
| workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry | built on the #365 stack; public workflow execution is unbuilt |
| caller-owned storage transaction | publishes several changes, including journal events, in one transaction nothing else enlists in | built on main |
| live durable-operation coordinator | explicitly coordinates structured live execution with existing Yield publication while leaving replay and callback effects unchanged | built on the #365 stack |
| Workspace coordination API | fails closed by default and lets a Workspace operation explicitly select provider coordination | built on the #365 stack; the atomic Deno Workspace handler is unbuilt |
| Workspace coordination API | fails closed by default; replaceable context routes only a one-use provider selection, while the selected provider directly invokes an execution-owned credentialed capability for execution, publication and failure activation | built on the #365 stack; the Deno provider installs an adapter-private atomic handler |
| explicit WorkflowRun journal route | binds one already-filtered publication to one exact active transaction and otherwise uses ordinary serialized journal storage | built on the #365 stack |
| `API.Service` / `startService()` | creates an authenticated, supervised loopback service attachment through a provider-neutral operation | built on main |
| `API.Files` | routes every document filesystem operation to the installed provider, with no host default and structural failure data | built on the #227 stack |
Expand All @@ -909,7 +958,7 @@ Status is measured against main.
| Repository / Worktree / transactional Git effects | compose named checkouts and publish local mutations with their journal result | defined in `specs/workflow-workspace-spec.md`, unbuilt |
| workflow inspection and history fork | reads status/history without advancing a run and creates a new run from a checkpoint | defined in `specs/workflow-workspace-spec.md`, unbuilt |
| read-only workflow Agent / generated XMD | lets an Agent inspect a derived view and propose constrained executable changes | defined in `specs/workflow-workspace-spec.md`, unbuilt |
| Deno-local DOFS provider | owns one authoritative SQLite/DOFS connection per run path, captures arbitrary canonical retained roots, and privately restores them with cache-coherent savepoints | built on the #365 stack; public mutation and effect-transaction integration are unbuilt |
| Deno-local DOFS provider | owns one authoritative SQLite/DOFS connection per run path, captures arbitrary canonical retained roots, privately restores them, and atomically coordinates an adapter-private mutation proof with its filtered Yield | built on the #365 stack; public mutation and workflow lifecycle reachability are unbuilt |
| scoped Worker Shell | executes `just-bash` through the Workspace adapter inside a Deno Worker | containment and effect-transaction POCs complete (#351, #357); production integration unbuilt |
| `<Retry max timeout>` | retry a region until it completes | defined, unbuilt |
| suspension effect | suspend durably | defined, unbuilt |
Expand Down
15 changes: 15 additions & 0 deletions packages/durable-streams/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,21 @@ same result after publication completes. Replay bypasses the coordinator,
executor, continuation, and live append; a partially replayed run coordinates
only its live suffix.

The coordinator receives a non-operational publication identity and an
infrastructure-failure activator. The canonical durable-stream module keeps the
association between a claimed backend and that identity in private weak state.
`guardDurableStream` associates only the wrapper it constructs with its source's
identity, including through nested guards. Streams expose no identity property
or inheritance callback; copied properties, custom wrappers and wrappers from
another loaded copy acquire no authority.

A provider
whose execution and publication share a larger durability boundary calls it
when that boundary cannot commit. It returns the first active failure by
identity, so the provider throws that exact failure and later durable work stays
fenced. The default coordinator does not call this continuation: an ordinary
execution failure remains the existing failed protocol `Result`.

The publication continuation uses the ordered append fence described above. A
backing append failure therefore activates the same fail-stop state and raises
`DurablePersistenceError` with the adapter error as its cause. A marked
Expand Down
33 changes: 24 additions & 9 deletions packages/durable-streams/effect.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,9 @@ import {
rememberDurabilityFailure,
} from "./durability.ts";
import { StaleInputError } from "./errors.ts";
import { durablePublicationIdentity } from "./guard.ts";
import {
type ActivateDurabilityFailure,
defaultLiveDurableOperationCoordinator,
type LiveDurableOperationCoordinator,
} from "./live-coordinator.ts";
Expand Down Expand Up @@ -351,15 +353,28 @@ export function createDurableOperation<T extends Json>(

try {
const coordinator = options.coordinator ?? defaultLiveDurableOperationCoordinator;
const result = yield* coordinator.run(execute, function* (published) {
const event: Yield = {
type: "yield",
coroutineId: ctx.coroutineId,
description: desc,
result: published,
};
yield* appendDurableEvent(ctx, event);
});
const activateFailure: ActivateDurabilityFailure = (failure) => {
const existing = activeDurabilityFailure(ctx);
if (existing) {
return existing;
}
const error = failure instanceof Error ? failure : new Error(String(failure));
return rememberDurabilityFailure(ctx, error);
};
const result = yield* coordinator.run(
execute,
function* (published) {
const event: Yield = {
type: "yield",
coroutineId: ctx.coroutineId,
description: desc,
result: published,
};
yield* appendDurableEvent(ctx, event);
},
activateFailure,
durablePublicationIdentity(ctx.stream),
);
resolve(protocolToEffection<T>(result));
} catch (err) {
resolve({
Expand Down
51 changes: 50 additions & 1 deletion packages/durable-streams/guard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,53 @@ import { createContext, type Operation } from "effection";
import type { DurableStream } from "./stream.ts";
import type { DurableEvent } from "./types.ts";

class PublicationIdentity {
#opaque = undefined;
}

/** Non-operational identity for one durable publication backend. */
export type DurablePublicationIdentity = PublicationIdentity;

const publicationIdentities = (() => {
// This security identity is deliberately canonical-module-local. A loaded
// copy cannot enroll its wrappers into this copy's authority.
const identities = new WeakMap<DurableStream, DurablePublicationIdentity>();

return {
claim(stream: DurableStream): DurablePublicationIdentity {
if (identities.has(stream)) {
throw new Error("this durable stream already has a publication identity");
}
const identity = new PublicationIdentity();
identities.set(stream, identity);
return identity;
},

inherit(source: DurableStream, target: DurableStream): void {
const identity = identities.get(source);
if (identity !== undefined) {
identities.set(target, identity);
}
},

get(stream: DurableStream): DurablePublicationIdentity | undefined {
return identities.get(stream);
},
};
})();

/** Claim the exact backend identity retained by a provider. */
export function claimDurablePublicationIdentity(stream: DurableStream): DurablePublicationIdentity {
return publicationIdentities.claim(stream);
}

/** @internal The live durable path reads identity without receiving stream authority. */
export function durablePublicationIdentity(
stream: DurableStream,
): DurablePublicationIdentity | undefined {
return publicationIdentities.get(stream);
}

export interface DurableEventRejectionOccurrence {
rejected: boolean;
error?: unknown;
Expand Down Expand Up @@ -60,7 +107,7 @@ export type DurableEventGate = (event: DurableEvent) => Operation<void>;
* event with an `err` result, and that close crosses the gate on its own.
*/
export function guardDurableStream(stream: DurableStream, gate: DurableEventGate): DurableStream {
return {
const guarded: DurableStream = {
readAll: () => stream.readAll(),

*append(event: DurableEvent): Operation<void> {
Expand All @@ -80,4 +127,6 @@ export function guardDurableStream(stream: DurableStream, gate: DurableEventGate
yield* stream.append(event);
},
};
publicationIdentities.inherit(stream, guarded);
return guarded;
}
8 changes: 8 additions & 0 deletions packages/durable-streams/live-coordinator.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
import type { Operation } from "effection";
import type { DurablePublicationIdentity } from "./guard.ts";
import { serializeError } from "./serialize.ts";
import type { Json, Result } from "./types.ts";

/** Activates the first infrastructure failure for the enclosing durable run. */
export type ActivateDurabilityFailure = (failure: unknown) => Error;

/** Coordinates one live structured durable operation with its publication. */
export interface LiveDurableOperationCoordinator {
run<T extends Json>(
execute: () => Operation<T>,
publish: (result: Result) => Operation<void>,
activateFailure: ActivateDurabilityFailure,
publicationIdentity: DurablePublicationIdentity | undefined,
): Operation<Result>;
}

Expand All @@ -15,6 +21,8 @@ export const defaultLiveDurableOperationCoordinator: LiveDurableOperationCoordin
*run<T extends Json>(
execute: () => Operation<T>,
publish: (result: Result) => Operation<void>,
_activateFailure: ActivateDurabilityFailure,
_publicationIdentity: DurablePublicationIdentity | undefined,
): Operation<Result> {
let result: Result;
try {
Expand Down
8 changes: 6 additions & 2 deletions packages/durable-streams/mod.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ export { InMemoryStream } from "./stream.ts";

// Pre-persistence gate — runs before an event reaches its backend
export { guardDurableStream } from "./guard.ts";
export type { DurableEventGate } from "./guard.ts";
export { claimDurablePublicationIdentity } from "./guard.ts";
export type { DurableEventGate, DurablePublicationIdentity } from "./guard.ts";

// HTTP-backed stream adapter
export { useHttpDurableStream } from "./http-stream.ts";
Expand Down Expand Up @@ -79,7 +80,10 @@ export type { Executor } from "./effect.ts";

// Structured live-operation coordination
export { defaultLiveDurableOperationCoordinator } from "./live-coordinator.ts";
export type { LiveDurableOperationCoordinator } from "./live-coordinator.ts";
export type {
ActivateDurabilityFailure,
LiveDurableOperationCoordinator,
} from "./live-coordinator.ts";

// Workflow-enabled effects
export { durableAction, durableCall, durableSleep, versionCheck } from "./operations.ts";
Expand Down
Loading
Loading