Spike: host Cloudflare DOFS directly in Deno with SQLite and FUSE

[taras]

Spike

Prove whether Executable.md can host a persistent SQLite Workspace directly in Deno, reuse Cloudflare Computer's filesystem implementation, and expose that filesystem to native subprocesses without running workerd.

This spike compares a Deno-local SQLite/FUSE topology with the bundled-workerd evidence in #347 / PR #348. It informs #346 but does not select a topology or implement <Workspace> from #218.

Source audit already established

• Deno provides node:sqlite directly.
• Cloudflare Computer's repository is MIT licensed.
• packages/dofs is the authoritative SQLite filesystem layer. Its Database accepts a structural DurableObjectStorageLike containing sql.exec() and synchronous transaction support.
• Cloudflare's own Node tests adapt packages/dofs to node:sqlite with SQLiteTestStorage; that adapter currently opens only :memory:.
• SQLiteWorkspaceProvider exposes the same database through @platformatic/vfs and is the provider mounted by computerd.
• packages/computerd contains the FUSE operation adapter, real FUSE mounting, and a userspace materialization/reconciliation shim.
• @cloudflare/dofs and @cloudflare/computerd are private, unpublished workspace packages. Their source can be reused under MIT, but they are not stable npm dependency surfaces.
• Importing the published @cloudflare/computer@0.1.1 entrypoint directly in Deno currently fails because its module graph contains the cloudflare: URL scheme.

The spike must preserve these distinctions: legal reuse is established; supported dependency reuse and Deno portability are not.

Questions to answer

1. Can a file-backed node:sqlite adapter in Deno satisfy Cloudflare DOFS's storage contract without changing its schema or filesystem primitives?
2. Can the runtime-neutral part of the published Computer package be tree-shaken/bundled for Deno, or must XMD pin/vendor packages/dofs source?
3. Can Cloudflare's SQLiteWorkspaceProvider and FUSE driver run in the Deno process through @platformatic/vfs and fuse-native?
4. Can a native subprocess use the mounted directory while reads and writes remain immediately visible through the Workspace filesystem API?
5. Can the compiled XMD artifact carry and materialize every required native addon/library, and what system FUSE installation or privilege remains required?
6. Which filesystem semantics and crash guarantees hold for closed files, flush/fsync, open write buffers, renames, deletes, and abrupt termination?
7. Is Cloudflare's userspace shim a viable portable fallback when FUSE is unavailable, and which coherence guarantees does it lose?
8. What does this topology gain and lose compared with bundled workerd?

Experimental slices

1. Direct persistent DOFS

• Pin the exact Cloudflare Computer source commit and published package versions used by the spike.
• Implement the smallest file-backed DurableObjectStorageLike adapter over Deno's node:sqlite.
• Use Cloudflare's existing Database, schema initialization, and filesystem operations; do not reimplement the filesystem schema.
• Exercise mkdir, writeFile, readFile, readdir, stat, rename, symlink, and rm.
• Close the Deno process, reopen the same database, and prove the exact filesystem frontier survives, including create/delete/create.
• Prove separate database paths and Workspace identities remain isolated.

2. Reuse boundary

Evaluate these approaches independently:

1. direct import from @cloudflare/computer;
2. a tree-shaken bundle that excludes or adapts Cloudflare-runtime-only modules;
3. a pinned vendored copy of packages/dofs; and
4. an upstream package/export change that makes DOFS consumable directly.

For each, record whether it works, how upgrades are consumed, whether MIT notices must be carried, and how much Cloudflare code XMD would own. Do not create an untracked source copy.

3. Native filesystem bridge

• Construct Cloudflare's SQLiteWorkspaceProvider over the same file-backed database.
• Adapt it through @platformatic/vfs exactly as computerd does.
• Attempt to reuse the pinned Cloudflare FUSE operation adapter and mount code under Deno.
• Run a native command through XMD's process capability with its cwd inside the mount.
• Prove API writes are visible to the command and command writes are visible through the API without a push/pull copy step.
• Exercise create, overwrite, positional write, truncate, rename, symlink, delete, directory traversal, and concurrent readers.

4. Packaging and platform requirements

• Compile a proof executable without workerd or an external Node runtime.
• If fuse-native requires a native addon or shared library beside the executable, embed, materialize, digest-check, and load it explicitly.
• Record results for Linux x64/arm64, macOS x64/arm64, and Windows x64 rather than inferring support from package names.
• Name unavoidable host prerequisites: /dev/fuse, mount permissions, macFUSE/kernel approval, or missing Windows support.
• Measure compiled size, extracted assets, cold start, warm start, mount latency, and basic file-operation overhead.

5. Durability and teardown

• Test clean close/reopen, host SIGTERM, host SIGKILL, workload termination, SQLite WAL recovery, and a second host attempting to open the same Workspace.
• Test writes followed by close, flush, and fsync separately from a process killed with an open dirty file.
• Determine when Cloudflare's buffered FUSE writes become committed SQLite state.
• Prove unmount and cleanup behavior for success, failure, and Effection cancellation.
• Treat corrupt or unsupported schema state as an explicit failure; do not silently recreate it.

6. No-FUSE fallback

• Run Cloudflare's userspace shim against the same persistent database.
• Measure its reconciliation latency and demonstrate its documented limitations around conflicts, symlinks, metadata, watches, and large files.
• Decide from evidence whether it is a supported fallback, a development-only fallback, or unsuitable for XMD.

7. Comparison with #347

Compare at least:

• runtime and artifact size;
• startup and attachment latency;
• filesystem and process semantics;
• supported platforms and host prerequisites;
• sandbox/security boundary;
• schema ownership and upgrade path;
• process and mount supervision;
• command streaming, cancellation, and retained execution handles;
• compatibility with hosted Cloudflare Computer; and
• whether Cloudflare's sync protocol provides a future migration/replication path.

Acceptance

• A reproducible proof uses Deno node:sqlite with Cloudflare's existing filesystem schema and operations against an on-disk database.
• Filesystem state survives complete Deno process restarts and preserves deletions and replacements.
• The exact supported code-reuse mechanism is demonstrated; license and update obligations are documented.
• At least one native subprocess reads and mutates the Workspace through either real FUSE or an explicitly identified fallback.
• Real FUSE and the userspace shim receive separate evidence-backed verdicts.
• Crash tests identify the precise commit boundary for FUSE writes and SQLite state.
• The proof records platform prerequisites and does not claim support for untested release targets.
• The result states whether this can remain a single-download XMD distribution and what it must materialize at runtime.
• The result compares this topology with Spike: bundle workerd into the compiled XMD host #347 and recommends select, reject, or limit for Define the local Workspace host topology #346.
• No production Workspace API or architecture contract changes until Define the local Workspace host topology #346 records the product decision.

Non-goals

• Reimplementing Cloudflare's filesystem schema from scratch.
• Forking Cloudflare code without a pinned provenance and MIT notice.
• Productionizing the provider or changing Implement retained Workflow Workspace restoration #218.
• Providing a hardened sandbox for native commands.
• Claiming that a local SQLite file is itself a Cloudflare Durable Object.
• Adding snapshots, rollback, or arbitrary journal-prefix rewind.

Dependencies

• Alternative evidence path to Spike: bundle workerd into the compiled XMD host #347 / PR 🔬 Spike: bundle workerd into the compiled XMD host (#347) #348.
• Informs and blocks the decision in Define the local Workspace host topology #346.
• Define the local Workspace host topology #346 continues to block Implement retained Workflow Workspace restoration #218.
• End goal: Document adversarial implementation workflow WIP #181.

[taras]

Spike complete — PR #350 (spikes/349-dofs/, runs via deno task spike:349). Answers to the eight questions, full detail in evidence/EVIDENCE.md, comparison in evidence/COMPARISON.md:

1. File-backed node:sqlite adapter: yes. ~100 lines mirroring upstream's own test fixture satisfies the storage contract with zero changes to Cloudflare's schema or filesystem code. Frontier (incl. create→delete→create, rename, symlinks) survives full restarts; db-path isolation holds; a newer on-disk schema is refused loudly (Unsupported workspace filesystem schema version 99), never recreated; WAL checkpoints to a single file on close. 110 MB proof, 0.08 s per full op cycle.
2. Reuse boundary settled by testing all four separately: direct import of @cloudflare/computer@0.1.1 fails (cloudflare:workers imported unconditionally); tree-shaking is structurally impossible (the published surface never exports Database/initializeSchema/WorkspaceFilesystem); vendoring packages/dofs works fully and ships in this PR (pinned v0.1.1/63d3636, MIT notice + provenance, source unmodified, 3 manifest edits documented); a 3-line upstream export diff producing a node:-only DOFS chunk is drafted and mechanically verified as the exit path.
3. Provider + FUSE driver under Deno: split verdict. The full stack (spliced SQLiteWorkspaceProvider → @platformatic/vfs → upstream driver.ts verbatim) is sound — but fuse.mount() aborts the Deno process in its uv polyfill (uv_default_loop returns null; addon loading itself works; minimal repro committed at evidence/fuse-linux/g2-min.mjs, worth filing on denoland/deno). The identical stack mounts under a Node sidecar in ~120 ms, sharing the WAL database with a Deno writer — that split topology is proven live, not hypothesized.
4. Native subprocess: yes, both ways. Through real FUSE (sidecar): every op passes (dd seek, truncate, mv, symlink, find, concurrent readers), bidirectional visibility immediate for new files (≤1 s kernel-cache staleness on API overwrites). Through the committed shim: a subprocess reads an API-written file and its writes land in SQLite [in-repo test]. The cwd-inside-mount spawn deadlock is real and unrecoverable by SIGKILL — upstream's cd-prefix technique is mandatory (runbook recorded).
5. Packaging: deno compile carries and loads the N-API addon both ways (142 MB); the mount then hits the same uv abort. Platform matrix from testing: linux-x64 works (sidecar); darwin-arm64 is a dead end at this pin (no prebuild; source-built addon SIGSEGVs — no arm64 dylib slice; macFUSE absent); linux-arm64 has no prebuild; Windows unsupported.
6. Durability: the commit boundary is observed RELEASE only — fsync is a durability no-op and close() races an async release (kill immediately after close loses the write; +500 ms commits). SIGTERM/SIGKILL of the mount host auto-unmounts cleanly; WAL recovers; a second process reads and writes the live db concurrently.
7. Userspace shim: development-only fallback (separate verdict). Runs under Deno byte-identical; ~80-110 ms reconcile medians; demonstrated losses: VFS-wins conflicts, symlinks degrade to content copies, chmod invisible, up to one 250 ms poll window of external writes clobbered after a crash. On darwin-arm64 it is currently the only bridge.
8. vs Spike: bundle workerd into the compiled XMD host #347: 110 MB vs 191 MB; 0.08 s vs 0.27 s; one process vs supervised workerd; no Computer execution backends vs all three; filesystem schema identical either way. COMPARISON.md recommends limit for Define the local Workspace host topology #346: Deno-local DOFS for the filesystem-only Workspace, bundled workerd where Computer backends are required, subprocess access under this topology explicitly limited until the Deno uv gap closes.

Evidence for the #346 decision posted there.

[taras]

Spike complete. PR #350 proves Deno-local SQLite can host Cloudflare DOFS directly and records the FUSE/native-command boundary. #346 / draft PR #358 selects this as the initial authoritative local Workspace foundation.