Skip to content

feat(sync): automerge document as source of truth for notebooks #885

@rgbkrk

Description

@rgbkrk

Problem

When a saved notebook is reopened, the daemon's new_fresh() in notebook_sync_server.rs deletes the persisted automerge document and creates a fresh empty one — treating .ipynb as the sole source of truth. If the .ipynb is missing or stale (e.g. user forgot to save), the automerge document's content is lost.

PR #884 (autosave) mitigates this by keeping .ipynb current, and we've added snapshot-before-delete as a safety net. But the fundamental design treats automerge docs as disposable for file-based notebooks.

Current model

  1. new_fresh() deletes any existing notebook-docs/{hash}.automerge for file-based notebooks
  2. A fresh empty automerge doc is created
  3. .ipynb is streamed into the empty doc via streaming_load_cells()
  4. Automerge doc is persisted on a 500ms debounce, but discarded on next reopen

The automerge document is ephemeral — it exists only while the notebook is open (plus cached between sessions, but overwritten on reopen).

Proposed model

Flip the source of truth: the automerge document accumulates all history and is never deleted.

  1. On reopen, load the existing automerge doc instead of creating a fresh one
  2. If .ipynb exists and is newer than the last automerge persist (by mtime), merge its cells into the existing doc
  3. If .ipynb is missing, use the automerge state as-is — no data loss
  4. Autosave (PR feat(notebook): autosave .ipynb files on daemon #884) becomes a pure export mechanism — writing .ipynb for interoperability with git, Jupyter, VS Code, etc.

Key considerations

External edits

The file watcher currently reloads from .ipynb on external change. Under the new model, external edits would need to be merged into the automerge doc rather than replacing it. This preserves edit history while incorporating changes from other tools.

Document size

AutoCommit::save() produces a compacted single-change representation, so document size grows slowly even with many edits. Benchmarking typical notebook sizes would help validate this. Periodic compaction could be added if needed for very long-lived documents.

Migration

No breaking change. Notebooks without an existing automerge doc would continue to bootstrap from .ipynb (the existing load_or_create code path). The change is purely about what happens when both an automerge doc and .ipynb exist.

Interaction with autosave (PR #884)

Autosave ensures .ipynb is almost always up-to-date, making the "which is newer?" check straightforward. The two features are complementary: autosave keeps the export current, automerge-as-source-of-truth prevents data loss.

Files involved

  • crates/runtimed/src/notebook_sync_server.rsnew_fresh(), streaming_load_cells(), file watcher
  • crates/runtimed/src/daemon.rshandle_open_notebook(), needs_load decision logic
  • crates/notebook-doc/src/lib.rsNotebookDoc::load(), NotebookDoc::new()

Metadata

Metadata

Assignees

No one assigned

    Labels

    architectureArchitecture proposals and structural changesdaemonruntimed daemon, kernel management, sync serverenhancementNew feature or requestsyncAutomerge CRDT sync protocol

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions