diff --git a/.braid/snapshot.jsonl b/.braid/snapshot.jsonl index f7819b2b9..bc757d9b3 100644 --- a/.braid/snapshot.jsonl +++ b/.braid/snapshot.jsonl @@ -153,6 +153,7 @@ {"id":"bd-3y7ul","title":"CI on main broken: Instant::now() panics on WASM in pass_one","description":"After bd-m7x9s (Parallelize Pass-1 via rayon, 665bbb34), CI is failing on origin/main. hub-client wasm tests (assetManifestProject, themeFingerprint, customNodeWireFormatProject) panic with 'PanicError: time not implemented on this platform' from `std::time::Instant::now()` in `ProjectPipeline::pass_one` (orchestrator.rs:847). The Instant::now() and the subsequent .elapsed() feed the `perf.pass1` gauge, but they always run — not gated for WASM.\n\nFix: route monotonic-clock access through SystemRuntime with a WASM shim using `performance.now()`. Add `monotonic_now_nanos() -> u64` to the trait; native default uses a process-start Instant; WasmRuntime overrides with js_sys/performance.\n\nFailing run: https://github.com/quarto-dev/q2/actions/runs/26310309964/job/77457143629","status":"closed","priority":0,"issue_type":"bug","created_at":"2026-05-22T21:03:37.847050Z","created_by":"cscheid","updated_at":"2026-05-22T21:16:17.323985Z","closed_at":"2026-05-22T21:16:17.323861Z","close_reason":"Fixed: SystemRuntime::monotonic_now_nanos shim (performance.now on WASM) + pass_one_dispatch_async to avoid pollster::block_on on WASM. Native + WASM tests green, full cargo xtask verify passes."} {"id":"bd-3zp3z4jx","title":"Link URL corrupted on write-back: a new/edited link in a multi-link paragraph gets an adjacent link's URL","description":"Repro (found via rich-text editor, but the bug is in the shared text-channel write-back, NOT the editor): editing a paragraph that ends up with two links commits correct markdown but persists the wrong URL.\n\nTS commit (verified via console log) sends:\n dest = {t:0, r:[75,208], d:0}\n newText = 'This is an [ordinary](https://example.org/ord) paragraph with **bold text**, *italic text*, some `inline code`, and a [hyperlink](https://quarto.org) to click.'\n\nAfter commitTextEdit -> parse_qmd_content(newText) -> apply_node_edit -> incremental_write, the file becomes:\n 'This is an [ordinary](https://quarto.org) paragraph ... and a [hyperlink](https://quarto.org) ...'\n\ni.e. the NEW link 'ordinary' gets the OTHER link's URL (quarto.org) instead of its own (example.org/ord). The serializer/round-trip is fine (a TS round-trip with two distinct links keeps both URLs). The corruption is in crates/pampa/src/apply_node_edit.rs + incremental writer / reconcile, likely in Link target source-info (targetS / url pool) handling when a paragraph is re-serialized after a link is added/changed. Because the text channel (commitTextEdit) is shared with the monospaced textarea editor, this affects link editing there too — suspected pre-existing. Single-link or no-link paragraph edits are unaffected (round-trips clean in the rich-text suite).","status":"closed","priority":1,"issue_type":"bug","created_at":"2026-06-23T23:01:30.815805Z","created_by":"Carlos Scheidegger","updated_at":"2026-06-24T15:14:31.024188Z","closed_at":"2026-06-24T15:14:31.024188Z","close_reason":"Merged as PR quarto-dev/q2#336","labels":["write-back"],"dependencies":{"bd-sjb4pzx8:discovered-from":{"depends_on_id":"bd-sjb4pzx8","type":"discovered-from","created_at":"2026-06-23T23:01:30.815805Z","created_by":"Carlos Scheidegger"}},"comments":{"c-hw3md4bm":{"id":"c-hw3md4bm","author":"Carlos Scheidegger","created_at":"2026-06-23T23:36:28.965290Z","text":"Root-caused + fixed on branch braid/bd-3zp3z4jx-link-url-corrupted-write (off origin/main 19aff019), commit be7a6d20. Root cause: reconciler matched container inlines by type discriminant only, so a new Link matched an old Link regardless of target; the inline splice then copied the old link's ](url) delimiter verbatim. Fix: require non-child identity (Link/Image target+attr, Span attr) to match before recursing; else UseAfter re-serializes via the qmd writer. TDD tests added (failing first). Full workspace nextest: 10337 pass, 0 regressions. NOT pushed yet (awaiting permission)."}}} {"id":"bd-3zst4hwy","title":"Clean up clippy debt and gate clippy in CI","description":"CI never runs clippy; ~69 violations have accrued across 11 crates (quarto-hub 42, xtask 11, quarto-system-runtime 7, others) once the existing [workspace.lints.clippy] allow-policy is respected. Dominant lints: collapsible_if (32), map_unwrap_or (13), needless_borrows_for_generic_args (5) — mostly clippy --fix auto-fixable. A few hand-fixes / #[allow]: should_implement_trait, naive_bytecount, get_unwrap. Plan: fix all 69, then add a CI step + xtask leg running 'cargo clippy --workspace --all-targets -- -D warnings' so it stays clean (gate-and-grow, but small enough to finish in ~1-2 sessions). Architectural lints (result_large_err, large_enum_variant, too_many_arguments, type_complexity, ptr_arg) are already allowed in the workspace table and stay allowed. Discovered while adding q2 mcp --print-config (bd-9a8yu2gw).","status":"closed","priority":2,"issue_type":"chore","created_at":"2026-06-13T15:45:10.467531Z","created_by":"Carlos Scheidegger","updated_at":"2026-06-13T16:38:17.617590Z","closed_at":"2026-06-13T16:38:17.617590Z","close_reason":"Workspace clippy clean (cargo clippy --workspace --all-targets -- -D warnings exits 0); gate added to CI (test-suite.yml) + cargo xtask verify Step 1. ~525 violations fixed (auto + hand). 10036 tests pass. Branch beads/clippy-cleanup-gate, commits c9445494/25c1e187/d1cb1dec(+plan). Not pushed; hub-client test leg fails on pre-existing missing-WASM env, orthogonal.","dependencies":{"bd-9a8yu2gw:discovered-from":{"depends_on_id":"bd-9a8yu2gw","type":"discovered-from","created_at":"2026-06-13T15:45:10.467531Z","created_by":"Carlos Scheidegger"}},"comments":{"c-g0rfqzgg":{"id":"c-g0rfqzgg","author":"Carlos Scheidegger","created_at":"2026-06-13T16:09:44.958893Z","text":"SCOPE CORRECTION + progress checkpoint. The initial '69 violations' was a measurement artifact: clippy -D warnings aborts a crate on its first lint, masking the rest, so quarto-core (~240) was hidden until upstream crates were fixed. True scope is 250+ across the workspace, quarto-core-dominated. Progress: auto-fixed the bulk via cargo clippy --fix (now ~150-file diff, +/-); hand-fixed/allowed the judgment lints. Down to 28 remaining (mostly MaybeIncorrect single_match/filter_map_next + test-data approx_constant + deliberate from_str should_implement_trait). NOTE: had to use --broken-code to apply quarto-core fixes (normal cargo fix reverts a whole crate if any fix breaks compile); this exposed a clippy bug that expanded a matches!() macro into its raw template (compile_theme_css.rs) + cascaded an import removal — caught and hand-fixed; workspace cargo check --all-targets is green. NOT YET DONE: full nextest run to validate the auto-fixes, finish last 28, add the CI gate. Remaining list saved to claude-notes/2026-06-13-clippy-remaining.txt. Uncommitted on branch beads/clippy-cleanup-gate."}}} +{"id":"bd-3zuggmsr","title":"q2 preview: active page outside _quarto.yml render: list shows \"Project render produced no output\"","description":"In project preview, navigating to (or launching on) a page that is NOT in the project's `render:` list (e.g. a scratch .qmd at the project root) shows the Render Error banner \"Project render produced no output for the active page\", even though the page is valid.\n\nROOT CAUSE (confirmed): RenderMode::ActivePage filters project.files, which discovery builds from the render: globs. A page outside those globs is absent from project.files, so pass_two produces zero outputs; the WASM entry point (wasm-quarto-hub-client/src/lib.rs:1633) then hits the catch-all because the active page is not among pass1_failures either. It is NOT sibling-capture-failure poisoning and NOT a poisoned engine host (fail_fast defaults false; proven with a clean no-broken-siblings native repro at the orchestrator level).\n\nQ2 has no single-file escape hatch for an in-project file: crates/quarto/src/commands/preview.rs resolve_project_and_initial_page forces project mode for any file under a _quarto.yml ancestor (single_file: None, pinned by test at preview.rs:575); --no-project is mutually exclusive with a positional path. So without a fix, a render:-excluded in-project page is unreachable in preview.\n\nQ1 (TypeScript quarto-cli) comparison, source-read only: it is mode-dependent. Direct 'quarto preview foo.qmd' renders a render:-excluded file (cmd.ts single-file render() fallback bypasses project.files.input). But navigating to a non-member inside a running project preview REFUSES via previewUnableToRenderResponse() (serve.ts:877) / 404 -- the direct analog of Q2's message. So Q2's current behavior matches Q1's navigation mode; the real defect is (a) the confusing message and (b) no way to preview the page at all.\n\nDESIGN DECISION (open -- no fix landed):\n- Option 1: orchestrator injects the ActivePage target into project.files so it renders on-demand in project mode. Prototyped and verified end-to-end (native RED->GREEN test + q2 preview + headless Chromium: sum_sines2.qmd rendered with 5 marimo islands). BUT the render:-excluded page renders with DEGRADED nav chrome: sidebar container present but EMPTY (no links, missing 'nav-sidebar docked' body classes) because the page is not a member of any website.sidebar.contents entry. Looks half-wired.\n- Option 2: route the render:-excluded active page through the single-doc renderer (standalone, no project chrome) -- closer to Q1's direct-invocation single-file semantics; sidesteps the empty-sidebar artifact; but does not give project theme/nav and is a larger CLI+WASM change.\n\nSTATUS: Option 1 prototype was REVERTED at Gordon's request pending the design decision above. Nothing committed. See conversation 2026-07-24.","status":"open","priority":1,"issue_type":"bug","created_at":"2026-07-24T16:57:57.818860Z","created_by":"Gordon Woodhull","updated_at":"2026-07-24T22:04:58.757367Z","labels":["bug","preview"],"comments":{"c-0b5awo7m":{"id":"c-0b5awo7m","author":"Gordon Woodhull","created_at":"2026-07-24T22:04:58.757367Z","text":"2026-07-24: Prototyped Option 1 (orchestrator injects ActivePage target into project.files). Verified end-to-end but found it renders render:-excluded pages with an empty sidebar rail (page is not a member of any configured sidebar). Reverted the prototype (both orchestrator.rs and the regression test) pending decision between Option 1 (project render, degraded nav) and Option 2 (single-file render for excluded pages, Q1-parity). No code committed."}}} {"id":"bd-41dros0u","title":"node_tests.rs unused PathBuf import fails clippy on Windows (-D warnings)","description":"std::path::PathBuf is imported unconditionally at crates/quarto-mcp-launcher/tests/integration/node_tests.rs:11, but its only consumer is fake_node at line 22, which is #[cfg(unix)]-gated. On Windows this makes the import unused, failing cargo clippy --all-targets -D warnings. CI (Linux/Mac) is unaffected. Fix: move the use inside the #[cfg(unix)] block or gate it with #[cfg(unix)]. Discovered during bd-i9i5ad2t Phase 5 verify while fixing the sibling bd-nj9nnkn1 dead-code issue.","status":"closed","priority":2,"issue_type":"bug","created_at":"2026-07-02T11:54:46.698369Z","created_by":"cderv","updated_at":"2026-07-02T11:56:14.007894Z","closed_at":"2026-07-02T11:56:14.007894Z","close_reason":"Moved the PathBuf import into the #[cfg(unix)] fake_node fn (its only consumer); verified with cargo clippy -p quarto-mcp-launcher --all-targets -- -D warnings","labels":["windows"],"dependencies":{"bd-i9i5ad2t:discovered-from":{"depends_on_id":"bd-i9i5ad2t","type":"discovered-from","created_at":"2026-07-02T11:54:46.698369Z","created_by":"cderv"}}} {"id":"bd-42owi5yb","title":"Selection-sync: pure helpers + parent applySelection (TDD scaffold)","description":"Pure, unit-testable core: sourceOffsetToLineCol(byteLineMap, offset)->{line,col}; parent-side applySelectionMessage(msg)->monaco.Selection with echo-guard (isSyncingRef window + editorHasFocusRef). No wiring yet. jsdom/vitest tests are the deliverable. Phase 0 of bd-6dib1198.","status":"open","priority":2,"issue_type":"task","created_at":"2026-06-25T21:27:50.121566Z","created_by":"shikokuchuo","updated_at":"2026-06-25T21:27:50.121566Z","dependencies":{"bd-6dib1198:parent-child":{"depends_on_id":"bd-6dib1198","type":"parent-child","created_at":"2026-06-25T21:27:50.121566Z","created_by":"shikokuchuo"}}} {"id":"bd-45tfp790","title":"Localization design: title-block labels, date locales, locale-week tokens","description":"Deferred from the title-block parity epic (bd-gx9cic8z, decisions Q3/Q4 + P4 design doc): a lang-driven localization system covering title-block metadata labels (Q1's _language.yml), localized date named styles + month/day names, locale-week date tokens (w ww wo gggg), and local-timezone resolution for today/now keywords (currently UTC). Design doc pointers: claude-notes/plans/2026-07-17-date-formatting-design.md deviations 2/6; claude-notes/plans/2026-07-15-html-title-block-parity.md Q3.","status":"open","priority":2,"issue_type":"feature","created_at":"2026-07-17T15:13:49.191932Z","created_by":"Carlos Scheidegger","updated_at":"2026-07-17T15:13:49.191932Z","dependencies":{"bd-y71ga2l8:discovered-from":{"depends_on_id":"bd-y71ga2l8","type":"discovered-from","created_at":"2026-07-17T15:13:49.191932Z","created_by":"Carlos Scheidegger"}}} @@ -278,6 +279,7 @@ {"id":"bd-9cyza5vy","title":"Single-file q2 preview: resolve the deck's full transitive sibling deps (includes + their assets), and re-resolve on edit","description":"Follow-up to bd-kpuweafo. ROOT CAUSE (verified E2E 2026-06-16): single-file q2 preview (no _quarto.yml) only syncs a narrow, statically-parsed set of siblings into the VFS, whereas q2 render reads the whole filesystem and project-mode preview walks the whole dir. So in single-file preview, anything the deck transitively needs that isn't pre-synced is broken.\n\nVerified matrix (single deck main.qmd with {{< include part.qmd >}} where part.qmd has ![](./inc-image.png)):\n- q2 render (single file): include resolves + image displays (naturalWidth 320). WORKS (reads disk directly).\n- q2 preview project (touch _quarto.yml): include resolves + image displays (blob). WORKS (dir walk syncs part.qmd + image).\n- q2 preview single-file: include renders as literal '?include' placeholder; 'Included Section' absent; image absent. BROKEN.\n\nSo the user's presumed inconsistency (includes work but their images don't) is actually: INCLUDES DON'T WORK EITHER in single-file preview - same root cause as the direct-image gap. bd-kpuweafo fixed only DIRECT image refs in the deck (parse deck -> collect Image URLs -> sync). It did NOT fix: (a) {{< include >}} of sibling .qmd (included file not in VFS -> shortcode fails); (b) images referenced INSIDE included files (transitive); (c) refs added mid-edit (resolution is one-shot at session start in build_hub_config).\n\nWHY HARD: bd-tnm3k single-file mode deliberately does NOT walk the directory (a bare 'q2 preview ~/Downloads/x.qmd' must not index all of ~/Downloads). So every sibling the deck needs must be DISCOVERED BY PARSING. quarto-hub has no qmd parser; extraction lives in quarto-preview (config::resolve_single_file_assets) and is injected via HubConfig. Includes also can't be resolved on-demand mid-render because the WASM include-expansion reads synchronously from the VFS - the file must be pre-synced.\n\nPROPOSED FIX: make single-file extraction TRANSITIVE and recursive: parse deck -> collect {{< include >}} targets (extract_include_path, already used by deps.rs) + Image URLs; for each included .qmd, recursively parse it -> collect its includes + images; sync included .qmd as text and all images as binary into the VFS. Bonus: re-run extraction on deck-change watch events so mid-edit-added refs sync without reload. Keep the under-deck-dir canonicalization guard (no ../ escape). This makes single-file preview match render/project for the deck's own dependency closure without a full dir walk. Plan: claude-notes/plans/2026-06-16-single-file-preview-referenced-assets.md (v1 limits section).","status":"closed","priority":2,"issue_type":"feature","created_at":"2026-06-16T23:40:26.135173Z","created_by":"Carlos Scheidegger","updated_at":"2026-06-17T02:05:04.565000Z","closed_at":"2026-06-17T02:05:04.565000Z","close_reason":"Implemented + merged in PR #300 (feature/bd-9cyza5vy-single-file-preview-transitive-deps). Single-file q2 preview now resolves the deck's full transitive include + image closure via the renderer's own include-expansion run natively; included .qmd sync as invisible text deps; the watch set covers the closure. TDD + E2E verified; CI clean. Render-side nested-include retargeting bug remains open as bd-udrn0q47.","labels":["preview"],"dependencies":{"bd-kpuweafo:discovered-from":{"depends_on_id":"bd-kpuweafo","type":"discovered-from","created_at":"2026-06-16T23:40:26.135173Z","created_by":"Carlos Scheidegger"}},"comments":{"c-bni1dft5":{"id":"c-bni1dft5","author":"Carlos Scheidegger","created_at":"2026-06-16T23:51:38.773727Z","text":"Design-exploration plan written: claude-notes/plans/2026-06-16-single-file-preview-vfs-bootstrapping.md. Frames the bootstrapping paradox (populating the ephemeral VFS requires parsing files, but parsing requires the VFS to be populated), the verified behavior matrix, why single-file can't walk the dir (bd-tnm3k), the full set of dependency channels, and 5 design options (A transitive static extract / B lazy on-miss fetch / C render-driven fixpoint reusing pipeline deps / D bounded walk / E async file_read) with eval criteria + a tentative lean toward C. No implementation beyond bd-kpuweafo's direct-image sync; this is the explore-the-space step the user requested."},"c-podosqfn":{"id":"c-podosqfn","author":"Carlos Scheidegger","created_at":"2026-06-17T01:18:01.366247Z","text":"Implementation complete (pending review/commit). All 4 phases + spike done, TDD throughout.\n\nPhase 0 (spike): chosen entry point = run the renderer's OWN ParseDocumentStage + IncludeExpansionStage natively via pollster::block_on, so include path resolution matches q2 render exactly (incl. the bd-udrn0q47 nested-include behavior — preview inherits it automatically).\n\nPhase 1 (quarto-preview/config.rs): resolve_single_file_deps -> SingleFileDeps { qmd_files, binary_files }. Text deps = doc.recorded_includes (transitive, cycle-truncated); binary deps = collect_referenced_asset_urls over the EXPANDED AST, deck-dir-anchored (render parity / no retargeting). Under-root guard preserved. Deleted superseded resolve_single_file_assets. 6 new tests.\n\nPhase 2 (quarto-hub + preview): included .qmd ride a new text-only path — HubConfig.single_file_text_deps -> ProjectFiles.text_dep_files + with_text_deps (in text_files()/all_files()/counts, NOT qmd_files, so invisible/no nav). build_hub_config rewired to one resolve_single_file_deps call filling both fields. discovery.rs + context.rs tests.\n\nPhase 3 (quarto-hub/watch.rs + server.rs): WatchConfig.single_file_deps; FileWatcher subscribes deck + each closure dep (NonRecursive, best-effort) and accepts only allow-set members. server.rs builds the set from project_files().all_files() minus the deck. bd-tnm3k safety preserved (unrelated siblings still ignored) — pinned by test.\n\nPhase 4 (E2E): q2 preview main.qmd (no _quarto.yml), browser inspection of the render iframe: 'Included Section' renders (include expanded, no literal ?include), and the image INSIDE the include displays via blob URL (naturalWidth 1, the 1x1 PNG loaded). \n\ncargo xtask verify --skip-hub-build: All verification steps passed (clippy -D warnings + full workspace nextest). Plan: claude-notes/plans/2026-06-16-single-file-preview-transitive-deps.md"}}} {"id":"bd-9eltv","title":"Profile q2 render on a large website (quarto-web)","description":"Use external-sources/quarto-web as a large-project fixture to characterize q2's render performance. Follow the native-proxy-first workflow in claude-notes/instructions/performance-profiling.md: flamegraph, env-gated counters, geometric scaling, then synthesis. Produces a written analysis and per-hotspot follow-up issues; not a fix.\n\nPlan: claude-notes/plans/2026-05-21-q2-render-website-profile.md","status":"closed","priority":2,"issue_type":"task","created_at":"2026-05-21T21:13:06.218065Z","created_by":"cscheid","updated_at":"2026-06-08T18:28:45.164112Z","closed_at":"2026-06-08T18:28:45.164112Z","close_reason":"Audit 2026-06-08: quarto-web render profiling deliverable produced (plan + research note + flamegraphs); blocker bd-wlza2 closed.","dependencies":{"bd-wlza2:blocks":{"depends_on_id":"bd-wlza2","type":"blocks","created_at":"2026-05-21T21:13:06.218065Z","created_by":"cscheid"}}} {"id":"bd-9ez3ngt1","title":"reference-location ignored from front matter (as_str vs PandocInlines)","description":"reference-location set in document YAML front matter is silently ignored in ALL placements (top-level, nested under format.html) and for ALL values (margin/block/section) — rendering always falls back to the 'document' default.\n\nCONFIRMED root cause (runtime dump of get_reference_location in the real q2 render pipeline):\n\n reference-location: margin -> kind=PandocInlines, as_str()=None, as_plain_text()=Some(\"margin\")\n reference-location: !str margin -> kind=Scalar, as_str()=Some(\"margin\")\n\nIn document-metadata context, a bare YAML string value is parsed as markdown and stored as ConfigValueKind::PandocInlines (see quarto-pandoc-types/src/config_value.rs:189-195), NOT Scalar(String). ConfigValue::as_str() (config_value.rs:641-649) returns None for PandocInlines. So:\n\n FootnotesTransform::get_reference_location (crates/quarto-core/src/transforms/footnotes.rs:76-81)\n AppendixStructureTransform reference-location read (crates/quarto-core/src/transforms/appendix.rs:92)\n\nboth call .and_then(|v| v.as_str()) and get None -> ReferenceLocation::default() = Document.\n\nThe key is PRESENT; the accessor is wrong. The codebase already has the right convention: ConfigValue::as_plain_text() (config_value.rs:675-684) handles both Scalar(String) and PandocInlines, and is used deliberately elsewhere for exactly this (filter_resolve.rs:90,212 with the comment 'handle both Scalar(String) and PandocInlines forms'; document_profile.rs reads all meta strings via as_plain_text). The !str tag is a working-but-undocumented escape hatch today.\n\nFIX: replace .as_str() with .as_plain_text() in footnotes.rs:78 and appendix.rs:92 (audit appendix.rs for other reference-location reads too). Low risk, matches existing convention. TDD: add a failing e2e render test (front matter reference-location: margin -> margin-note class, no doc-endnotes section) before fixing.\n\nDistinct from bd-po3gn41h (named [^id] footnote refs, PR #264) and from bd-1kly (block/section numbering). Discovered while verifying bd-po3gn41h margin-mode parity.","status":"closed","priority":2,"issue_type":"bug","created_at":"2026-06-09T14:54:08.953791Z","created_by":"Carlos Scheidegger","updated_at":"2026-06-09T15:47:46.081783Z","closed_at":"2026-06-09T15:47:46.081783Z","close_reason":"Fixed and merged via PR #265 — reference-location honored from front matter (as_str -> as_plain_text).","external_ref":"https://github.com/quarto-dev/q2/pull/265","labels":["bug","footnotes"],"dependencies":{"bd-po3gn41h:discovered-from":{"depends_on_id":"bd-po3gn41h","type":"discovered-from","created_at":"2026-06-09T14:54:08.953791Z","created_by":"Carlos Scheidegger"}},"comments":{"c-1fy79tfx":{"id":"c-1fy79tfx","author":"Carlos Scheidegger","created_at":"2026-06-09T14:59:18.005535Z","text":"PR opened: https://github.com/quarto-dev/q2/pull/265 (branch bugfix/bd-9ez3ngt1-reference-location-front-matter, base main, commit a35df31a). Fix: as_str -> as_plain_text in get_reference_location of footnotes.rs + appendix.rs. e2e test + full verify green. Left in_progress pending merge."}}} +{"id":"bd-9fwn1504","title":"quarto-ast-reconcile: proptest counterexample — reconciliation does not preserve structure (full AST)","description":"CI on PR #415 hit a failing random case in property_tests::reconciliation_preserves_structure_full_ast (crates/quarto-ast-reconcile/src/lib.rs:1242, 'Result should be structurally equal to after'). Reproduced locally with the CI seed on BOTH the PR branch and main — pre-existing latent bug, unrelated to that PR (which touches only quarto-hub/preview/hub-provider). Deterministic reproducer: create crates/quarto-ast-reconcile/proptest-regressions/lib.txt containing the line 'cc 2c379a4ae900cb3d235f771b204e0cb0e307ee6808562d588dc6ecd81088e38e' and run cargo nextest run -p quarto-ast-reconcile -E 'test(reconciliation_preserves_structure_full_ast)'. Note proptest hit max_shrink_iters=1024 while shrinking, so the stored case may be large; raising PROPTEST_MAX_SHRINK_ITERS should shrink it further for debugging. When fixing, commit the seed file as the regression pin (TDD: it fails first). Deliberately NOT committed on PR #415 — it would make that unrelated PR deterministically red.","status":"open","priority":1,"issue_type":"bug","created_at":"2026-07-24T21:23:43.948082Z","created_by":"Carlos Scheidegger","updated_at":"2026-07-24T21:23:43.948082Z","dependencies":{"bd-eiku4ymo:discovered-from":{"depends_on_id":"bd-eiku4ymo","type":"discovered-from","created_at":"2026-07-24T21:23:43.948082Z","created_by":"Carlos Scheidegger"}}} {"id":"bd-9fz5fweg","title":"Figures/floats/layout-panels CSS from _quarto-rules (blocked on class taxonomy)","description":"Once the float/layout DOM class taxonomy (bd-hcp8m3ve) lands, port _quarto-rules.scss: .quarto-layout-* family (L38–103), .quarto-figure* + alignment variants + figcaption.quarto-float-caption-* + div[id^=tbl-] positioning (L105–130, 140–151), figure.quarto-float-tbl captions (L243–253). Adapt selectors to whatever taxonomy bd-hcp8m3ve chooses. figure > p:empty/:first-child (L132–138) stay dropped — Q2 figures have no

children. Note _bootstrap-rules.scss already carries dead fragments (.quarto-layout-cell[data-ref-parent], responsive .quarto-layout-row) — reconcile, don't duplicate. Inventory rows 3/4a/4b/8.","status":"open","priority":3,"issue_type":"task","created_at":"2026-07-21T17:52:15.888586Z","created_by":"Carlos Scheidegger","updated_at":"2026-07-21T17:52:15.888586Z","labels":["css","parity"],"dependencies":{"bd-4doe9lvt:parent-child":{"depends_on_id":"bd-4doe9lvt","type":"parent-child","created_at":"2026-07-21T17:52:15.888586Z","created_by":"Carlos Scheidegger"},"bd-hcp8m3ve:blocks":{"depends_on_id":"bd-hcp8m3ve","type":"blocks","created_at":"2026-07-21T17:52:15.888586Z","created_by":"Carlos Scheidegger"}}} {"id":"bd-9h2g","title":"Cargo: upgrade scraper v0.22.0 → v0.26.0","description":"Major upgrade surfaced by cargo-upgrade survey 2026-05-04. Current 0.22.0 is range-pinned in workspace; latest is 0.26.0. Type: pre-1.0 minor (semver-breaking); four minor steps. Review changelog and bump deliberately. See claude-notes/plans/2026-05-04-cargo-upgrade-survey.md and bd-hb8h.","status":"closed","priority":3,"issue_type":"chore","created_at":"2026-05-04T18:15:55.070022Z","created_by":"cscheid","updated_at":"2026-05-04T20:30:45.306184Z","closed_at":"2026-05-04T20:30:45.306039Z","close_reason":"merged: 86464160","labels":["cargo","deps"],"dependencies":{"bd-hb8h:discovered-from":{"depends_on_id":"bd-hb8h","type":"discovered-from","created_at":"2026-05-04T18:16:05.270861Z","created_by":"cscheid"}}} {"id":"bd-9hlja","title":"Coalesce per-page diagnostics by source location","description":"Add a coalescing pass in the render-summary printer: same-(code, source-location, title) diagnostics across pages collapse into one DiagnosticMessage emission with an 'Affected files: a, b, c (and N others)' tail. Default cap: 3 names + count.\n\nAPI lives in quarto-error-reporting (proposed coalesce.rs module). Call site is print_render_diagnostics in crates/quarto/src/commands/render.rs:704-735, which today walks pass2_failures and outputs[i].render_output.diagnostics independently and emits each verbatim.\n\nNon-coalescable shapes (SourceInfo::Concat, FilterProvenance) pass through as singletons in the first cut.\n\nThis is one of two children of the theme-diagnostic overhaul epic. Companion issue makes the theme error structured so this coalescer has something to coalesce.\n\nPlan: claude-notes/plans/2026-05-22-diagnostic-coalescing.md","status":"closed","priority":2,"issue_type":"feature","created_at":"2026-05-22T13:43:11.958574Z","created_by":"cscheid","updated_at":"2026-05-22T15:05:25.328709Z","closed_at":"2026-05-22T15:05:25.328553Z","close_reason":"Implemented. Theme errors across all pages collapse into 1 ariadne block + 'Affected files: …' line. Verified end-to-end against quarto-web: 345 -> 1 emission.","labels":["diagnostics","website"],"dependencies":{"bd-l26u6:parent-child":{"depends_on_id":"bd-l26u6","type":"parent-child","created_at":"2026-05-22T13:43:11.958574Z","created_by":"cscheid"},"bd-pgczr:related":{"depends_on_id":"bd-pgczr","type":"related","created_at":"2026-05-22T13:43:11.958574Z","created_by":"cscheid"}}} @@ -402,7 +404,7 @@ {"id":"bd-egcyeym9","title":"Extract quarto-yaml-validation into a standalone, non-Quarto-specific repository","description":"Architectural investigation + design for moving quarto-yaml-validation (and its outbound dependency closure: quarto-source-map, quarto-yaml, quarto-error-reporting) out of quarto-dev/q2 so non-Quarto developers can use the YAML-schema-validation infrastructure. Key design problem: error codes (Q-1-x) and the centralized catalog/docs URLs are Quarto-specific; the standalone library needs its own error-code provider while the q2-embedded build keeps existing codes. Requires extracting quarto-error-reporting into a reusable library with pluggable/remappable error-code catalogs. This session gathered current-state architecture; see linked research doc.","status":"open","priority":2,"issue_type":"epic","created_at":"2026-06-17T13:56:31.311381Z","created_by":"Carlos Scheidegger","updated_at":"2026-06-29T13:33:54.125548Z","labels":["architecture","research"],"comments":{"c-4iakr487":{"id":"c-4iakr487","author":"Carlos Scheidegger","created_at":"2026-06-26T17:16:26.928582Z","text":"Design refinement 2026-06-26 (cont.). User decisions:\n\nSEQUENCING (firm): extract-first, migrate-q2-LAST. Order: (1) foundation repo under posit-dev/ = quarto-source-map + error-reporting-core (split-out catalog-agnostic half), publish to crates.io, validate standalone; (2) yaml repo = quarto-yaml + quarto-yaml-validation as the FIRST CLIENT of the discipline, delete validate-yaml; (3) q2 migrates to published crates last. Rationale: invisible internal Posit consumers of quarto-yaml-validation need a real standalone repo; and the error-code discipline is a HOST contract that must be proven before its first client (error-reporting before yaml-validation). This inverts the earlier 'P0-P3 in q2 first' ordering.\n\nERROR-CODE DISCIPLINE crystallized into claude-notes/designs/cross-package-error-codes.md:\n- Fallback hierarchy: tier1 remapped to Q-code (best) > tier2 library origin code passthrough (acceptable) > tier3 codeless (FORBIDDEN). Library contract guarantees tier3 never happens (every emittable diagnostic carries a stable namespaced origin code), so the embedder remap is a pure upgrade tier2->tier1, optional/per-code. Refines I3: unmapped is acceptable, audit WARNS not fails.\n- Three contracts spelled out: library-author (own namespace, every error coded, stable, self-describe docs a la ESLint meta.docs.url), embedder/product (own remap+catalog, choose unmapped policy, never leak scheme upstream), shared-infra error-reporting-core (namespace-agnostic, provides CatalogProvider trait + remap hook + renderer = rustc LintStore analog).\n- n2 case (a non-q2 embedder) resolved: supplies its own remap+catalog over the same core; remap is per-embedder, not a q2 feature -> remap hook MUST live in error-reporting-core.\n- Clippy (clippy:: tool-lint namespace, own docs site, no E-codes) and ESLint (plugin/rule namespacing + meta.docs.url self-description) researched: both namespace + self-describe but EXPOSE the decomposition (they are platforms); q2 is a product that HIDES it, so q2 adds the remap they omit. TS = the compiler (flat numeric central, no cross-package design), good for presentation layer only.\n\nOPEN FORKS for user: repo granularity (1 foundation repo + 1 yaml repo [rec] vs single 4-crate workspace repo); keep dormant Q-1-* yaml catalog entries vs remove; project naming."},"c-5hksonc9":{"id":"c-5hksonc9","author":"Carlos Scheidegger","created_at":"2026-06-27T21:59:18.615124Z","text":"Phase 3 progress (2026-06-27): quarto-error-reporting standalone repo created + pushed. https://github.com/posit-dev/quarto-error-reporting (public). Standalone single-crate, version 0.1.0, depends on published quarto-source-map 0.1.0 + crates.io deps; json behind default-off feature (schemars optional). Builds default(json off, no schemars) + all-features; tests 51/61 + doctests + schema_drift; fmt+clippy clean both feature sets. CI workflow added (3 OSes, both feature sets), running. External-consumer smoke test passes (separate crate, default features, no schemars, EmptyCatalog + custom CatalogProvider both work). cargo publish --dry-run clean.\n\nSource fix for stable-clippy -D warnings: macros.rs items_after_test_module (q2 nightly clippy tolerated it) -> moved test mod below #[macro_export] macros + dropped redundant imports. q2 deletes its copy at cutover, so standalone is single source, no divergence. Dropped CONTRIBUTING-ERRORS.md (Quarto catalog policy) + rewrote README for the catalog-agnostic lib.\n\nNEXT: 3d publish to crates.io (USER step). Then 3e q2 cutover."},"c-6c2707we":{"id":"c-6c2707we","author":"Carlos Scheidegger","created_at":"2026-06-17T13:58:01.021204Z","text":"Current-state architecture written to claude-notes/research/2026-06-17-extract-quarto-yaml-validation.md\n\nKey findings:\n- Outbound closure to externalize = quarto-yaml-validation + 3 foundation crates (quarto-yaml, quarto-source-map [clean leaf], quarto-error-reporting).\n- Inbound: quarto-yaml-validation has only ONE in-repo consumer (validate-yaml bin) and is NOT wired into the render pipeline/config/WASM -> moves cleanly.\n- BUT the 3 foundation crates are heavily used inside q2 (source-map ~26 dependents, error-reporting ~19, quarto-yaml 8) -> they cannot move, must become shared/published deps consumed by both repos.\n- Hard design problem = error-code identity: yaml-validation hard-codes Q-1-x codes; quarto-error-reporting holds a centralized 145-entry error_catalog.json with quarto.org docs URLs, enforced by scripts/audit-error-codes.py. Standalone lib needs a pluggable/remappable catalog provider; q2-embedded build keeps Q-1-x. Likely split quarto-error-reporting into catalog-agnostic core + q2 catalog policy, or inject an ErrorCodeProvider trait.\n\nOpen questions (extraction strategy: own vs mirror vs publish-from-q2; core/catalog split; JSON wire types ownership; audit adaptation) listed at end of doc."},"c-6e55iwby":{"id":"c-6e55iwby","author":"Carlos Scheidegger","created_at":"2026-06-27T19:18:54.317557Z","text":"Step-1 plan revised (2026-06-27): TWO repos instead of one, and order flipped to leaf-first. Verified dependency is strictly one-directional: error-reporting-core -> quarto-source-map (SourceInfo is a field on DiagnosticMessage at diagnostic.rs:126; SourceContext threads through render/builder); source-map depends on nothing in error-reporting. This forces source-map to be published BEFORE error-reporting-core (crates.io rejects unpublished path deps), so the user's proposed 'error-reporting first' order is flipped. Plan restructured into: Phase 1 = extract quarto-source-map (trivial leaf; the warmup that proves repo-setup + crates.io publish + WASM cutover on the easy crate), Phase 2 = split error-reporting in place (independent, can overlap Phase 1), Phase 3 = extract error-reporting-core (needs source-map published + split done). q2 cutover is incremental (source-map first at 1d, core at 3e). Granularity decision now DECIDED=two-repos. Remaining open forks: core name, version start (0.1.0 rec), crates.io vs git dep, the two repo names."},"c-79nicwe8":{"id":"c-79nicwe8","author":"Carlos Scheidegger","created_at":"2026-06-27T20:26:52.855642Z","text":"PR #348 CI fully GREEN: all 5 checks pass — Run test suite (macos-latest)x2, (ubuntu-latest)x2, Hub-Client E2E. PR MERGEABLE / mergeStateStatus CLEAN, no review required. This confirms the quarto-source-map cutover in a clean CI env on both OSes incl. the WASM/hub leg. Phase 1 verified end-to-end; awaiting user decision to merge."},"c-91u3x7zk":{"id":"c-91u3x7zk","author":"Carlos Scheidegger","created_at":"2026-06-26T18:27:21.939037Z","text":"Step-1 plan written: claude-notes/plans/2026-06-26-extract-error-reporting-foundation.md — extract quarto-source-map + error-reporting-core (renamed, TBD) into a posit-dev/ repo, publish to crates.io, cut q2 over. Key measured finding: the catalog coupling inside error-reporting is ONE line (diagnostic.rs:290 docs_url -> catalog::get_docs_url) + only 2 external q2 callers (quarto-core project_resources.rs, theme_diagnostic.rs); direct dependents 14 (source-map) / 9 (error-reporting), not the transitive 26/19. Split: core = diagnostic+builder+macros+ErrorCodeInfo+CatalogProvider(OnceLock registry, std not once_cell, no schemars); q2 keeps error_catalog.json+QuartoCatalog provider+install() (quarto-error-catalog) + json.rs(wire)+coalesce.rs + a thin quarto-error-reporting façade so the 9 dependents compile unchanged. Three phases: A split-in-place (q2 green, TDD: installed catalog reproduces docs_url, empty catalog returns None), B new repo via git filter-repo + standalone CI with EmptyCatalog + publish, C q2 cutover (WASM is the risk surface, full xtask verify). Open forks at top: repo granularity (1 foundation repo rec), core name, version start (0.1.0 rec), crates.io vs git dep, repo name."},"c-98krqhge":{"id":"c-98krqhge","author":"Carlos Scheidegger","created_at":"2026-06-27T20:34:32.281773Z","text":"CI workflow for posit-dev/quarto-source-map added (.github/workflows/ci.yml, commit ee3780d, pushed to main). Stable Rust; test matrix ubuntu/macos/WINDOWS + fmt/clippy(-D warnings). First run GREEN on all 4 jobs (run 28300990636) — notably Windows builds clean (first time this crate has built on Windows; confirms it's genuinely cross-platform). Only the crates.io owner-add tidy-up remains (weekday)."},"c-d6qo0047":{"id":"c-d6qo0047","author":"Carlos Scheidegger","created_at":"2026-06-26T18:22:30.701335Z","text":"Discipline refinement 2026-06-26 (cont.3): APPEND-ONLY codes ('cool URLs for error codes', Berners-Lee 1998 applied to error ids). Folded into the design note + plan Q6.\n\nPrinciple: codes are unique and never deleted/repurposed. Lifecycle = Active -> Retired -> (never deleted). Legal transition Active->Retired (stop emitting, keep documented); FORBIDDEN: Active->Never (deletion) and code->different-meaning (repurposing, stronger than a major bump - simply off the table). A docs page accumulating no-longer-emitted codes is correct/expected (old versions, external references). This also protects the provenance breadcrumb: never-delete means a disclosed upstream code degrades to retired-but-documented, never 404 or silent-redefinition.\n\nFreeze binds at first PUBLIC exposure (emission or documentation), not first commit - pre-release dev churn is fine (semver pre-1.0 logic).\n\nEnforce intra-repo, encourage cross-repo (same asymmetry as provenance staleness): append-only is checkable within a repo (diff catalog vs git history/snapshot); not checkable across repos.\n\nCONCRETE CONSEQUENCE (the one place this bites existing code): q2's scripts/audit-error-codes.py is currently BIDIRECTIONAL. Must become: keep forward (every emitted code is documented), DROP reverse (every documented code is emitted - contradicts retirement), ADD append-only check. A retired/dormant code = legitimate catalog-only entry. ErrorCodeInfo.since_version already covers 'introduced'; optional retired_in/last_emitted for the window. This also largely settles the dormant-Q-1-* fork toward KEEP: any published yaml docs page is under the covenant."},"c-dhnsz7ix":{"id":"c-dhnsz7ix","author":"Carlos Scheidegger","created_at":"2026-06-26T16:58:23.019456Z","text":"Design session 2026-06-26. Two artifacts written:\n\n1. claude-notes/plans/2026-06-26-extract-quarto-yaml-validation-design.md — resolves the 7 open questions given the user's 'new repo owns foundation crates' lean. Key reframe: quarto-error-reporting is q2's whole diagnostics substrate (~19 dependents + shared JSON wire format), NOT an interchangeable foundation crate. Decision: split it into error-reporting-core (externalize, catalog-agnostic, CatalogProvider trait, no schemars, no quarto.org URLs) + quarto-error-catalog (stays in q2: Q-* catalog, quarto.org URLs, audit) + a thin quarto-error-reporting façade so the 19 call sites keep compiling. JSON wire types stay q2-side. Phased P0..P5 with P0-P3 landing entirely in q2 (deliver the seam) and P4-P5 the cross-repo commitment (separate go/no-go).\n\n2. claude-notes/designs/cross-package-error-codes.md — the general philosophy the user asked for. Two-identity model: ORIGIN codes (namespaced, package-owned, e.g. yaml-schema/type-mismatch — Clippy/ESLint precedent) + PRESENTATION codes (flat, product-owned Q-- — TS-compiler precedent). Product owns the remap. Invariant I1: subsystem != package, so q2 users never see the package decomposition. Corrected: 'TypeScript' = the language/compiler (flat central catalog, no cross-package design), not TS Quarto.\n\nUser clarifications folded in: validate-yaml to be DELETED (demo only; the ONLY in-repo consumer) -> quarto-yaml-validation will have ZERO q2 consumers, so the yaml remap is forward-looking/dormant until q2 wires the validator into config. Open forks for user: full repo move now vs stop after in-q2 split; keep dormant Q-1-* yaml catalog entries vs remove; project naming."},"c-erih8i4v":{"id":"c-erih8i4v","author":"Carlos Scheidegger","created_at":"2026-06-27T19:56:58.174762Z","text":"Phase 1 EXECUTED (2026-06-27): quarto-source-map extracted + PUBLISHED.\n\n- New repo: https://github.com/posit-dev/quarto-source-map (public). Standalone single-crate, version 0.1.0, edition 2024, builds on stable rustc 1.95 (no nightly). 104 unit + 4 doctests pass; cargo publish --dry-run clean.\n- PUBLISHED quarto-source-map 0.1.0 to crates.io (Carlos personal account; posit-dev owner-add deferred to a weekday). End-to-end crates.io pipeline exercised successfully (the point of doing the leaf first).\n- q2 cutover on branch braid/bd-egcyeym9-source-map-extraction: all 14 dependents consolidated onto { workspace = true }; root [workspace.dependencies.quarto-source-map] flipped path -> version=0.1.0; in-tree crates/quarto-source-map deleted. cargo build --workspace green; cargo nextest run --workspace = 10238 passed; Cargo.lock resolves from registry+crates.io with checksum. Full cargo xtask verify (WASM/hub leg) running now; cutover closes when green. NOT yet committed (awaiting verify + user go-ahead per GIT PUSH POLICY).\n- Gap to close: no GitHub Actions CI workflow in the new repo yet (tests run locally only).\n\nAll 4 crate names (source-map, error-reporting, yaml, yaml-validation) confirmed available on crates.io."},"c-gz6zngk0":{"id":"c-gz6zngk0","author":"Carlos Scheidegger","created_at":"2026-06-27T21:39:26.289270Z","text":"PR #349 (Phase 2) CI fully GREEN: all 5 checks pass (test suite macos x2 + ubuntu x2, Hub-Client E2E). MERGEABLE/CLEAN. Phase 2 verified in clean CI on both OSes incl. WASM. Awaiting user merge. Next: Phase 3 (extract quarto-error-reporting to posit-dev/ + crates.io), then the yaml stack."},"c-hr2nhh54":{"id":"c-hr2nhh54","author":"Carlos Scheidegger","created_at":"2026-06-29T13:33:54.125548Z","text":"Handoff note for the YAML stack written: claude-notes/plans/2026-06-29-yaml-stack-extraction-handoff.md. Self-contained plan for an agent to extract quarto-yaml + quarto-yaml-validation.\n\nSTRUCTURE DECISION (user, 2026-06-29): single repo posit-dev/quarto-yaml = a Rust WORKSPACE with two crates (NOT one-repo-per-crate like the foundation crates). They're tightly coupled (validation deps parser) + both Quarto-dialect-specific. Both still publish to crates.io independently, leaf-first (quarto-yaml then quarto-yaml-validation). Updated foundation plan decision #1 + the yaml design doc banner to point at the handoff.\n\nKey facts captured in the note: quarto-yaml is a clean leaf (only quarto dep = published source-map; NO error-reporting); 4 q2 consumers (pampa/config/core/lsp-core). quarto-yaml-validation deps quarto-yaml + published source-map + error-reporting; ONLY consumer is validate-yaml (delete it) -> ZERO q2 consumers after, so q2 deletes BOTH crates and doesn't depend on yaml-validation (it's published purely for external Posit consumers). WASM gotcha documented (pampa/core use workspace=true -> resolve to q2 root even in the wasm build; wasm likely needs no direct quarto-yaml dep; verify via full xtask verify). Error_code() Q-1-x -> origin codes yaml-schema/* per the discipline, flagged as a USER DECISION (breaking for existing Q-1-x consumers: A=origin from 0.1.0 [rec] vs B=defer to 0.2.0). All Phase 1/3 gotchas listed (CRLF/.gitattributes, stable-clippy, |tail masking, user-gated publish)."},"c-jgy534a7":{"id":"c-jgy534a7","author":"Carlos Scheidegger","created_at":"2026-06-27T20:07:14.077825Z","text":"Phase 1 committed + PR opened. Two commits on feature/bd-egcyeym9-source-map-extraction: (A) docs(design) — the discipline doc + 2 extraction plans; (B) build — the quarto-source-map cutover. PR #348: https://github.com/quarto-dev/q2/pull/348 (base main). Local full cargo xtask verify green (14 steps incl WASM). CI running (test suite macos+ubuntu, Hub-Client E2E). quarto-source-map 0.1.0 live at https://github.com/posit-dev/quarto-source-map + crates.io. Phase 1 done pending CI; tidy-ups deferred: new-repo CI workflow, crates.io owner-add."},"c-lrj5ylfo":{"id":"c-lrj5ylfo","author":"Carlos Scheidegger","created_at":"2026-06-27T20:04:07.040930Z","text":"Phase 1d cutover GREEN. Full cargo xtask verify passed all 14 steps incl. the WASM build (wasm-quarto-hub-client) + hub-client tests; cargo nextest run --workspace = 10238 passed. Both root Cargo.lock and wasm-quarto-hub-client/Cargo.lock resolve quarto-source-map 0.1.0 from registry+crates.io with matching checksum.\n\nGotcha hit + fixed: blanket crates/*/Cargo.toml path->workspace rewrite also touched wasm-quarto-hub-client, which is EXCLUDED from the main workspace and is its own standalone workspace (refs every q2 crate by path) -> 'workspace.dependencies not defined' at the wasm32 build. Fix: that crate gets a DIRECT version dep (quarto-source-map = \"0.1.0\"), only the 13 main-workspace members use { workspace = true }. Also: first verify's exit-0 was false (piped through tail, masking cargo's failure) — re-ran without tail.\n\nChange set on branch braid/bd-egcyeym9-source-map-extraction (UNCOMMITTED, awaiting go-ahead): root Cargo.toml dep path->version; 13 members ->workspace=true; wasm crate ->\"0.1.0\"; crates/quarto-source-map/ deleted; 2 Cargo.lock updated. Phase 1 functionally complete; remaining tidy-ups: new repo CI workflow + crates.io owner-add (both deferred)."},"c-ncwi1bbr":{"id":"c-ncwi1bbr","author":"Carlos Scheidegger","created_at":"2026-06-27T21:25:14.981381Z","text":"Phase 2 committed + PR opened. Two commits on feature/bd-egcyeym9-error-reporting-split: (A) refactor — catalog-agnostic + quarto-error-catalog extraction; (B) build — json feature gate. Split cleanly via revert-reapply (the two changes were interleaved in 5 shared files). PR #349: https://github.com/quarto-dev/q2/pull/349 (base main). Local full cargo xtask verify green (14 steps incl WASM); nextest 10240. CI running."},"c-pt5dfdvq":{"id":"c-pt5dfdvq","author":"Carlos Scheidegger","created_at":"2026-06-27T22:19:18.503399Z","text":"Phase 3 COMPLETE (2026-06-27): quarto-error-reporting 0.1.0 PUBLISHED to crates.io; q2 cut over. PR #350: https://github.com/quarto-dev/q2/pull/350. Cutover = pure dep-source flip (path->version 0.1.0) + delete in-tree crate; json wiring from Phase 2 carried over untouched; quarto-error-catalog + 4 json consumers stay in q2. Local: nextest --workspace 10177; full cargo xtask verify green (14 steps incl WASM). CLAUDE.md updated (both foundation crates now in an 'Externalized foundation crates' section). CI running.\n\nBOTH foundation crates now external & published: posit-dev/quarto-source-map 0.1.0 + posit-dev/quarto-error-reporting 0.1.0. NEXT: the YAML stack (quarto-yaml + quarto-yaml-validation) per the sibling plan — the original goal of the epic."},"c-q71dv5xw":{"id":"c-q71dv5xw","author":"Carlos Scheidegger","created_at":"2026-06-27T21:12:36.409886Z","text":"Phase 2 COMPLETE (2026-06-27): quarto-error-reporting is now catalog-agnostic; full cargo xtask verify GREEN (all 14 steps incl WASM + hub tests); workspace nextest 10240 passed.\n\nWhat changed:\n- quarto-error-reporting: catalog.rs gutted of data -> CatalogProvider trait + EmptyCatalog + std OnceLock registry + install_catalog; get_docs_url/get_error_info/get_subsystem keep signatures, delegate to installed provider. ERROR_CATALOG static + include_str removed. once_cell dropped (uses std OnceLock).\n- NEW crate quarto-error-catalog: error_catalog.json (git-moved) + QuartoCatalog provider + install(); 10 data-presence tests ported + 3 install/delegation integration tests. Example moved here.\n- json.rs behind default-off 'json' feature (schemars optional); 4 consumers (quarto, quarto-core, quarto-preview, wasm) opt in. cargo tree confirms schemars absent by default.\n- install() wired into q2 binary main. WASM deliberately does NOT install (catalog would include_str! 46KB into bundle, breaking hub-client 35MiB PWA precache limit; WASM never surfaces docs URLs -> EmptyCatalog is the correct per-embedder choice).\n- 2 quarto-core data-presence tests query quarto_error_catalog::ERROR_CATALOG directly (dev-dep). audit-error-codes.py + ~25 path refs updated to crates/quarto-error-catalog/.\n\nKey insight: catalog is fully decoupled from production rendering (docs_url has 0 consumers; 0 snapshots contain a URL) -> carve-out is behaviour-neutral, no snapshot churn.\n\nUncommitted on branch braid/bd-egcyeym9-error-reporting-split. Phase 3 (extract the crate to posit-dev/) is next."},"c-syi58b2z":{"id":"c-syi58b2z","author":"Carlos Scheidegger","created_at":"2026-06-27T22:32:57.802870Z","text":"PR #350 (Phase 3 cutover) CI fully GREEN: all 5 checks pass (test suite macos x2 + ubuntu x2, Hub-Client E2E). MERGEABLE/CLEAN. Awaiting user merge. Both foundation crates now external+published; YAML stack is the remaining work."},"c-szmu8xji":{"id":"c-szmu8xji","author":"Carlos Scheidegger","created_at":"2026-06-26T18:18:31.559624Z","text":"Discipline refinement 2026-06-26 (cont.2). Folded into claude-notes/designs/cross-package-error-codes.md:\n\nROLES ARE PER-NODE: 'library' vs 'product' collapses into one role applied at each hop. Every node = a DEFINER (mints terminal codes) + optional REMAPPER (relabels dependency codes). Q2 is just the TERMINAL remapper (its codes are user-facing). Chains bottom out at terminal codes; the library contract (every error has a stable code) guarantees every chain terminates.\n\nTERMINAL vs REMAPPED provenance lane added (developer-facing, not user-facing; structured/JSON only). Three rules: (1) provenance is INERT DATA {code, source_url?}, never a typed dependency on the upstream error enum (would rebuild the coupling we're removing); (2) disclose the IMMEDIATE upstream + self-declared terminal flag, NOT a resolved-ultimate pointer (stale-proof under no-traversal; immediate==terminal in the common 1-hop case); (3) NO automated cross-repo traversal and no resolved-ultimate; disclosure is best-effort breadcrumb, optional version/commit pin, no cross-repo CI possible. Design-for-1-hop: permit chaining, build zero resolver.\n\nerror-reporting-core naming flagged: general infra under posit-dev/, likely drops quarto- prefix (candidates: diagnostic-core/reportkit/etc) — deferred, bikeshed.\n\nRemaining nit: terminal as explicit bool flag vs implied-by-absence-of-provenance (lean: implied-by-absence)."},"c-uc2a0w79":{"id":"c-uc2a0w79","author":"Carlos Scheidegger","created_at":"2026-06-27T19:35:27.601940Z","text":"Naming decided (2026-06-27): externalized crates KEEP their current names — quarto-source-map, quarto-error-reporting (and quarto-yaml later); rename only 'if it comes to it'. Non-obvious consequence folded into the foundation plan: keeping the name means the externalized crate IS quarto-error-reporting (no error-reporting-core rename), which KILLS the planned q2-side façade (name collision). Simpler result: the 9 dependents depend on the external quarto-error-reporting directly (imports unchanged); the ONLY q2-side carve-out is quarto-error-catalog (the Q-* catalog DATA + ERROR_CATALOG static + QuartoCatalog provider + install()). json.rs + coalesce.rs STAY in the external crate, json behind a default-off 'json' feature (q2 enables it) -> zero import churn for json/coalesce consumers (wasm/hub/publish/trace/parse-errors/mcp/preview); reverses earlier Q4/Q5 'move json q2-side' but the feature-gate keeps non-Quarto builds schemars-free. Updated docs: foundation plan (full), design note (renamed error-reporting-core -> quarto-error-reporting throughout), yaml plan (superseding banner; its internal façade/core/json-relocation refs are now historical). Remaining open forks: version start (0.1.0 rec), crates.io vs git dep, the two posit-dev/ repo slugs."},"c-wwrizzhk":{"id":"c-wwrizzhk","author":"Carlos Scheidegger","created_at":"2026-06-27T22:02:56.740082Z","text":"quarto-error-reporting repo CI fully GREEN on all 3 OSes after a CRLF fix. Windows initially failed schema_drift (committed schemas/*.json checked out CRLF vs serde_json LF output); fixed with .gitattributes '* text=auto eol=lf' + renormalize. q2's CI (linux+macos only) never caught this — the standalone Windows matrix did. Same latent bug exists in q2's schema_drift copy but is moot (q2 deletes its copy at 3e cutover). Repo ready to publish. NEXT: 3d user publishes quarto-error-reporting 0.1.0 to crates.io."}}} {"id":"bd-ehyyfpjj","title":"revealjs code blocks: highlight spans emitted but highlight CSS missing (uncolored code)","description":"In format: revealjs, code cells get the tree-sitter hl-* span annotations (CodeHighlightStage runs) but render UNCOLORED because the compiled reveal theme CSS contains no .hl-* rules. HTML/q2-preview bundle highlight.scss into styles.css via the highlight_layer in every compile_theme_css variant; the reveal path (quarto-sass assemble_reveal_scss / compile_reveal_theme_css) omits highlight_layer entirely. Reproduces in all three paths (q2 render, q2 preview, hub-client) since it is shared Rust.\n\nhighlight.scss (resources/scss/html/templates/highlight.scss) is self-contained (/*-- scss:rules --*/, literal colors, no theme-variable deps), so the fix is to include the highlight layer in assemble_reveal_scss/compile_reveal_theme_css. Watch CSS specificity: reveal's own .reveal pre code rules may override bare .hl-* selectors; verify computed colors and scope under .reveal if needed.\n\nFix location: crates/quarto-sass/src/bundle.rs assemble_reveal_scss (~382) + crates/quarto-sass/src/compile.rs compile_reveal_theme_css (~374). TDD: assert reveal compiled CSS contains .hl-keyword, mirroring the existing HTML test (~compile.rs:620).","status":"closed","priority":2,"issue_type":"bug","created_at":"2026-06-22T16:11:41.351966Z","created_by":"Carlos Scheidegger","updated_at":"2026-06-22T16:36:50.812802Z","closed_at":"2026-06-22T16:36:50.812802Z","close_reason":"Fixed in 984874f8 (pushed to main): assemble_reveal_scss now bundles the highlight layer so revealjs code blocks get .hl-* colours. Verified end-to-end on q2 render (native) and hub-client/q2 preview (WASM) — identical colours to HTML. TDD test added; full workspace nextest + cargo xtask verify green.","labels":["bug"],"comments":{"c-n344uejb":{"id":"c-n344uejb","author":"Carlos Scheidegger","created_at":"2026-06-22T16:33:01.635122Z","text":"cargo xtask verify (full, incl. WASM rebuild + hub-client build:all + test:ci) GREEN. Verified hub-client preview end-to-end: code blocks now highlighted with identical colours to q2 render (hl-function blue rgb(38,139,210), hl-keyword green rgb(133,153,0), hl-number magenta rgb(211,54,130)). Fix confirmed across all three paths. Branch braid/bd-ehyyfpjj-revealjs-code-highlight-css (commit 561aaed6) ready for review/PR."},"c-wafa88b4":{"id":"c-wafa88b4","author":"Carlos Scheidegger","created_at":"2026-06-22T16:27:28.035816Z","text":"Root cause: assemble_reveal_scss (crates/quarto-sass/src/bundle.rs) omitted the highlight layer that every HTML compile bundles, so the compiled reveal theme had no .hl-* colour rules — the hl-* spans rendered uncoloured. Fix (commit 561aaed6): include load_highlight_layer() in assemble_reveal_scss, before user theme layers. Shared by native + WASM reveal compiles, so it fixes q2 render, q2 preview, and hub-client. Verified q2 render end-to-end (computed colours blue/green/magenta). Full workspace nextest green (10309). cargo xtask verify (WASM + hub-client) running to confirm the WASM-backed paths; will verify hub-client preview after the WASM rebuild."}}} {"id":"bd-eias3e39","title":"Audit _quarto-rules.scss: categorized selector inventory (port-now / blocked-on-emitter / dropped)","description":"Produce the categorized inventory that scopes the rest of epic bd-4doe9lvt. For each of the ~80 top-level selectors in TS Quarto's _quarto-rules.scss, determine:\n\n1. Is the rule ALREADY present in Q2's SCSS (_bootstrap-rules.scss / title-block.scss / copy-code.scss / highlight.scss / embed-example.scss / page-footer)? \n2. Does Q2's HTML writer EMIT the DOM the selector targets? (grep the writers/transforms; render a fixture exercising the feature and inspect.)\n3. Categorize: PORT-NOW (DOM emitted, rule missing) / BLOCKED-ON-EMITTER (rule would be dead CSS until Q2 emits the DOM — link the emitter feature) / ALREADY-PRESENT / INTENTIONALLY-DROPPED (with reason).\n\nDeliverable: a table in claude-notes/research/2026-07-DD-quarto-rules-scss-inventory.md, and follow-up child strands under bd-4doe9lvt: themed PORT-NOW strands (candidate groupings from the initial scan: figures/floats [quarto-figure, quarto-float-caption internals], code-overflow [code-overflow-wrap/scroll], footnotes [footnote-back, tippy footnote], layout panels [quarto-layout-panel], cover-image, unresolved-ref, details/summary, task-list tweaks), and BLOCKED strands with 'blocks' deps on their emitter features.\n\nInitial coverage scan (from bd-btjkyylx session, NOT authoritative — verify each): PRESENT-ish = quarto-layout-cell, quarto-float-caption, title-block-header(now complete), code-copy-outer-scaffold, task-list, tippy, panel-input, quarto-embedded-source-code. MISSING-ish = quarto-layout-panel, quarto-figure, code-overflow-wrap, footnote-back, quarto-cover-image, quarto-unresolved-ref, widget-subarea, knitsql-table, abstract-title, quarto-float-tbl. The MISSING set mixes port-now (Q2 emits the DOM) and blocked (Q2 doesn't yet) — that split is the whole point of the audit.\n\nParent epic: bd-4doe9lvt. Plan: claude-notes/plans/2026-07-21-quarto-rules-scss-parity-epic.md","status":"closed","priority":2,"issue_type":"task","created_at":"2026-07-21T16:00:22.550418Z","created_by":"Carlos Scheidegger","updated_at":"2026-07-21T17:53:14.041657Z","closed_at":"2026-07-21T17:53:14.041657Z","close_reason":"Audit complete: inventory at claude-notes/research/2026-07-21-quarto-rules-scss-inventory.md (commit 129dfedc); 9 follow-up strands filed under/around epic bd-4doe9lvt + task-list bug bd-obkvhlam","labels":["css","parity"],"dependencies":{"bd-4doe9lvt:parent-child":{"depends_on_id":"bd-4doe9lvt","type":"parent-child","created_at":"2026-07-21T16:00:50.909888Z","created_by":"Carlos Scheidegger"}},"comments":{"c-2pb8wvm2":{"id":"c-2pb8wvm2","author":"Carlos Scheidegger","created_at":"2026-07-21T17:25:47.622144Z","text":"Epic plan updated with code pointers for the audit handoff (bundle.rs / resources.rs / writers / compile-test / baseline). Recommend starting this in a fresh session or a dedicated worktree (/investigate-beads bd-eias3e39) — the useful context is fully captured in the plan + this strand."},"c-oc19f712":{"id":"c-oc19f712","author":"Carlos Scheidegger","created_at":"2026-07-21T17:30:59.472422Z","text":"Investigation done (plan skeleton at claude-notes/plans/2026-07-21-quarto-rules-scss-audit.md, commit 04882745 on main). Verdict: ready to design. Note: the 'epic plan updated with code pointers' comment refers to pointers that were never committed — they're reconstructed in the skeleton from the bd-btjkyylx plan. Extracted 144 depth-0 selectors (vs ~80 family-grouped) to plans/quarto-rules-scss-audit-investigation/top-level-selectors.tsv."}}} -{"id":"bd-eiku4ymo","title":"Capture docs: uncompressed audit/GC metadata envelope (createdAt, sourcePath, engines)","description":"Engine-capture binary docs (mimeType application/x-engine-capture+gzip) are orphaned in samod storage on every re-execution: perform_re_execute creates a new binary doc and repoints the index sidecar's CaptureRef, and nothing ever deletes the old doc. On quarto-hub deployments these accumulate forever, and bd-qbhp2cvv (embedding engine supporting-file bytes in captures) will make each one substantially bigger. Proposal: add a small UNCOMPRESSED top-level automerge 'meta' map beside content/mimeType/hash (via a create_binary_document_with_meta variant in quarto-hub/src/resource.rs) with kind:'engine-capture' + schema version, createdAt, sourcePath (project-relative qmd path), and engines (e.g. ['knitr']). Placement matters: mimeType already classifies capture docs without gunzipping, but anything inside the gzipped content payload is invisible to sync-server audits. With provenance in place, a storage-hygiene job gets a safe policy: capture-MIME docs not referenced by any index sidecar AND older than N days -> collect, with per-project accounting. Touches all three capture writers: quarto-preview/src/capture_driver.rs, quarto-preview/src/re_execute.rs (write_capture_doc), quarto-hub-provider/src/execute.rs. Context: discussion recorded in claude-notes/plans/2026-07-23-preview-engine-supporting-files.md (bd-qbhp2cvv).","status":"open","priority":2,"issue_type":"feature","created_at":"2026-07-23T19:03:07.254071Z","created_by":"Carlos Scheidegger","updated_at":"2026-07-23T19:03:07.254071Z","dependencies":{"bd-qbhp2cvv:related":{"depends_on_id":"bd-qbhp2cvv","type":"related","created_at":"2026-07-23T19:03:07.254071Z","created_by":"Carlos Scheidegger"}}} +{"id":"bd-eiku4ymo","title":"Capture docs: uncompressed audit/GC metadata envelope (createdAt, sourcePath, engines)","description":"Engine-capture binary docs (mimeType application/x-engine-capture+gzip) are orphaned in samod storage on every re-execution: perform_re_execute creates a new binary doc and repoints the index sidecar's CaptureRef, and nothing ever deletes the old doc. On quarto-hub deployments these accumulate forever, and bd-qbhp2cvv (embedding engine supporting-file bytes in captures) will make each one substantially bigger. Proposal: add a small UNCOMPRESSED top-level automerge 'meta' map beside content/mimeType/hash (via a create_binary_document_with_meta variant in quarto-hub/src/resource.rs) with kind:'engine-capture' + schema version, createdAt, sourcePath (project-relative qmd path), and engines (e.g. ['knitr']). Placement matters: mimeType already classifies capture docs without gunzipping, but anything inside the gzipped content payload is invisible to sync-server audits. With provenance in place, a storage-hygiene job gets a safe policy: capture-MIME docs not referenced by any index sidecar AND older than N days -> collect, with per-project accounting. Touches all three capture writers: quarto-preview/src/capture_driver.rs, quarto-preview/src/re_execute.rs (write_capture_doc), quarto-hub-provider/src/execute.rs. Context: discussion recorded in claude-notes/plans/2026-07-23-preview-engine-supporting-files.md (bd-qbhp2cvv).","status":"in_progress","priority":2,"issue_type":"feature","created_at":"2026-07-23T19:03:07.254071Z","created_by":"Carlos Scheidegger","updated_at":"2026-07-24T16:32:37.060297Z","dependencies":{"bd-qbhp2cvv:related":{"depends_on_id":"bd-qbhp2cvv","type":"related","created_at":"2026-07-23T19:03:07.254071Z","created_by":"Carlos Scheidegger"}},"comments":{"c-4ipghuue":{"id":"c-4ipghuue","author":"Carlos Scheidegger","created_at":"2026-07-24T14:12:34.813445Z","text":"Scope expanded on review (Carlos, 2026-07-24): in addition to the metadata envelope, design+implement minimal sync-server maintainer tools (hub admin scan/collect/restore/purge) — scan inventories a samod storage location and emits a versioned manifest of safely-removable orphaned capture docs; collect quarantines (never unlinks) from a manifest only, with re-verification; purge is the only unlink, behind a retention window. Full design for review: claude-notes/plans/2026-07-24-capture-meta-and-hub-admin-tools.md (branch braid/bd-eiku4ymo-capture-docs-uncompressed-auditgc)."},"c-58h7udo0":{"id":"c-58h7udo0","author":"Carlos Scheidegger","created_at":"2026-07-24T16:32:37.060297Z","text":"PR opened: https://github.com/quarto-dev/q2/pull/415 (feature/bd-eiku4ymo-hub-admin-tools, 6 commits). All phases done: meta envelope, classifier, scan+manifest, collect/restore/purge with quarantine + AdminLock, hub admin CLI, binary E2E (recorded in plan), runbook. verify --skip-hub-build green. Close on merge."},"c-70tg9rr7":{"id":"c-70tg9rr7","author":"Carlos Scheidegger","created_at":"2026-07-24T14:43:37.091068Z","text":"Phases A-B3 committed (38af41d3 meta envelope; B1 classifier; e1ed2c3b scan+manifest incl. the load_range splay discovery; 4ffc1f8f collect/restore/purge with AdminLock). B4 in progress: hub admin CLI wired; binary E2E first run caught a stale q2 binary whose captures were genuinely unstamped — the tools correctly protected them (live validation of the legacy-capture gate); rerunning with a fresh build. Runbook drafted at claude-notes/instructions/hub-storage-hygiene.md."}}} {"id":"bd-eips","title":"L9 follow-up: format.metadata.description as channel description fallback","description":"Q1 cascades feed.description → format.metadata.description → website.description for the channel description. v1 cascades feed.description → website.description (skips the per-format layer). The simpler cascade matches Q2's configuration model. File this if a user needs the third level. Site: feed/binding.rs::website_description helper.","status":"open","priority":4,"issue_type":"feature","created_at":"2026-05-08T17:33:26.773014Z","created_by":"cscheid","updated_at":"2026-05-08T17:33:26.773014Z","dependencies":{"bd-o90m:discovered-from":{"depends_on_id":"bd-o90m","type":"discovered-from","created_at":"2026-05-08T17:33:26.773014Z","created_by":"cscheid"}}} {"id":"bd-eity","title":"Generic file uploader dialog for hub-client","description":"Hub-client currently only supports dropping images onto the editor or sidebar. Users need to be able to upload arbitrary binary files (PDFs, CSVs, fonts, tree-sitter grammar .wasm files, etc.) into their Automerge-backed projects. The ingestion pipeline is already generic (processFileForUpload → createBinaryFile → VFS); what's missing is the UI affordance — a non-image-specific uploader dialog + triggers.\n\nPlan: claude-notes/plans/2026-04-21-generic-file-uploader.md\n\nBlocks: real-browser end-to-end verification for syntax-highlighting Phase 4 (loading a user tree-sitter grammar from _quarto/grammars/). See claude-notes/plans/2026-04-21-syntax-highlighting-phase-4.md step 4.6.\n\nScope (see plan for full details):\n- Generalize NewFileDialog's file-input accept filter (currently image/*,.pdf,.svg)\n- Route non-image editor drops to the upload dialog instead of discarding them\n- Add a '+' / 'Add files' entry point in the FileSidebar\n- Destination-path picker\n- Preserve existing image-drop markdown-insertion UX\n\nMostly UI-layer work; the binary ingestion pipeline stays unchanged.","status":"closed","priority":1,"issue_type":"feature","created_at":"2026-04-21T13:38:41.407334Z","created_by":"cscheid","updated_at":"2026-04-21T14:55:28.078909Z","closed_at":"2026-04-21T14:55:28.078161Z","close_reason":"Implemented in b0177b8d (plan 2026-04-21-generic-file-uploader)","dependencies":{"bd-n7x2:related":{"depends_on_id":"bd-n7x2","type":"related","created_at":"2026-04-21T13:38:41.407334Z","created_by":"cscheid"}}} {"id":"bd-eizgnxlx","title":"Migrate crossref codeblock-shorthand to the shared cell_options facility","description":"quarto-core/src/crossref/codeblock_shorthand.rs parses leading #| lines with a naive split_once(':') string matcher (no real YAML, no per-option source spans, hardcoded #| prefix) and hand-rewrites the block text. Replace parse_cell_options/partition_options/strip_consumed_lines with quarto_core::cell_options::partition_cell_options (bd-ohvl879u), which provides real YAML parsing, language awareness, and SourceInfo-mapped option spans. Behavior change to review: values like quoted strings / flow collections parse properly instead of string-matching; snapshots may move. Note: faithful source spans for AST-side consumers also want a body-only SourceInfo on CodeBlock (separate strand). Deferred from bd-ohvl879u decision 7.","status":"open","priority":3,"issue_type":"task","created_at":"2026-07-02T17:20:17.970751Z","created_by":"Carlos Scheidegger","updated_at":"2026-07-02T17:20:17.970751Z","dependencies":{"bd-ohvl879u:discovered-from":{"depends_on_id":"bd-ohvl879u","type":"discovered-from","created_at":"2026-07-02T17:20:17.970751Z","created_by":"Carlos Scheidegger"}}} diff --git a/.config/nextest.toml b/.config/nextest.toml index 86cf325fa..b731613b2 100644 --- a/.config/nextest.toml +++ b/.config/nextest.toml @@ -34,6 +34,23 @@ # watcher. Limited to one in-flight process at a time. quarto-preview-fs-watcher = { max-threads = 1 } +# Julia-engine e2e tests share ONE ambient transport file +# (`~/Library/Caches/quarto/julia/julia_transport.txt`): each test uses +# `setup_julia_project()` (temp project dir, but NOT a temp HOME), so +# concurrent `daemon: false` runs each boot their own QNR server and +# overwrite each other's transport entry — a client then reads a stale +# entry and fails the socket handshake with "Incorrect HMAC digest" at +# `isopen`. Observed 2026-07-03 (bd-h4rhohhy P3 verification): rotating +# victims (j1/j2 on one run, j3/j4 on a clean-machine rerun) confirm an +# intra-suite race, not any single bad test. Serializing the suite is +# the config-level fix; full hermetic isolation for the j-tests is +# tracked separately (they belong to the julia-validation plan). +julia-shared-transport = { max-threads = 1 } + [[profile.default.overrides]] filter = "package(quarto-preview) & binary(integration) & test(/^(staleness|eager_capture|boot)::/)" test-group = "quarto-preview-fs-watcher" + +[[profile.default.overrides]] +filter = "package(quarto-core) & binary(integration) & test(/^julia_engine_e2e::/)" +test-group = "julia-shared-transport" diff --git a/.github/workflows/test-suite.yml b/.github/workflows/test-suite.yml index 225a9bcbb..98fac2d41 100644 --- a/.github/workflows/test-suite.yml +++ b/.github/workflows/test-suite.yml @@ -16,6 +16,9 @@ concurrency: env: PANDOC_VERSION: "3.8.3" + # Deno version provisioned for the QUARTO_CI assertion test in ts_process.rs. + # Must provide `Deno.Command`/`Deno.stdin.isTerminal()` (available since Deno 1.40+). + DENO_VERSION: "2.9.0" jobs: test-suite: @@ -119,6 +122,20 @@ jobs: if: runner.os == 'macOS' run: brew install minisign + # Deno — required for ts_process.rs engine tests and the deno_available_when_quarto_ci + # assertion. Plain install (curl + $GITHUB_PATH on Linux; brew on macOS) avoids + # /opt/hostedtoolcache/ so the "Free disk space" invariant below is not broken. + - name: Set up Deno (Linux) + if: runner.os == 'Linux' + shell: bash + run: | + curl -fsSL https://deno.land/install.sh | sh -s -- v${DENO_VERSION} + echo "$HOME/.deno/bin" >> "$GITHUB_PATH" + + - name: Set up Deno (macOS) + if: runner.os == 'macOS' + run: brew install deno + # Free disk space on Linux runners (14 GB SSD is tight for Rust monorepo). # `remove_tool_cache: true` is safe — no step in this job uses /opt/hostedtoolcache/ # (no setup-node, setup-python, etc.). See claude-notes/2026-04-28-ci-disk-space-and-profile-ci.md. @@ -179,6 +196,9 @@ jobs: run: cargo nextest run --tests --cargo-profile ci env: RUSTFLAGS: "-D warnings" + # Turns the silent Deno-skip in deno_available_when_quarto_ci into a hard + # failure if the "Set up Deno" step above regresses. + QUARTO_CI: "1" wasm-tests: name: WASM Tests diff --git a/.github/workflows/ts-test-suite.yml b/.github/workflows/ts-test-suite.yml index fc5f40296..e711fc2cb 100644 --- a/.github/workflows/ts-test-suite.yml +++ b/.github/workflows/ts-test-suite.yml @@ -16,6 +16,8 @@ concurrency: env: PANDOC_VERSION: "3.8.3" + # Deno version for engine-host-deno vitest + deno-test + bundle freshness gate. + DENO_VERSION: "2.9.0" jobs: test-suite: @@ -117,6 +119,14 @@ jobs: shell: bash run: npm ci + # Deno — needed for engine-host-deno deno-test and the freshness gate. + # denoland/setup-deno is safe here: this workflow has no "Free disk space" + # step (and no remove_tool_cache invariant to preserve). + - name: Set up Deno + uses: denoland/setup-deno@v2 + with: + deno-version: ${{ env.DENO_VERSION }} + # WASM build for hub-client (must happen before TypeScript build) - name: Set up Clang (Linux) if: runner.os == 'Linux' @@ -145,3 +155,45 @@ jobs: run: | cd hub-client npm run test:ci + + # engine-host-deno: vitest suite (105+ Node-side harness tests) + - name: Run engine-host-deno tests (vitest) + shell: bash + run: npm run test -w @quarto/engine-host-deno + + # engine-host-deno: deno-native test (the one leg that requires Deno) + - name: Run engine-host-deno deno test + shell: bash + run: deno test --allow-all ts-packages/quarto-engine-host-deno/src/deno-host.deno-test.ts + + # plan1a.6 Phase 2 (Deno dial-back) — seam #8: connectControl round-trip + # over a REAL loopback socket (Deno.listen({ port: 0 }) in-test); no mock + # Deno.Conn. CI-only tier (cannot be a vitest test — the module references + # Deno.*). `--sloppy-imports` lets deno's type-checker resolve + # @quarto/types' `.js` internal specifiers pulled in transitively via + # ./types.ts (same reason the wire-parity step below needs it). + - name: Run engine-host-deno control-transport deno test + shell: bash + run: deno test --allow-all --sloppy-imports ts-packages/quarto-engine-host-deno/src/control-transport.deno-test.ts + + # T-Gate-parity: TS↔Rust wire-dual parity (Plan 2 Phase B gate). Reads the + # Rust-serialized fixture (crates/quarto-core/tests/fixtures/ts_wire_parity.json, + # produced by the regen-gated #[test] in ts_protocol.rs) and set-equates each + # instance's keys against a KEYS list pinned to the TS wire type via + # `satisfies`/`_Exhaustive`. `--sloppy-imports` lets deno's type-checker + # resolve @quarto/types' `.js` internal specifiers (so the compile guards + # are checked here, not just the runtime set-equality). @quarto/types is + # mapped in the repo-root deno.jsonc. + - name: Run engine-host-deno wire-parity deno test + shell: bash + run: deno test --allow-all --sloppy-imports ts-packages/quarto-engine-host-deno/src/wire-parity.deno-test.ts + + # Bundle freshness gate: rebuild engine-host-deno.js from source and assert + # the committed bytes are unchanged. Fails if someone edited TS but forgot + # to run `npm run bundle`. build-info.json is gitignored (volatile builtAt); + # the diff targets only the bundle itself. + - name: Check engine-host-deno bundle freshness + shell: bash + run: | + npm run bundle -w @quarto/engine-host-deno + git diff --exit-code -- ts-packages/quarto-engine-host-deno/dist/engine-host-deno.js diff --git a/.gitignore b/.gitignore index b1c5179f1..0540ec1e2 100644 --- a/.gitignore +++ b/.gitignore @@ -25,6 +25,14 @@ node_modules/ **/.quarto/hub/hub.json ts-packages/*/dist/ ts-packages/*/dist-bundle/ +# Exception: the engine-host-deno bundle is embedded into the q2 binary via +# include_str! (plan1a-host "Bundle embedding"), so the single bundle file must +# be tracked even though it lives under an otherwise-ignored dist/. A committed +# placeholder lets fresh clones compile; Plan 1b overwrites it with the real +# esbuild output (same path, new bytes). Other dist/ artifacts stay ignored. +!ts-packages/quarto-engine-host-deno/dist/ +ts-packages/quarto-engine-host-deno/dist/* +!ts-packages/quarto-engine-host-deno/dist/engine-host-deno.js ts-packages/*/*.tsbuildinfo q2-demos/*/dist/ crates/wasm-quarto-hub-client/pkg/ @@ -57,3 +65,10 @@ CLAUDE.local.md # local-prod mode data directory .local-prod-data/ + +# Hermetically-regenerated TS engine extension bundles (built at test time +# via crate::engine_fixture_build; see plan1c3 Task 6). legacy-python's +# committed stub is unaffected — a gitignore entry cannot untrack an +# already-tracked file. +crates/quarto-core/tests/fixtures/extensions/*/dist/ +resources/extension-build/deno.lock diff --git a/.superpowers/sdd/1c-task-1-report.md b/.superpowers/sdd/1c-task-1-report.md new file mode 100644 index 000000000..ccf4e35ce --- /dev/null +++ b/.superpowers/sdd/1c-task-1-report.md @@ -0,0 +1,106 @@ +# Task 1 Report — Engine-contribution data types + static-claim → LanguageClaim conversion + +## Status + +DONE + +## Files Changed + +- `crates/quarto-core/src/extension/types.rs` — sole file modified + +## What Was Added + +### New types +- `EngineContribution` enum (`External { path, name, claims, file_extensions, claims_files }` + `Reorder { name }`) +- `StaticLanguageClaim` struct (`kind`, `priority`, `when_class`) +- `ClaimKind` enum (`Primary`, `Interop`, `Fallback`) + +### New field on `Contributes` +```rust +pub engines: Vec, +``` +All three existing `Contributes { .. }` literals already used `..Default::default()`, so no manual updates were needed: +- `crates/quarto-core/src/filter_resolve.rs:488` +- `crates/quarto-core/src/transforms/shortcode_resolve.rs:2048` +- `crates/quarto-core/src/stage/stages/metadata_merge.rs:1630` + +### New functions +- `static_claim_to_language_claim(claim, first_class) -> LanguageClaim` +- `lookup_static_claim(claims, language, first_class) -> LanguageClaim` + +### `LanguageClaim` derives +Already had `#[derive(Debug, Clone, Copy, PartialEq, Eq)]` at `engine/mod.rs:104` — no changes needed. + +## TDD Sequence + +**RED** — wrote tests with two stubs: +- `static_claim_to_language_claim`: converted correctly but ignored `when_class` (always converted) +- `lookup_static_claim`: always returned `Primary(1)` ignoring map contents + +Ran `cargo nextest run -p quarto-core -E 'test(extension::types::tests::static_claim) or test(extension::types::tests::lookup)'`: +- 5 PASS (positive conversion cases — stubs handled those correctly) +- 4 FAIL (the required RED cases): + - `static_claim_when_class_mismatch_returns_none`: got `Primary(1)`, expected `None` + - `static_claim_when_class_mismatch_no_first_class_returns_none`: got `Primary(1)`, expected `None` + - `lookup_absent_language_returns_none`: got `Primary(1)`, expected `None` + - `lookup_present_mismatched_when_class_returns_none`: got `Primary(1)`, expected `None` + +**GREEN** — replaced stubs with correct implementations: + +`static_claim_to_language_claim`: added `when_class` guard before the `match`: +```rust +if let Some(ref required) = claim.when_class { + if first_class != Some(required.as_str()) { + return LanguageClaim::None; + } +} +``` + +`lookup_static_claim`: proper absent-check + delegation: +```rust +match claims.get(language) { + None => crate::engine::LanguageClaim::None, + Some(claim) => static_claim_to_language_claim(claim, first_class), +} +``` + +## Test Results + +### New tests (15 total in `extension::types::tests`) +``` +cargo nextest run -p quarto-core -E 'test(extension::types)' +Summary [0.062s] 15 tests run: 15 passed, 2556 skipped +``` + +Tests added (9 new, 6 pre-existing): +1. `static_claim_primary_no_when_class_default_priority` — `Primary(1)` default ✓ +2. `static_claim_primary_no_when_class_explicit_priority` — `Primary(5)` explicit ✓ +3. `static_claim_interop_and_fallback_default_priority` — `Interop(0)`, `Fallback(0)` ✓ +4. `static_claim_when_class_match_converts` — `"marimo"` == `"marimo"` → `Primary(1)` ✓ +5. `static_claim_when_class_mismatch_returns_none` — `"marimo"` != `"python"` → `None` ✓ (P1-14 binding) +6. `static_claim_when_class_mismatch_no_first_class_returns_none` — `"marimo"` != `None` → `None` ✓ (P1-14 binding) +7. `lookup_absent_language_returns_none` — absent key → `None` ✓ +8. `lookup_present_matching_when_class_converts` — present + match → converts ✓ +9. `lookup_present_mismatched_when_class_returns_none` — present + mismatch → `None` ✓ + +Pre-existing tests also updated: +- `test_contributes_default`: added `assert!(c.engines.is_empty())` ✓ + +### Broader regression check +``` +cargo nextest run -p quarto-core -E 'test(extension::) or test(engine::)' +Summary [6.699s] 460 tests run: 460 passed, 2111 skipped +``` + +### Build verification +``` +cargo build -p quarto-core +Finished `dev` profile [optimized + debuginfo] target(s) in 2.78s +``` +No warnings, no errors. All three existing `Contributes` literals compile correctly via `..Default::default()`. + +## Notes + +- No `serde` derives added (Task 2 owns YAML parsing). +- No changes to `parse_contributes`, `TsEngine`, or resolution code. +- `LanguageClaim` needed no derive additions. diff --git a/.superpowers/sdd/1c-task-11-report.md b/.superpowers/sdd/1c-task-11-report.md new file mode 100644 index 000000000..099d91292 --- /dev/null +++ b/.superpowers/sdd/1c-task-11-report.md @@ -0,0 +1,99 @@ +# Task 11 Report — P2-12 + P2-13 + +## Summary + +Both P2-12 and P2-13 are implemented, tested GREEN, and all 2595 quarto-core +tests pass. Clippy reports zero warnings. + +--- + +## P2-13 (implemented earlier in this session) + +### What changed + +`partition_cells` in `crates/quarto-core/src/engine/jupyter/text_execute.rs` +gained a `multi_engine: bool` third parameter. When `false` (single-engine +sequence), owned-but-unrunnable cells are passed through unexecuted rather than +raising `NoHandlerForLanguage`. When `true` (multi-engine), the existing loud +error fires. + +`ExecutionContext` gained `multi_engine: bool` (default `false`) and +`with_multi_engine(bool)`. `engine_execution.rs` computes `let multi_engine = +to_run.len() > 1` before the `into_iter()` move and passes it via +`.with_multi_engine(multi_engine)`. + +### Tests (P2-13) + +| Test | File | Result | +|------|------|--------| +| `test_partition_cells_owned_unrunnable_fails_loudly` | `text_execute.rs` | GREEN (updated to `multi_engine=true`) | +| `test_partition_cells_single_engine_owned_unrunnable_passthrough` | `text_execute.rs` | GREEN (new, `multi_engine=false` → Ok) | +| `test_partition_cells_cede` | `text_execute.rs` | GREEN (updated to pass `false`) | +| `test_partition_cells_execute` | `text_execute.rs` | GREEN (updated to pass `false`) | +| `test_partition_cells_mixed` | `text_execute.rs` | GREEN (updated to pass `false`) | + +--- + +## P2-12 — Registered owning engine unavailable → loud error + +### What changed + +`get_engine_with_fallback` in `engine_execution.rs` return type changed from +`Arc` to `Result, PipelineError>`. + +New behaviour matrix: + +| Registered? | `is_available()` | In `spliced_engines`? | Result | +|-------------|------------------|----------------------|--------| +| Yes | true | — | `Ok(engine)` | +| Yes | false | No | **`Err(PipelineError::stage_error(...))`** ← P2-12 | +| Yes | false | Yes | `Ok(markdown)` silently (capture replay) | +| No | — | No | `Ok(markdown)` + warning | +| No | — | Yes | `Ok(markdown)` silently | + +`run()` now propagates the error via `?` at the call site. + +### Blast-radius analysis + +One test asserted the now-obsolete silent-fallback contract: + +| Test | File | What changed | Why | +|------|------|-------------|-----| +| `q2_preview_without_capture_still_warns_unavailable_engine` | `pipeline.rs` | Renamed to `q2_preview_without_capture_errors_unavailable_engine`; assertion changed from "Ok + `not available` warning" → "Err + engine name in message" | Was asserting the old silent-fallback behaviour P2-12 intentionally removes | + +Three unit tests call `get_engine_with_fallback` directly (all test the +UNREGISTERED path, which still returns `Ok`): + +| Test | Change | Why | +|------|--------|-----| +| `test_engine_fallback_with_unavailable_engine` | Added `.expect()` | Return type changed to `Result` | +| `test_spliced_engine_suppresses_fallback_warning` | Added `.expect()` | Return type changed to `Result` | +| `test_unspliced_engine_still_warns_when_sibling_spliced` | Added `.expect()` (both calls) | Return type changed to `Result` | + +These tests still exercise unregistered engines (else branch), which return +`Ok(markdown)` both before and after P2-12 — their semantics did not change. + +### New P2-12 tests + +| Test | File | RED → GREEN | +|------|------|-------------| +| `test_p2_12_owning_engine_unavailable_fails_loudly` | `engine_execution.rs` | RED (result.is_ok(), expected Err) → GREEN | +| `test_p2_12_spliced_unavailable_engine_still_silent` | `engine_execution.rs` | Was already PASS (spliced path unchanged); GREEN from start | + +### Vacuity check + +Vacuity revert described in test comment: removing the `is_available()` gate +(falling back to markdown for ALL registered engines) causes +`test_p2_12_owning_engine_unavailable_fails_loudly` to fail with "got Ok (old +silent-fallback behaviour)". Confirmed at RED run before implementation. + +--- + +## Test counts + +- Tests added: 2 (P2-12: `test_p2_12_owning_engine_unavailable_fails_loudly`, + `test_p2_12_spliced_unavailable_engine_still_silent`) +- Tests modified: 1 renamed + 1 assertion changed (`q2_preview_without_capture_*`) + + 4 `.expect()` additions + 5 P2-13 arg updates +- Total quarto-core: **2595 passed, 0 failed** +- Clippy: **0 warnings** diff --git a/.superpowers/sdd/1c-task-2-report.md b/.superpowers/sdd/1c-task-2-report.md new file mode 100644 index 000000000..449b499b5 --- /dev/null +++ b/.superpowers/sdd/1c-task-2-report.md @@ -0,0 +1,99 @@ +# Task 2 Report — Parse `contributes.engines` + `.js` validation + warning emitter + +## Status: DONE + +## Commit + +`7d0d047ab` on `feature/ts-engine-extensions` + +## Files changed + +- `crates/quarto-core/src/extension/read.rs` — added `parse_engines`, `parse_external_engine`, + `parse_claims_map`, `parse_static_language_claim`, `parse_string_list`; updated + `parse_contributes` to call them and extended the "at least one" check; 17 new tests. +- `crates/quarto-core/src/extension/types.rs` — added `engine_contribution_missing_fields_warning` + and its `use quarto_error_reporting::DiagnosticMessage` import; 6 new tests. + +## TDD RED → GREEN sequence + +### Phase 1 — types.rs warning emitter (P1-11) + +**RED**: Added 6 tests in `extension::types::tests` that call +`engine_contribution_missing_fields_warning`. Compile error: +``` +error[E0425]: cannot find function `engine_contribution_missing_fields_warning` in this scope + --> crates/quarto-core/src/extension/types.rs:395:17 +``` +(6 identical errors, one per call site) + +**GREEN**: Implemented the function. Logic: match on `External`, collect which of +`name`/`claims`/`file_extensions`/`claims_files` are `None` (not `Some(empty)`), format a +`DiagnosticMessage::warning` naming those fields; return `None` for `Reorder` or a fully-declared +`External`. + +### Phase 2 — read.rs engine parser (P1-8, P1-9, happy path, None/Some(empty), shorthand, +engines-only) + +**RED**: Added 8 new tests in `extension::read::tests`. Compile warnings (unused imports for +`EngineContribution`/`ClaimKind`/`StaticLanguageClaim`) plus runtime failures — all +`test_engine_*` tests would panic or error because the engine parsing code didn't exist yet. + +**GREEN**: Implemented the full engine parsing chain in `read.rs`. + +## Test list and counts + +``` +cargo nextest run -p quarto-core -E 'test(extension::)' +68 tests run: 68 passed, 2517 skipped +``` + +New tests added (17 total): + +**read.rs (11 new)** +- `test_engine_ts_path_rejected` (P1-8) +- `test_engine_uppercase_js_rejected` (P1-9a) +- `test_engine_mjs_path_rejected` (P1-9b) +- `test_engine_external_happy_parse` +- `test_engine_claims_present_but_empty_is_some` +- `test_engine_absent_optional_fields_are_none` +- `test_engine_claims_shorthand_forms` +- `test_engines_only_extension_is_valid` + +**types.rs (6 new)** +- `warning_names_missing_name_field` (P1-11) +- `warning_names_missing_claims_field` (P1-11) +- `warning_names_missing_file_extensions_field` (P1-11) +- `warning_names_missing_claims_files_field` (P1-11) +- `no_warning_when_all_fields_present_even_empty` (P1-11, Some(empty) = declared) +- `no_warning_for_reorder_variant` (P1-11) + +## Verification commands and output + +``` +cargo build -p quarto-core + Finished `dev` profile [optimized + debuginfo] target(s) in 3.14s + +cargo nextest run -p quarto-core -E 'test(extension::)' + Summary [0.253s] 68 tests run: 68 passed, 2517 skipped + +cargo nextest run --workspace --exclude wasm-qmd-parser + Summary [69.746s] 10474 tests run: 10474 passed, 197 skipped +``` + +## Implementation notes + +- `parse_static_language_claim` uses `yaml_rust2::Yaml` enum variants directly (Scalar arm + pattern-matches on `Boolean(false)`, `Boolean(true)`, `Integer(n)`) since that's what the + `ProjectConfig` interpretation context produces. +- The `fallback` key in `claims` gets special treatment: its object form is `{ priority?: int }` + with kind implicitly `Fallback` (no `kind` field in the YAML). Other keys use the full + `{ kind, priority?, whenClass? }` form. +- `file-extensions` and `claims-files` use the hyphen YAML keys; the struct fields use + snake_case (`file_extensions`, `claims_files`). +- The "at least one sub-field" error message was extended to include `engines` in the list. +- `engine_contribution_missing_fields_warning` writes the message into `w.title` (the + `DiagnosticMessage::warning(msg)` API) — tests assert on `w.title.contains("field-name")`. + +## Concerns + +None. All brief requirements implemented as specified. diff --git a/.superpowers/sdd/1c-task-5-report.md b/.superpowers/sdd/1c-task-5-report.md new file mode 100644 index 000000000..1435e798d --- /dev/null +++ b/.superpowers/sdd/1c-task-5-report.md @@ -0,0 +1,49 @@ +# Task 5 Report — Infra leaves + +## Status: COMPLETE + +## Files changed + +- `crates/quarto-util/src/data_dir.rs` — new module: `data_dir_from()` pure helper + `quarto_data_dir()` IO wrapper + 5 tests +- `crates/quarto-util/src/lib.rs` — added `pub mod data_dir;` + `pub use data_dir::quarto_data_dir;` +- `crates/quarto-system-runtime/src/traits.rs` — added `is_interactive()` (default `false`) + `running_in_ci()` (default reads `env_get("CI")`) to `SystemRuntime` trait; added 5 tests with `CiMockRuntime` inline mock +- `crates/quarto-system-runtime/src/native.rs` — added `is_interactive()` override (`std::io::stdin().is_terminal()`); added `is_interactive_native_false_under_nextest` test + +## Verification commands and output + +``` +cargo nextest run -p quarto-util -p quarto-system-runtime +``` + +136 tests run: **136 passed, 0 skipped** + +New tests by file: +- `quarto-util data_dir::tests::data_dir_from_both_none_returns_none` PASS +- `quarto-util data_dir::tests::data_dir_from_data_dir_branch_last_component_is_quarto` PASS +- `quarto-util data_dir::tests::data_dir_from_falls_back_to_data_dir_with_quarto_suffix` PASS +- `quarto-util data_dir::tests::data_dir_from_override_wins_and_is_used_as_is` PASS +- `quarto-util data_dir::tests::quarto_data_dir_returns_existing_directory` PASS +- `quarto-system-runtime traits::tests::running_in_ci_true_for_nonempty_value` PASS +- `quarto-system-runtime traits::tests::running_in_ci_true_for_one` PASS +- `quarto-system-runtime traits::tests::running_in_ci_false_for_empty_string` PASS +- `quarto-system-runtime traits::tests::running_in_ci_false_when_not_set` PASS +- `quarto-system-runtime traits::tests::is_interactive_default_is_false` PASS +- `quarto-system-runtime native::tests::is_interactive_native_false_under_nextest` PASS + +``` +cargo build -p quarto-core +``` + +Clean — no existing `impl SystemRuntime` required updating (both new methods have defaults). + +## Design decisions + +**`QUARTO_DATA_DIR` override semantics**: honored as-is (no `quarto` suffix appended). The `quarto` suffix is only appended to the `dirs::data_dir()` fallback branch. This mirrors Q1's `quartoDataDir()` which treats `QUARTO_DATA_DIR` as the quarto data root directly. Documented in the `data_dir_from` doc-comment and asserted in `data_dir_from_override_wins_and_is_used_as_is`. + +**`running_in_ci` test isolation**: used a minimal inline mock (`CiMockRuntime`) that controls only `env_get("CI")`. All other required methods are `unimplemented!()`. This avoids reading or mutating the real process environment — no parallel-test races. + +**`is_interactive` true-path not unit-tested**: `NativeRuntime::is_interactive()` delegates to `std::io::stdin().is_terminal()`. The `true` path requires an actual PTY, which nextest does not provide. The test asserts `false` under nextest (correct — no TTY) and notes the true path is not covered by unit tests. This is acceptable: the implementation is a one-liner with no logic to test beyond the bool flip. + +## Out of scope + +`HostGlobalConfig` construction (Task 7) — not touched. `QUARTO_DATA_DIR` is not consumed anywhere except `quarto_data_dir()`. diff --git a/.superpowers/sdd/1c-task-6-report.md b/.superpowers/sdd/1c-task-6-report.md new file mode 100644 index 000000000..493c104f4 --- /dev/null +++ b/.superpowers/sdd/1c-task-6-report.md @@ -0,0 +1,66 @@ +# Task 6 Report — Registry primitives: `contribution_order` + engine-shutdown machinery + +## Status: COMPLETE + +## Commit + +`9191eb1b0` on branch `feature/ts-engine-extensions` + +## Changes + +Three files touched: + +### `crates/quarto-core/src/engine/traits.rs` +Added `ExecutionEngine::shutdown()` default method (no-op, returns `Ok(())`) with full +idempotency contract documented in the doc comment. Placed after `quarto_required()`, +before the trait's closing `}`. + +### `crates/quarto-core/src/engine/ts_engine.rs` +Added `TsEngine::shutdown()` override near the `quarto_required()` override (~line 728): +```rust +fn shutdown(&self) -> Result<(), ExecutionError> { + self.host.shutdown() +} +``` +Delegates directly to `TsEngineHost::shutdown()`, which is already idempotent via +`Option::take()` guards on all subprocess handles. + +### `crates/quarto-core/src/engine/registry.rs` +- Added `use super::ExecutionError;` import. +- Added `pub contribution_order: Vec` field to `EngineRegistry` struct (with doc + comment explaining consumer intent). +- Initialized `contribution_order: Vec::new()` in all three struct-literal constructors: + `new()`, `empty()`, and `with_replay_many()`. +- Added `shutdown_all(&self) -> Result<(), ExecutionError>` method: best-effort iteration + over all engines, returns first error, continues through the rest. + +## Tests (TDD — RED then GREEN) + +Three tests added to `engine::registry::tests`: + +| Test | Gate | Result | +|------|------|--------| +| `test_shutdown_all_noop_on_builtins` | always | PASS | +| `test_contribution_order_roundtrip` | always | PASS | +| `test_shutdown_all_kills_ts_engine` | `deno_is_available()` | PASS (Deno ran) | + +Command: +``` +cargo nextest run -p quarto-core -E 'test(engine::registry) or test(engine::ts_engine::tests::shutdown)' +``` +Output: `16 tests run: 16 passed, 2587 skipped` + +The Deno-gated test (`test_shutdown_all_kills_ts_engine`) ran and passed. It: +1. Spawned a real subprocess via `TsEngineHost::start_with_command(sh -c 'cat >/dev/null', ...)` +2. Asserted `host.is_alive() == true` (exercised-guard) +3. Wrapped the host in a `TsEngine`, registered it in a `EngineRegistry::empty()` +4. Called `registry.shutdown_all()` +5. Asserted `host.is_alive() == false` + +Build verification: +- `cargo build -p quarto-core` — clean (no errors, no warnings) +- `cargo build -p quarto-core --tests` — clean (no errors, no warnings) + +## Concerns + +None. Implementation is exactly as specified in the brief. No scope creep. diff --git a/.superpowers/sdd/1c-task-7b-report.md b/.superpowers/sdd/1c-task-7b-report.md new file mode 100644 index 000000000..989e2c7e4 --- /dev/null +++ b/.superpowers/sdd/1c-task-7b-report.md @@ -0,0 +1,155 @@ +# Task 7b Report — Build real engine registry on ProjectContext + +## Status: COMPLETE + +Commit: `bd0e7dde9` + +--- + +## Construction sequence as built + +All changes in `crates/quarto-core/src/`: + +### `lib.rs` +- Added `pub fn version() -> &'static str { env!("CARGO_PKG_VERSION") }` for `HostGlobalConfig.quarto_version`. + +### `project/mod.rs` + +**New function `build_engine_registry` (native-only, `#[cfg(not(target_arch = "wasm32"))]`)** + +Signature: +```rust +fn build_engine_registry( + extensions: &[Extension], + binary_dependencies: &BinaryDependencies, + runtime: &dyn SystemRuntime, +) -> Result> +``` + +Steps implemented exactly per brief: + +1. **HostGlobalConfig**: `resource_dir` from `BUILTIN_EXTENSIONS.path()` (empty if None), `runtime_dir`/`data_dir` from `quarto_util::quarto_runtime_dir()`/`quarto_data_dir()` (IO errors propagated as `QuartoError`), `pandoc_path` from `binary_dependencies.pandoc`, `is_interactive_session`/`running_in_ci` from `runtime`, `quarto_version` from `crate::version()`. +2. **`Arc::new(global)`** — NOT spawned (cheap; no subprocess). +3. **`EngineRegistry::new()`** — built-ins markdown/knitr/jupyter. +4. **Per-extension contribution loop**: + - `Reorder { name }` → push name to `order` vec (no register). + - `External { path, name, claims, file_extensions, claims_files }`: + - 4a: `!path.exists()` → `Err("…no bundled .js file… Run 'q2 build-ts-extension'…")` + - 4b: `key = name.unwrap_or(ext.id.to_string())`, `name_declared = name.is_some()` + - 4c: `registry.has_engine(&key)` → collision `Err("…both '{}' and '{}'…")` naming both contributors (tracked in `key_to_contributor: HashMap`, built-ins pre-seeded as "built-in") + - 4d: `TsEngine::new(…)`, `registry.register(Arc::new(engine))`, push `key` to `order` + - 4e: `engine_contribution_missing_fields_warning(…)` → push to `registry.diagnostics` if Some +5. **`contribution_order`**: dedup first-occurrence from `order`. Comment left for Task 9 `_quarto.yml` engines splice. +6. **Validation**: for each name in `contribution_order`, if `!registry.has_engine(name)` → `Err("'{}' was specified in the list of engines… Available engines are: …")` (sorted, joined). +7. **Return** `Arc::new(registry)`. Comment: `// Task: drain registry.diagnostics at orchestrator (plan step 10)`. + +**`ProjectContext::discover` updated**: +- Captures `single_file_input = input_file.clone()` before `input_file` is consumed into `files`. +- Computes `binary_dependencies` before the registry build. +- `discovery_anchor`: single-file → `single_file_input` (file path, `start_dir = dir`); project → `dir.join("_quarto.yml")` (parent = `dir`, so `start_dir = dir`). +- `builtin_dir`: native = `BUILTIN_EXTENSIONS.path()`, WASM = None. +- Calls `discover_extensions(anchor, project_dir_opt, builtin_dir, runtime)`. +- Native: `registry = build_engine_registry(&extensions, &binary_dependencies, runtime)?`. +- WASM: `registry = Arc::new(EngineRegistry::new())`. + +**`ProjectContext::single_file` updated**: +- Same pattern: compute `binary_dependencies`, `builtin_dir`, `extensions`, then `registry` (native/WASM gated). + +--- + +## Seams bound + +| Seam | Test | Status | +|------|------|--------| +| P1-1: engine registered | `p1_1_extension_engine_appears_in_engine_names` | BOUND | +| P1-5 (reg half): declared name + zero-spawn | `p1_5_named_engine_registered_without_spawn` | BOUND | +| P1-6 (alias reg half): ext-id key | `p1_6_unnamed_engine_registered_under_ext_id` | BOUND | +| P1-4: collision names both contributors | `p1_4_name_collision_errors_and_names_both_contributors` | BOUND | +| P1-3: unknown reorder lists available | `p1_3_unknown_reorder_hint_errors_listing_available` | BOUND | +| P1-2: contribution_order populated | `p1_2_contribution_order_contains_declared_engines` | BOUND | +| Warning: missing static fields in diagnostics | `warning_missing_static_fields_appears_in_diagnostics` | BOUND | +| Bundle-missing: Err mentions build-ts-extension | `bundle_missing_errors_with_build_ts_extension_hint` | BOUND | + +## Seams deferred (per brief) + +- **P1-7**: name-mismatch fires at first LoadEngine — Task 14 / mock-load test. +- **P1-5 full resolution** (engine: echo with no-spawn resolution) — Task 9. + +--- + +## Test counts + +- New tests: **8** (all in `engine_registry_build.rs` behind `#[cfg(not(target_arch = "wasm32"))]`) +- Total quarto-core tests: **2578 passed, 0 failed, 33 skipped** + +--- + +## Pre-existing test interactions + +No pre-existing tests tripped the new validation. Checked: `cargo nextest run -p quarto-core` ran 2578 tests with 0 failures. The existing `project_pipeline` and related tests use temp dirs without `_extensions/` subdirectories, so `discover_extensions` returns an empty vec and `build_engine_registry` produces the same built-ins-only registry as before. + +--- + +## Exact commands + output + +``` +cargo build -p quarto-core → Finished (no errors/warnings) +cargo build -p quarto-core --tests → Finished (no errors/warnings) +cargo nextest run -p quarto-core -E 'test(engine_registry_build)' + → 8 tests run: 8 passed, 2603 skipped +cargo nextest run -p quarto-core + → 2578 tests run: 2578 passed, 33 skipped +``` + +--- + +## Notes + +- **`quarto_version`**: uses `quarto-core`'s `CARGO_PKG_VERSION` (not the `quarto` binary crate). Both track the workspace release version — acceptable per brief. +- **WASM path**: `build_engine_registry` is native-only; WASM `discover` / `single_file` keep `EngineRegistry::new()` (built-ins only). Extension discovery runs on WASM but only format/filter contributions matter there. +- **`discovery_anchor` for multi-file projects**: uses `dir.join("_quarto.yml")` whose parent is `dir`, ensuring `discover_extensions` starts its walk at the project root. File need not exist; `Path::parent()` is purely path arithmetic. + +--- + +## Fix pass (review findings) — commit `8d20b9def` + +### Changes + +**`crates/quarto-core/src/project/mod.rs`** + +1. **Lazy host construction** (`build_engine_registry`): + - Extracted `any_external_engine(extensions: &[Extension]) -> bool` (native-only, `#[cfg(not(target_arch = "wasm32"))]`): scans the extension list for at least one `EngineContribution::External`. `Reorder`-only and empty lists return `false`. + - Restructured `build_engine_registry`: the `needs_host = any_external_engine(extensions)` predicate gates the entire `HostGlobalConfig` / `TsEngineHost` construction block. When `false`, only Reorder hints are harvested — `quarto_runtime_dir()` / `quarto_data_dir()` are never called. + - All existing step numbering and behavior preserved for the `needs_host = true` path. + +2. **Shared discovery helper** (`discover_extensions_and_build_registry`): + - New function factoring the `builtin_dir` + `discover_extensions` + `build_engine_registry` (or `EngineRegistry::new()` on WASM) block, called by both `discover` and `single_file`. + - Eliminates ~14 duplicated lines. + +3. **No change to the WASM path**: the new helper correctly uses `Arc::new(EngineRegistry::new())` on wasm32. + +**`crates/quarto-core/tests/integration/engine_registry_build.rs`** + +- **`p0_no_extension_project_builds_builtins_only`**: new integration test — project with `_quarto.yml` but no `_extensions/` → `discover` succeeds, registry contains all three built-ins, `contribution_order` is empty. +- **`p1_5_named_engine_registered_without_spawn`**: added comment explaining the test binds only REGISTRATION (not spawn-count), and that the no-spawn guarantee is structural + covered by TsEngine unit tests (T4 P1-12). + +**`crates/quarto-core/src/project/mod.rs` (unit tests)** + +- **`needs_host_tests` submodule** (4 tests, `#[cfg(not(target_arch = "wasm32"))]`): + - `needs_host_false_for_no_extensions` — `any_external_engine(&[])` returns false. + - `needs_host_false_for_reorder_only` — Reorder-only extension → false. + - `needs_host_true_for_external_engine` — External engine → true. + - `needs_host_true_when_external_mixed_with_reorder` — External + Reorder → true. + +### Covering tests + result + +``` +cargo nextest run -p quarto-core -E 'test(engine_registry_build)' + → 9 tests run: 9 passed (was 8; p0_no_extension_project_builds_builtins_only added) + +cargo nextest run -p quarto-core -E 'test(needs_host)' + → 4 tests run: 4 passed (all new any_external_engine predicate tests) + +cargo nextest run -p quarto-core + → 2583 tests run: 2583 passed, 33 skipped (was 2578; +5 new tests) +``` diff --git a/.superpowers/sdd/1c-task-9-report.md b/.superpowers/sdd/1c-task-9-report.md new file mode 100644 index 000000000..bdb14a736 --- /dev/null +++ b/.superpowers/sdd/1c-task-9-report.md @@ -0,0 +1,129 @@ +# Task 9 Report — Make `resolve_engines` DRIVE execution + +## Status: COMPLETE + +## Test summary + +``` +cargo nextest run -p quarto-core +Summary [20.811s] 2586 tests run: 2586 passed, 33 skipped +``` + +`cargo clippy -p quarto-core --all-targets` — 0 errors, 0 warnings after fixing the `mut` lint on `raw_explicit`. + +## Changes made + +### 1. `EngineExecutionStage::run` — execution driven by `resolution.sequence` + +**File:** `crates/quarto-core/src/stage/stages/engine_execution.rs` + +- Removed the `detect_engine_sequence` call at step 1; `resolve_engines` now receives `ctx.claimed_engine_name.as_deref()` instead of `None`. +- The `to_run` loop iterates `resolution.sequence` (each `DetectedEngine`) rather than the old `sequence.engines`. +- Removed the `dropped_duplicates` warning loop — `resolve_engines` returns a de-duplicated sequence by construction. +- The fast path (empty `to_run` → passthrough) is preserved. +- `handled_languages_for` + `.with_handled_languages` wiring is unchanged (P2-8 already wired). + +### 2. `resolve_engines` claimed short-circuit (P2-10) + +**File:** `crates/quarto-core/src/engine/resolution.rs` + +Added at the very top of `resolve_engines`, before any tier logic: + +```rust +if let Some(name) = claimed { + return EngineResolution { + sequence: vec![DetectedEngine::new(name)], + ownership: LinkedHashMap::new(), + }; +} +``` + +Deleted the old seed handling (`explicit_with_seed`/`seed` contributed to `present`). Simplified `is_implicit` to `!has_engine_key && raw_explicit.is_empty()` (the `claimed.is_none()` clause is gone — the short-circuit above makes it unreachable). + +Fixed `mut raw_explicit` → `raw_explicit` (clippy lint). + +### 3. `contribution_order` in `candidate_engines` + +**File:** `crates/quarto-core/src/engine/resolution.rs` + +Added a splice between the explicit list and `BUILTIN_ORDER`: + +```rust +for name in ®istry.contribution_order { + let name = name.as_str(); + if !seen.contains(name) && registry.has_engine(name) { + seen.insert(name); + order.push(name); + } +} +``` + +Extension engines registered via `registry.register()` are now promoted ahead of `knitr`/`jupyter`/`markdown` in the candidate order. The `is_implicit` gate is unchanged — auto-promotion does not disable T4. + +### 4. Delete `KNOWN_ENGINES` / `is_known_engine` + +**File:** `crates/quarto-core/src/engine/detection.rs` + +- Deleted `KNOWN_ENGINES` const. +- Deleted `is_known_engine` function. +- Deleted `test_is_known_engine` and `test_detect_engine_top_level_key` / `test_detect_engine_top_level_knitr` tests (replaced by resolver-level tests for top-level key via registry). +- The top-level-key scan in `detect_engines` now uses `registry.engine_names()` (passed in as a slice) instead of `KNOWN_ENGINES`. + +**File:** `crates/quarto-core/src/engine/mod.rs` + +- Removed `KNOWN_ENGINES` and `is_known_engine` from re-exports. + +## New tests added (in `resolution.rs`) + +All binding the seams called out in the brief: + +- **P2-1** — `{julia}` cells + julia `Primary(1)` engine → `sequence == [julia]` +- **P2-2** — `engine: markdown` on a doc with `{r}` cells → `sequence == [markdown]` (explicit beats knitr tier) +- **P2-4** — `{notaknownlang}` cell, no claimer → `sequence == [jupyter]` (implicit-Fallback) +- **P2-5** — no executable cells → `sequence` is empty +- **P2-7** — `{r}`+`{python}` → `sequence == [knitr]`, `ownership[python] == knitr` (Interop) +- **P2-9** — pure `{python}`, no python extension → `sequence == [jupyter]` (knitr absent, presence-gated) +- **P2-10** — `claimed = Some("echo")`, front-matter `engine: knitr`, `{echo}`+`{python}` cells → `sequence == [echo]`; `engine: knitr` ignored; `{python}` NOT owned by a second engine +- **contribution_order auto-promotion** — unlisted extension engine with same-kind/same-priority claim as built-in wins tiebreak by contribution_order position +- **top-level key via registry** — top-level `:` key with extension engine registered selects that engine + +## Existing tests updated + +All existing engine-execution and preview-record tests that relied on the engine sequence being driven by metadata alone (without code cells) were updated to reflect the Task 9 behavioral change: **engines only appear in the sequence if they claim at least one cell language from the ORIGINAL AST.** + +The core change: every mock/probe/passthrough engine needs `claims_language` implemented, AND every test document needs code cells for the engine to claim. + +### `engine_execution.rs` test updates + +- `MockIncludesEngine` and `MockAppendingEngine`: added `claims_language` returning `Primary(1)` for their own language names. +- `test_unknown_engine_falls_back`: removed the diagnostic assertion — with no cells, the sequence is empty and no warning fires (correct new behavior: no cells = nothing to execute = no warning). +- `test_duplicate_engine_dedups_and_warns`: removed the "Duplicate engine 'fixture-a'" diagnostic assertion — de-duplication now happens silently in `candidate_engines`. +- `test_two_engines_run_in_sequence_with_handoff`, `test_multi_engine_trace_records_per_engine_snapshots_and_captures`, `test_multi_engine_record_then_replay_is_byte_clean`: rewritten — the "engine A generates engine B cells at runtime" handoff pattern is incompatible with resolution-driven execution (sequence is fixed from original AST). Both `{fixture-a}` and `{fixture-b}` cells are now present in the ORIGINAL document. +- Several other tests: added `{engine-name}` cells to content. + +### `preview_record.rs` test updates + +- `PassthroughTestEngine`: added `claims_language` returning `Primary(1)` for `"test-passthrough"`. Test content already had `{test-passthrough}` cells; this was the only missing piece. + +### `replay_engine.rs` integration test updates + +- `capture_engine_input` helper's `ProbeEngine`: added `claims_language` returning `Primary(1)` for `self.name`. +- `replay_capture_in_options_overrides_engine_through_render_to_file`: added `{replay-only-engine-4b}` cell to QMD file content. The `capture_engine_input` probe now runs and captures the serialized QMD; the replay pass matches it and returns the recorded markdown. +- `replay_capture_miss_surfaces_as_render_error`: added `{replay-only-engine-4b}` cell. `ReplayEngine` is now in the sequence (it already had `claims_language`), runs, finds `input_qmd` mismatch → "replay miss" error as expected. + +### `pipeline.rs` test updates + +- `test_render_qmd_to_html_uses_replay_registry_from_config`: `ProbeEngine` now has `claims_language`; content updated with `{replay-only-engine}` cell. Two-pass probe+replay pattern still works. +- `q2_preview_without_capture_still_warns_unavailable_engine`: **strategy changed** from using unregistered `replay-only-engine` to a purpose-built `AlwaysUnavailableEngine` (registered, `is_available()=false`, `claims_language("always-unavailable")=Primary(1)`). This tests the behavior deterministically regardless of whether R/Python runtimes are installed. Content updated with `{always-unavailable}` cell and custom registry passed via `engine_registry` parameter. + +### `project_resources.rs` integration test updates + +- `orchestrator_drains_replay_engine_report_to_output_dir`: `ProbeEngine` now has `claims_language` for `"replay-real-pipeline-engine"`; QMD file content updated with `{replay-real-pipeline-engine}` cell. The probe captures the new serialized input (with cell); the replay capture's `result.markdown` is unchanged (engine output replaces the cell with processed markdown). + +## Deferred: `set_project` / per-render `EngineProjectContext` + +Per the brief, the per-render `EngineProjectContext` setup (`set_project` on TS engines before `execute`) is deferred to a follow-up / Plan 4. It is **inert for Plan 1c's tests** (echo ignores project context; `ensure_launched` uses `unwrap_or_default()`). This is a known Phase-2 completeness gap, NOT a silent omission — noted here explicitly. + +## Concerns + +None. All 2586 tests pass; 0 clippy warnings. diff --git a/.superpowers/sdd/task-p0-report.md b/.superpowers/sdd/task-p0-report.md new file mode 100644 index 000000000..5a4186452 --- /dev/null +++ b/.superpowers/sdd/task-p0-report.md @@ -0,0 +1,407 @@ +# Task P0 report — reproduce Bug A / Bug B / Bug C (bd-h4rhohhy) + +**Status: DONE.** All three defects reproduced deterministically on this machine +(2026-07-02). Harnesses committed (diagnosis-only; zero product changes). Fix +SHAPES proposed below for the controller checkpoint — **no fixes implemented.** + +Commit: `2931d7692 test(preview-capture): P0 repro harnesses for Bug A/B/C (bd-h4rhohhy)` + +Files: +- `crates/quarto-core/tests/integration/ts_process_framing_probe.rs` (Bug C, new) +- `crates/quarto-core/tests/integration/main.rs` (register, +1 line) +- `crates/quarto-core/tests/integration/julia_engine_e2e.rs` (PC4a / Bug A) +- `q2-preview-spa/e2e/engine-capture-splice.spec.ts` (PC5 / Bug B, new) + +--- + +## Verdict table + +| Bug | Reproduced? | Deterministic? | Root cause status | +|-----|-------------|----------------|-------------------| +| A (close/busy) | YES (verbatim below) | YES | Root-caused (plan + confirmed live) | +| B (capture → pane) | YES, in **chromium** | YES (fails by timeout every run) | Localized to delivery chain; NOT Bug C. Precise link = P2 | +| C (wire framing) | Reader framing triaged | YES (3 probes) | Reader escalation confirmed; leak source engine-side (P1) | + +**Does Bug C explain Bug B? NO — definitively.** PC5 reproduces Bug B with the +**echo** engine, which spawns no julia child, emits no `ts_process` error, and +the capture IS recorded server-side. Bug C and Bug B are independent defects +(see §Bug B and §Bug C). + +--- + +## Bug A — oneShot close hits a busy worker → capture discarded + +**Reproduced: YES, verbatim.** Deterministic via two concurrent oneShot renders +of one sleeping-cell doc sharing one julia server (the second render's pre-run +`close` collides with the first's still-running worker). Proven live via the +`q2` binary under an isolated HOME (never touching the user's real server +pid 9828); codified as `pc4a_shared_server_busy_close` (in-process, +`#[ignore]` + `QUARTO_PC4A_LIVE=1`, isolated HOME). + +### Repro transcript (binary probe, 2026-07-02) + +Isolation env (protects the user's server + reuses the pre-instantiated depot): +``` +HOME= JULIA_DEPOT_PATH=~/.julia +QUARTO_JULIA_PROJECT=~/Library/Caches/quarto/julia +PATH=:$PATH # NOT the juliaup shim (drifts under temp HOME) +``` +Doc `sleepy.qmd`: `engine: julia`, `execute: {daemon: false}`, cell `sleep(25)\n1 + 1`. +Render A (background) starts the isolated server (transport up in 4s), worker +busy on `sleep(25)`; render B (same file, +6s) → **verbatim**: + +``` +Rendering single file: …/sleepy.qmd +Error: Execution failed in julia: Julia server returned error after receiving "close" command: + +Failed to close notebook: …/sleepy.qmd + +The underlying Julia error was: + +Tried to close file "…/sleepy.qmd" but the corresponding worker is busy. + +1 error +``` + +This is the exact user-reported failure. The pre-run close is +`executeJulia` julia-engine.ts:703-718 (`isopen`→`close`); there is no +busy handling anywhere in the engine (plan grep). The error propagates through +`render_to_file` and the whole render/capture is discarded. + +### Root cause (verified file:line, per plan + confirmed live) +`~/src/quarto-julia-engine/src/julia-engine.ts`: +- pre-run close: `:703-718` (oneShot/restart → `isopen`→`close`, no busy guard) +- post-run close: `:742-749` (oneShot → `close`, no busy guard — latent: can + discard a capture whose run SUCCEEDED) +- `startOrReuseJuliaServer` `:330-448` reuses ANY existing transport file + regardless of `oneShot` (`:440-446`), so a busy/orphaned shared worker is + reached by a fresh oneShot render. + +### Proposed fix SHAPE (needs controller ratification before PC4 freeze) +- **PC1** (post-run close, `:742-749`): a post-run close failure after a + successful run must be **non-fatal** — warn + return the run result. +- **PC2** (pre-run close, `:703-718`): a pre-run close-busy must NOT surface a + bare protocol error. Either **recover** or **fail with an actionable message** + naming the stale-server/transport remedy. + - **Concrete recovery lead for P1:** julia-engine.ts already exposes a + forceful close — the CLI `close` command calls `closeWorker(file, force)` + with a `--force` option (`:~1002-1003`). QNR therefore supports a forced + close; the pre-run close could pass `force: true` (or fall back to it on + busy). P1 should confirm the QNR socket-command surface for `close` accepts + a force flag before committing to recovery-vs-actionable-message. +- **Frozen PC4 post-fix assertion (controller signs off at fix time):** render B + either SUCCEEDS via forced close, or FAILS with the PC2 actionable-remedy + substring — never the bare `"worker is busy"` protocol error. (My harness's + pre-fix assertion is `msg.contains("worker is busy")`; the fix flips it.) + +--- + +## Bug B — recorded capture never reaches the browser pane + +**Reproduced: YES, in chromium** (the user saw it in Firefox; it reproduces in +chromium too — a notable finding, it is NOT browser-specific). Deterministic: +`engine-capture-splice.spec.ts` (PC5) fails by timeout on every run. + +### Repro transcript (PC5, `test.fail()` temporarily disabled to harvest evidence) + +Real `q2 preview` + chromium, temp project with the committed **echo** engine +and `index.qmd` containing one `{echo}` cell with source `PC5_ECHO_SOURCE_TOKEN`. +The pane renders the INERT source, then times out (15s) waiting for the executed +marker `ECHO_EXECUTED`: + +``` +Error: pane must show the executed echo marker after the capture splices in; pane text was: + + PC5 echo capturePC5 headingPC5_ECHO_SOURCE_TOKEN + +console: +[log] WASM module initialized successfully, template loaded +[log] Waiting for peer connection... +[log] Peer connected - online mode +[warning] An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing. + +Expected: true +Received: false +``` + +Server side (independent manual `q2 preview` run, `RUST_LOG=quarto_preview=debug`) +— the capture IS recorded and the sidecar written: +``` +INFO quarto_preview::capture_driver: recorded engine capture(s) rel_path=index.qmd engines=echo +INFO quarto_preview::capture_driver: recorded engine captures count=1 +# data-dir/captures/.bin written (gzip EngineCapture) +``` + +### Boundary evidence / where it breaks +- **Server**: capture recorded + `IndexDocument::set_capture` writes the sidecar + (`capture_driver.rs:184-205`). WORKING. +- **Browser**: SPA WASM initialized, **"Peer connected - online mode"** (samod + sync is up), doc rendered — but the pane shows only the inert source, and + there is **no capture-related console log and no error**. The executed marker + never splices in. +- Because the eager capture is recorded at server startup (before the browser + connects), the SPA should receive it via the **initial** `onCapturesChange` + (`quarto-sync-client/src/client.ts:779-781` / `:1351-1353`, fired off the + IndexDocument's `captures` map). Captures ride on the IndexDocument + (`getCapturesFromIndex`, `:339-364`); the SPA is synced to that doc (it + rendered), so the sidecar entry should be visible on first fire. + +**Conclusion: Bug B is an independent delivery-chain defect, NOT Bug C.** Root +cause NOT determined at P0 (that is P2's job); it is localized to the +`set_capture → samod → onCapturesChange → PreviewApp → getBinaryDocById → WASM +splice` chain, browser-side of "Peer connected". Candidate links for P2, in +descending suspicion: +1. The **capture BINARY doc** (a separate samod doc referenced by + `captureDocId`, written by `write_capture_doc` capture_driver.rs:326) is not + synced/resolvable to the SPA — `getBinaryDocById` (client.ts:~1007-1019) + returns nothing and the splice silently no-ops (consistent with "no error, + no log"). +2. Initial `onCapturesChange` fires before the render is ready and the + `contentTick` bump (PreviewApp.tsx:729-738) is lost / the render effect + (`:1005-1030`) does not re-fire. +3. `state.activeFile` key vs the sidecar `rel_path` key mismatch in the render + effect (plan candidate). + +### Proposed fix SHAPE +Deferred to P2 by design (plan §P2: "location unknown until P0"). P2's entry +point = PC5's failure + this boundary evidence; recommended first step is +targeted SPA instrumentation (console.log at `onCapturesChange`, +`getBinaryDocById`, and the render effect — **rebuild the SPA per the binding +rules**, revert before commit) to identify the silent link, then a minimal fix +on that link. **Frozen post-fix assertion = PC5 as written** (`ECHO_EXECUTED` +appears in the pane without reload; remove `test.fail()`), plus the julia leg +PC6 and the jsdom-tier PC7. + +--- + +## Bug C — engine-host stdout wire-frame corruption + +**Reader framing triaged (the P0 mandate); leak source is engine-side (P1).** +Three deterministic probes (`ts_process_framing_probe.rs`, deno-gated), all GREEN: + +``` +PASS ts_process_framing_probe::pc_c_a_large_single_line_frame_parses +PASS ts_process_framing_probe::pc_c_b_foreign_line_is_malformed +PASS ts_process_framing_probe::pc_c_b_prime_interleaved_bytes_corrupt_frame +3 tests run: 3 passed +``` + +### Findings +- **(a) Large-frame suspect RULED OUT.** A >1 MB single-line frame parses to + `Ok(Response)` — `BufRead::read_line` has no size cap and loops the 8 KB + BufReader buffer to the terminating `\n`. The reader does NOT truncate or + mis-split large frames. So the "legit executeResult frame rejected" symptom + is NOT a large-frame reader bug. +- **(b) Foreign / interleaved bytes → `RecvError::Malformed`.** A stray ANSI + julia log line on the wire (symptom #1) and foreign bytes spliced into a + frame's middle (symptom #2) both fail framing at `StdioReadHalf::recv` + (ts_process.rs:292-308). Both live symptoms are the **same** root cause: a + foreign writer on the engine-host's stdout fd. +- **Catastrophic escalation (by reading, ts_process.rs:930-954):** a single + `Malformed` makes `reader_loop` set `shutting_down`, **broadcast an error to + EVERY pending slot, and kill the whole Deno subprocess.** One stray line + destroys the entire engine host and every in-flight capture. This is why the + user "sometimes sees no result" — the executeResult is dropped AND the host + dies. + +### Leak source (engine-side; candidate, not the P0 mandate) +- **Ruled out for the live session:** the `Deno.stdout.writeSync` sites in + julia-engine.ts (`:1035` `logStatus`, `:1056` `printJuliaServerLog`) are + **Cliffy CLI subcommand handlers**, not the engine-host module path — they do + not fire when julia-engine.ts is imported as an engine. +- **Candidate:** `start_quartonotebookrunner_detached.jl` runs + `run(detach(cmd), wait = false)` with **no stdio redirection**, so the + detached QNR server inherits fds; its startup banner (`[ Info: Log started + at …`) can land on an inherited stdout. Precise mechanism is engine-side + forensics owned by P1 — not required to close the P0 reader triage. + +### Proposed additive seam row (controller sign-off required) + +| ID | Tier | Real unit | Seam → assertion | Mock boundary | Revert hunk → RED | +|----|------|-----------|------------------|---------------|-------------------| +| PC-C | int-rs, deno-gated (framing) + unit-ts (demux resilience) | `StdioReadHalf::recv` framing + `reader_loop` Malformed arm | (framing, DONE) >1MB frame → `Ok`; foreign/interleaved line → `Malformed`. (resilience, POST-FIX) a stray non-JSON line does NOT kill the host and does NOT fail an unrelated in-flight request; a following valid frame is still delivered | none (real pipe) for framing; MockReadHalf for resilience | revert the log-and-skip resilience hunk → one stray line kills all pending → RED | + +### Proposed fix SHAPE (two independent, both need sign-off) +1. **Engine-side (P1/upstream) — the root fix:** ensure NO child process + inherits the engine-host's stdout fd. The detached julia launcher must + redirect the server's stdout/stderr to its log file / devnull rather than + inherit, so a QNR banner can never reach the protocol channel. +2. **Reader-side defense-in-depth (quarto-core, `reader_loop:930-954`) — + POLICY CHANGE, explicit sign-off:** make the reader resilient to a stray + non-protocol line (bounded log-and-skip) instead of `Malformed → kill-all`, + so one leaked banner does not discard every in-flight capture. This changes + the current "compromised channel ⇒ kill subprocess" contract; the comment at + `:930-935` (finding #7, "one terminal error per exit") documents the + intent, so a change here must be deliberate. + +--- + +## Ruled-out hypotheses (evidence, not assumption) +- **"Large executeResult frame is rejected by the reader."** RULED OUT — probe + `pc_c_a` shows a 2 MB single-line frame parses to `Ok`. +- **"Bug C is the root of Bug B."** RULED OUT — PC5 reproduces Bug B with echo + (no julia child, no `ts_process` error, capture recorded server-side). +- **"Bug B is Firefox-specific."** RULED OUT as a necessary condition — it + reproduces in chromium. +- **"The julia-engine.ts `Deno.stdout.writeSync` calls corrupt the wire in the + live session."** RULED OUT — they are CLI-only, not on the engine module path. + +## Constraints honored +- No product code changed (harnesses/probes only; `git status` clean but for the + 4 committed files). Never pushed. +- PC4a ran only under an isolated temp HOME; its julia server (pid 1478) was + killed after the probe. The user's server (pid 9828) and the pre-existing + leaked workers (bd-l9jhy5u0; they use the user's transport file / TempDir + projects, not my isolated HOME) were left untouched. +- PC5 uses `test.fail()`; PC4a uses `#[ignore]` + `QUARTO_PC4A_LIVE` opt-in; + the Bug C probe is deno-gated and GREEN — none break the default suites. +- SPA + WASM + q2 binary were rebuilt (per the binding rebuild rules) before the + browser-tier PC5 run; the first run had shown the placeholder SPA. + +--- + +# Fix wave (review response — task-p0-review.md, 2026-07-02) + +Addresses the four Important findings. Still P0 (harnesses/diagnosis only; the +one product-file touch — PreviewApp.tsx instrumentation — was quarantined and +reverted; `git status` shows no product changes). + +## Fix #1 + #2 — PC4a rewritten to the specified scenario, with cleanup + +`pc4a_shared_server_busy_close` → **`pc4a_abandoned_worker_close_busy`**. Now +matches the plan §P0 spec exactly: +- Drives **`record_capture`** (`quarto_core::engine::preview_record::record_capture`), + not `render_to_file` — the soft-fail caller path the bug actually lives on. +- The first run's client is **ABANDONED mid-run**: its Deno engine-host is + killed (identified by the isolated-HOME bundle path in its cmdline) while the + worker is provably executing, closing the QNR socket → EPIPE → the worker is + left **orphaned-busy** (QNR does not cancel the task). Then a **fresh** + `record_capture` of the same doc → oneShot pre-run close hits the abandoned + worker. +- **Real signal, not a timer** (addresses the Minor): the cell writes a sentinel + file before sleeping; the harness abandons the client only after the sentinel + appears (worker provably mid-run). The old flat `sleep(6)` guess is gone. +- **Cleanup (Fix #2):** an `IsolatedJuliaServerGuard` Drop kills the detached + server's **process group** (pid read from the isolated transport file) on + scope exit, **even on panic** — so no server leaks per run. `#[cfg(unix)]` + gated (process groups); `#[ignore]` + `QUARTO_PC4A_LIVE=1` opt-in unchanged. + +### New verbatim failure (record_capture #2, live 2026-07-02) +``` +Stage 'engine-execution' failed: Execution failed in julia: Julia server returned error after receiving "close" command: + +Failed to close notebook: /var/folders/…/T/.tmpncK0gc/sleepy.qmd + +The underlying Julia error was: + +Tried to close file "/var/folders/…/T/.tmpncK0gc/sleepy.qmd" but the corresponding worker is busy. +``` +Note the entry point is now `Stage 'engine-execution' failed` (the +`record_capture` pipeline), not a CLI render — the correct soft-fail path. + +### Cleanup verified +`pgrep -f quartonotebookrunner.jl | wc -l` = **4 before, 4 after** the run; the +only survivors use the user's transport file (pre-existing bd-l9jhy5u0 workers). +The isolated server the harness started was killed by the guard — no net leak. + +### Fix-shape update (scenario distinction, per review #1) +The **PC4 frozen assertion targets the ABANDONED-worker scenario**: the fresh +`record_capture` either RECOVERS via a forceful close (→ succeeds) or fails with +the actionable PC2 remedy — never the bare `"worker is busy"`. Force-close is +correct here **because the worker is abandoned**. The **concurrent-live-render** +case (where force-close would harm a legitimate in-flight execution) is +explicitly OUT of PC4's scope — it is the plan's documented-not-gold-plated +`oneShot`-server-reuse design question for the upstream PR (plan §P1). P1 must +not let a force-close recovery kill a *live* peer's worker; scoping the recovery +to detectably-abandoned workers (or to the actionable-message path) is the safe +default. + +## Fix #3 — PC5 sync-client state harvested (re-ranks the P2 candidates) + +Quarantined instrumentation (reverted before commit) stashed every +`onCapturesChange` payload on `window.__pc5CaptureLog` and logged the render +effect's capture lookup; `window.__renderTicks` (a production counter) gave the +render count. Harvested from the PC5 failure: + +``` +sync-client state: +{ + "captureLog": [ + { "keys": ["index.qmd"], + "captures": { "index.qmd": { "captureDocId": {"val":"31WYwMd3ZW8Xv92aSghTL6oT1xLo"}, + "staleness": false } } } + ], + "renderTicks": 1 +} +console: + PC5-DIAG onCapturesChange ["index.qmd"] + PC5-DIAG renderEffect {"activeFile":"index.qmd","captureKeys":["index.qmd"], + "hasRef":true,"docId":"31WYwMd3ZW8…","gotBinary":true,"bytes":567} +``` + +**This CHANGES the P2 candidate ranking decisively.** The entire delivery chain +is confirmed WORKING, end to end: +- `onCapturesChange` fired with the entry keyed `index.qmd` (sidecar synced to + the SPA). ⇒ RULES OUT "sidecar not delivered". +- `activeFile == "index.qmd" == sidecar key` ⇒ RULES OUT the activeFile-vs-rel_path + key-mismatch candidate. +- `hasRef: true` ⇒ the capture ref resolved in state. +- `gotBinary: true, bytes: 567` ⇒ `getBinaryDocById` SUCCEEDED and the gzipped + capture bytes were fetched ⇒ RULES OUT "capture binary doc not synced/resolvable" + (my prior #1 candidate). +- `renderTicks: 1` ⇒ the render effect fired (once, WITH the capture present) ⇒ + RULES OUT "contentTick effect not re-firing". + +`renderPageForPreview("index.qmd", undefined, captureGzJson[567])` was therefore +called **with** the capture bytes, yet the pane rendered the **inert** source +(no `ECHO_EXECUTED`). **The break is inside the WASM `render_page_for_preview` +ReplayEngine splice**, downstream of everything the delivery chain does. + +### Revised P2 candidate ranking (Bug B) +1. **PRIMARY — WASM ReplayEngine splice rejects/ignores the capture.** Strongest + sub-candidate: the **canonical `input_qmd` staleness check in WASM replay** + (the plan's explicitly "accepted-untested" item) rejects the capture on a + byte mismatch between the recorded `input_qmd` and what the WASM recomputes, + silently falling back to the default markdown engine → inert render. (Note the + sidecar's own `staleness:false` is the SERVER's flag; the WASM replay applies + its OWN canonical-input check — they are independent.) The 567 bytes reaching + WASM but no splice is exactly this signature. +2. Secondary — ReplayEngine construction from `captureGzJson` (gunzip/parse) + fails silently WASM-side. + +Bug B's fix therefore most likely lands in **quarto-core's WASM replay path** +(`render_page_for_preview` / `ReplayEngine`), requiring a hub WASM rebuild +(plan §P2). This retires the plan's "add a seam only if P0 diagnosis proves it's +Bug B's cause" condition on the canonical-input staleness seam: **P0 now +implicates it** — P2 should add that seam. + + +## Fix #4 — full `npm run test:e2e` proves the new spec doesn't break the suite + +Ran the entire suite once (`q2` re-embedded from the reverted/clean SPA source, +verified `strings target/debug/q2 | grep -c PC5-DIAG` = 0): + +``` +37 passed (28.9s) +1 failed +``` + +- **PC5 (`engine-capture-splice.spec.ts`, #30) is among the 37 passed** — its + `test.fail()` records the pre-fix RED as an EXPECTED failure; it is NOT in the + failures detail section. +- The **1 failed is pre-existing and orthogonal**: `firefox-ws-queue.spec.ts` + under the **firefox** project fails to launch because Firefox is not installed + on this machine (`browserType.launch: Executable doesn't exist … firefox-1522/ + firefox/Nightly.app`). The SAME spec **passes under chromium** (#31 ✓). My + changes touch nothing in that spec. The brief provisioned chromium only. + +Net: the new PC5 spec does not break the default e2e suite. + +## Files (fix wave) +- `crates/quarto-core/tests/integration/julia_engine_e2e.rs` — PC4a rewritten + (`pc4a_abandoned_worker_close_busy`): `record_capture` + client-abandonment + + sentinel signal + `IsolatedJuliaServerGuard` cleanup. +- `q2-preview-spa/e2e/engine-capture-splice.spec.ts` — PC5 now also harvests + `renderTicks` (+ `__pc5CaptureLog` when diagnostic instrumentation is present). +- `q2-preview-spa/src/PreviewApp.tsx` — instrumentation was added, harvested, + and **reverted** (no net change; confirmed clean in the binary). diff --git a/.superpowers/sdd/task-p1-report.md b/.superpowers/sdd/task-p1-report.md new file mode 100644 index 000000000..5e148efba --- /dev/null +++ b/.superpowers/sdd/task-p1-report.md @@ -0,0 +1,258 @@ +# Task P1 report — Bug A fix + Bug C engine root + PC4 RED→GREEN (bd-h4rhohhy) + +**Status: DONE_WITH_CONCERNS.** Both engine defects fixed upstream on +`q2-close-busy-fix` (not pushed), wired into the q2 fixture, PC1/PC2/PC4 +RED→GREEN proven. One concern: the PC4a *live harness* isolation is imperfect +on macOS (see §7). + +- Upstream `~/src/quarto-julia-engine` `q2-close-busy-fix` @ **93bce7b** — + "Recover from busy/failed oneShot worker close; redirect detached server stdio" +- q2 worktree `braid/bd-h4rhohhy-q2-preview-engine-capture` @ **4efa5be84** — + "Wire Bug A/C engine fix into julia fixture; flip PC4a to recovery (bd-h4rhohhy)" + +--- + +## 1. Decision gate — QNR force-close surface (work item 2) + +**Answer: YES — QNR exposes a forceful close.** Evidence (file:line, upstream): +- `julia-engine.ts:774` — `ServerCommand` union includes + `| { type: "forceclose"; content: { file: string } }` +- `julia-engine.ts:783` — response map `forceclose: { status: true }` +- `julia-engine.ts:1100-1106` — `closeWorker(file, force)` sends + `type: force ? "forceclose" : "close"`; the CLI `close --force` help reads + "This will terminate the worker if it is running." +- **Live confirmation:** the existing upstream smoke test + `force-closing a running worker` is GREEN. + +**Selected decision branch: YES = recovery** (per the plan's ratified rule). +Pre-run close falls back to `forceclose` on busy; **frozen PC4 assertion = the +fresh `record_capture` SUCCEEDS with a real capture.** The answer fits the YES +branch cleanly — no NEITHER-branch ambiguity, no controller escalation needed. +Countersign requested on this evidence. + +--- + +## 2. PC1/PC2 — upstream deno TDD (work items 3, 4) + +**Seam:** the close orchestration was extracted into a pure `src/worker-close.ts` +module (`preRunClose` / `postRunClose` over an injectable `CloseCommandWriter`), +so the busy-recovery logic is unit-testable with a mocked command writer — +exactly the frozen "QNR socket/`writeJuliaCommand`" boundary. `executeJulia` now +calls these helpers through a thin adapter over `writeJuliaCommand`. + +Tests: `tests/unit/julia-engine/worker-close.test.ts` (6 tests). `run-tests.{sh,ps1}` +now discover `tests/unit/` alongside `smoke/`. + +### RED (pre-fix extraction reproduces Bug A at the unit tier) + +The module was first written as a verbatim extraction of the inline `:703-718` / +`:742-749` logic (no busy handling). Both busy tests failed with the exact QNR +message: + +``` +running 6 tests from ./tests/unit/julia-engine/worker-close.test.ts +PC1: a busy post-run close after a successful run is non-fatal ... FAILED +PC2: a busy pre-run close recovers via forceclose ... FAILED +... +error: Error: Julia server returned error after receiving "close" command: + ... + Tried to close file "/tmp/sleepy.qmd" but the corresponding worker is busy. + at postRunClose (src/worker-close.ts:40:9) + at preRunClose (src/worker-close.ts:31:11) +FAILED | 4 passed | 2 failed +``` + +### GREEN (fix applied) + +``` +running 6 tests from ./tests/unit/julia-engine/worker-close.test.ts +PC1: a busy post-run close after a successful run is non-fatal (warns, resolves) ... ok +PC1: a clean post-run close does not warn ... ok +PC2: a busy pre-run close recovers via forceclose and does not throw ... ok +PC2: a non-busy pre-run close error is NOT swallowed (propagates, no forceclose) ... ok +PC2: a closed (not-open) file skips the close entirely ... ok +isWorkerBusyError matches the QNR busy message and nothing else ... ok +ok | 6 passed | 0 failed +``` + +- **PC1** (post-run, `:742-749`): a failed cleanup close after a *successful* + run warns (`quarto.console.warning`) and returns the result — non-fatal. Mock + fails ONLY the close (honors the vacuity note structurally: `postRunClose` + never sends `run`, and `executeJulia` still awaits the run directly, so run + errors cannot be swallowed). +- **PC2** (pre-run, `:703-718`): a busy `close` falls back to `forceclose` + (frozen assertion = the sequence `isopen→close→forceclose` and no throw). A + **non-busy** close error still propagates (binds the `isWorkerBusyError` + scoping — reverting the guard reddens this test). + +Full upstream suite (`tests/run-tests.sh`): **9 passed** (2 existing smoke +suites incl. `force-closing a running worker` + 6 new unit tests), 0 failed. + +--- + +## 3. Bug C engine-side root fix (work item 5) + +`start_quartonotebookrunner_detached.jl`: +`run(detach(cmd), wait=false)` → `run(pipeline(detach(cmd), stdout=devnull, +stderr=devnull), wait=false)`. The detached QNR server no longer inherits the +launcher's (hence the Deno engine-host's) stdout/stderr, so its early output +can't land on the JSON protocol channel. `quartonotebookrunner.jl` still logs to +its own `logfile` via its internal pipe → no diagnostics lost. Redirecting to +the logfile instead would race QNR's own `open(logfile,"w")` truncation, so +devnull is the conflict-free choice. + +**No deno-mock regression test** — the launcher is a Julia subprocess and the +fd-inheritance behavior is an OS/Julia-runtime property, not mockable in deno. +Covered by: the PC-C framing probes (GREEN at P0) + PC4a live GREEN end-to-end +(the whole capture survives through the real launcher). Windows path (PowerShell +`Start-Process -WindowStyle Hidden`) does not inherit stdio → unaffected, left +as-is. + +--- + +## 4. Rebundle into q2 (work item 6) + +- Upstream rebundle: `quarto call build-ts-extension src/julia-engine.ts` + (78 modules, 45323 B). Verified fix markers present (`forceclose`, + "returning results anyway", 2× `devnull`). +- Fixture rebundle: copied `src/julia-engine.ts`, new `src/worker-close.ts`, + and `start_quartonotebookrunner_detached.jl` into + `crates/quarto-core/tests/fixtures/extensions/julia-engine/`; rebuilt via the + compat-log §4 temp-symlink workaround (`q2 build-ts-extension + _extensions/julia-engine`, symlink removed after). +- **Byte-identity property survives:** the q2 build and the Q1 `quarto call` + build of the fixed source produce **identical** bundles — + `82bff64cc5d060cb48983945060a6932`, 45323 B (was `d9d5120…`, 44512 B). +- Engine-host bundle NOT rebuilt (no engine-host TS changed). + +--- + +## 5. PC4 freeze + RED→GREEN (work item 7) + +`pc4a_abandoned_worker_close_busy` (julia_engine_e2e.rs) assertion flipped from +`expect_err("worker is busy")` to the **frozen YES-branch**: `record_capture` +#2 must **succeed**, and (non-vacuous) the returned julia capture's +`result.markdown` must contain `cell-output` (proves the recovered run actually +executed the cell). Docstring updated to record the selected branch. + +### RED — flipped assertion vs the PRE-fix bundle (`d9d5120…`) + +``` +thread 'pc4a_abandoned_worker_close_busy' panicked at julia_engine_e2e.rs:1046:9: +PC4a: fresh record_capture against the ABANDONED busy worker must SUCCEED +post-fix (pre-run close recovers via forceclose); got Err: +Stage 'engine-execution' failed: Execution failed in julia: Julia server +returned error after receiving "close" command: +... +Tried to close file ".../sleepy.qmd" but the corresponding worker is busy. +Summary [21.4s] 1 test run: 0 passed, 1 failed +``` + +### GREEN — flipped assertion vs the FIXED bundle (`82bff64…`) + +``` +PASS [83.253s] (1/1) quarto-core::integration julia_engine_e2e::pc4a_abandoned_worker_close_busy +Summary [83.254s] 1 test run: 1 passed (1 slow), 393 skipped +``` + +(83s because the recovered #2 actually runs the cell, which sleeps 60s.) +Run under `QUARTO_PC4A_LIVE=1` + isolation env (real julia 1.11.7, +`JULIA_DEPOT_PATH`, `QUARTO_JULIA_PROJECT`). User server pid 9828 verified +ALIVE before and after both runs. + +--- + +## 6. Docs (work item 8) + +- **Compat log §15** (new): supersedes §4/§5's zero-change/byte-identity claim; + documents Bug A (decision gate + PC1/PC2 fix), Bug C (devnull redirect), the + full upstream diff summary, the **oneShot-reuse design question** for the + upstream PR, testing, and the harness-isolation concern. +- **Migration guide**: headline "zero source changes" corrected with an UPDATE + banner + a closing caveat — porting took zero changes, but q2's harder + `preview` exercise later exposed two latent *engine* bugs (present in Q1 too) + that required source changes. Framed as engine maintenance, not q2 adaptation. +- **Plan §P1** checkboxes reconciled (all three checked with outcome notes). + +--- + +## 7. Concern — PC4a live-harness isolation is imperfect (macOS) + +The harness isolates via a temp `HOME`, but the julia runtime/transport dir +resolves to `QUARTO_JULIA_PROJECT` (the shared real `~/Library/Caches/quarto/ +julia`), so the transport file is **not** actually isolated. The GREEN run +spawned QNR servers on the *shared* transport, and the temp-HOME-reading cleanup +guard missed them (6 leaked). **Cleaned up manually:** killed my 6 servers by +PID (process groups), removed the stale transport entry they left (it pointed at +a server I'd killed; the file read empty pre-test). **User server pid 9828 was +never touched** (verified alive throughout); other agents' pre-existing pool +(bd-l9jhy5u0) and a teammate's concurrent `quarto-core` run were left alone. + +This is a **harness** defect, not a product defect, and it does not undermine +the RED/GREEN evidence (the close/busy→forceclose recovery was exercised and +observed). But the harness needs a real runtime-dir override (or an explicit +skip) before it can run safely unattended. Recommend a follow-up (relates to +bd-l9jhy5u0). Documented in compat log §15. + +--- + +## 8. Verification counts (work item 9) + +| Suite | Result | +|-------|--------| +| Upstream deno (`tests/run-tests.sh`) | 9 passed, 0 failed (incl. 6 new PC1/PC2) | +| q2 `cargo nextest run -p quarto-core` | 2633 passed, 34 skipped, 0 failed (incl. live j1..j6 on rebundled fixture) | +| q2 `cargo nextest run -p quarto-preview` | 87 passed, 1 skipped, 0 failed | +| PC4a live (`QUARTO_PC4A_LIVE=1`) | RED (pre-fix) → GREEN (fixed) | + +Constraints honored: neither repo pushed; path-scoped commits in each; upstream +edits only on `q2-close-busy-fix`; frozen seams strengthened (never weakened); +`feature/ts-engine-extensions`, marimo, and unrelated julia processes untouched. + +### Report-scope correction (review Minor) + +For full honesty: commit **4efa5be84** (§4 above) also carried the controller's +P0/P2 checkbox-reconciliation edits to the plan file that were sitting +uncommitted in the worktree — not only my own §P1 edits. Likewise the fix-wave +commit below carries the controller's PC4 **countersign** note and the P3 +"PC6 + PC4a shared-transport isolation" item (added to the plan file by the +controller, uncommitted in the worktree). These are disclosed here rather than +silently bundled. + +--- + +# Fix wave (review response — task-p1-review.md, 2026-07-02) + +Addresses the one Important + one Minor. + +## Important — forceclose-itself-failing now bound + documented + +Added a **7th** PC2 unit test +(`tests/unit/julia-engine/worker-close.test.ts`): mocked writer with +`close → busy`, `forceclose → rejects`; asserts the forceclose error +**propagates with its real message** (`assertRejects(..., forcecloseError)`), +is not swallowed or retried, and the command sequence stops at +`["isopen","close","forceclose"]` (no run attempted). Added a one-line contract +comment at the forceclose call site in `worker-close.ts`: + +> Last line of defense. If the forced close ITSELF fails, that is a genuine +> environment failure (control server unreachable, etc.) — let it propagate; +> do not swallow or retry. + +No behavior change (binds existing behavior). **Fail-on-revert proven:** wrapping +the forceclose in a swallowing `try/catch` reddened ONLY this test +(`6 passed | 1 failed`); restored → `7 passed | 0 failed`. Full upstream suite: +**10 passed** (7 unit + 3 smoke steps), 0 failed. + +**Bundle unchanged (recorded honestly):** the comment + test did **not** change +`julia-engine.js` — `deno bundle` strips comments and the test is not bundled. +Both the upstream and q2-fixture bundles stay `82bff64cc5d060cb48983945060a6932` +(45323 B). No fixture bundle rebundle was needed; only the fixture's +`src/worker-close.ts` was synced for source parity. Compat log §15 updated. + +## Minor — see "Report-scope correction" above. + +## Fix-wave commits +- upstream `q2-close-busy-fix` @ **697a462** — "Bind + document the + forceclose-itself-fails contract (review follow-up)" +- q2 @ (see final message) — fixture `worker-close.ts` parity + docs/plan. diff --git a/.superpowers/sdd/task-p1c-report.md b/.superpowers/sdd/task-p1c-report.md new file mode 100644 index 000000000..9c00ace6f --- /dev/null +++ b/.superpowers/sdd/task-p1c-report.md @@ -0,0 +1,257 @@ +# Task P1c report — Bug C reader-side resilience (quarto-core ts_process, seam PC-C resilience leg) + +**Status: DONE.** The engine-host stdout reader (`reader_loop`, +`crates/quarto-core/src/engine/ts_process.rs`) now log-and-skips up to a +bounded number of consecutive non-JSON stray lines instead of escalating a +single stray line into a whole-subprocess kill that broadcasts an error to +every in-flight request. TDD RED→GREEN proven; the frozen P0 framing probes +stay untouched and GREEN; `quarto-core` + `quarto-preview` nextest suites are +fully green (2721 passed, 0 failed, 35 skipped). + +File touched: `crates/quarto-core/src/engine/ts_process.rs` (only file changed +— path-scoped, no other file in the tree modified). + +--- + +## 1. TDD RED transcript + +Two tests were added (replacing the stale `test_malformed_distinct_from_crash`, +whose name/assertions described exactly the escalate-on-first-line behavior +this task changes): + +- `test_stray_lines_below_bound_are_skipped_not_fatal` — the discriminating + test: two unrelated in-flight requests (A, B); two stray non-JSON lines + land on the shared channel between them; both A and B must still receive + their own real responses, and `shutting_down` must stay `false`. +- `test_malformed_beyond_bound_escalates_distinct_from_crash` — the + invariant-preservation test: `MAX_CONSECUTIVE_MALFORMED_LINES + 1` + consecutive stray lines (no valid frame in between) must still escalate to + the pre-existing kill-channel behavior (`Other`, not `ProcessCrashed`; + `shutting_down` set). + +Supporting test infra: `MockState.malformed` was widened from +`Option` to a `VecDeque`, and a new +`MockWriteHalf::signal_malformed_many(&[&str])` was added so a test can queue +N consecutive stray lines **atomically under one lock** — avoiding a race +where the reader thread could drain a `signal_malformed` call before a +second one is queued (the queue is what makes the below/above-bound split +testable at all). + +Ran against the **unmodified** `reader_loop` (constant +`MAX_CONSECUTIVE_MALFORMED_LINES` declared but not yet wired into the match +arm): + +``` +$ cargo nextest run -p quarto-core --lib -- \ + ts_process::tests::test_stray_lines_below_bound_are_skipped_not_fatal \ + ts_process::tests::test_malformed_beyond_bound_escalates_distinct_from_crash + + Summary [ 0.225s] 2 tests run: 1 passed, 1 failed, 2272 skipped + FAIL (2/2) quarto-core engine::ts_process::tests::test_stray_lines_below_bound_are_skipped_not_fatal +``` + +Failure (RED for the right reason — the discriminating test failed on the +exact old-code escalation path, not a typo/compile error): + +``` +thread '' panicked at crates/quarto-core/src/engine/ts_process.rs:1940:13: +A must still receive its own response after unrelated stray lines: Err(Other( + "engine-host protocol error: non-JSON line on stdout (likely a stray + console.log/console.info in the engine): \"[ Info: Log started at + 2026-07-02T13:11:20.379\"" +)) +``` + +(The `test_malformed_beyond_bound_escalates_distinct_from_crash` test passed +even pre-fix — expected, since old code already escalates on the *first* +malformed line, so N≥1 stray lines trivially also escalate. That test exists +to lock the invariant across the refactor, not to redden; the discriminating +RED evidence is the first test above.) + +## 2. Implementation (minimal) + +`reader_loop` now tracks a local `consecutive_malformed: u32` counter: + +- Reset to `0` on every well-formed frame (the `Ok(Response {..})` arm). +- On `RecvError::Malformed(line)`: increment the counter, log an `ERROR` with + a 200-char excerpt and the `n/BOUND consecutive` count. If + `consecutive_malformed <= MAX_CONSECUTIVE_MALFORMED_LINES`, `continue` the + loop — `shutting_down`/`pending`/`child` are left untouched. Otherwise, fall + through to the **unchanged** pre-existing escalation: set `shutting_down`, + drain and error every pending slot, kill the child, `break`. + +No other logic changed (routing, EOF/crash, I/O-error arms are untouched). + +## 3. TDD GREEN transcript + +``` +$ cargo nextest run -p quarto-core --lib -- \ + ts_process::tests::test_stray_lines_below_bound_are_skipped_not_fatal \ + ts_process::tests::test_malformed_beyond_bound_escalates_distinct_from_crash + + Starting 2 tests across 1 binary (2272 tests skipped) + PASS [ 0.086s] (1/2) quarto-core engine::ts_process::tests::test_malformed_beyond_bound_escalates_distinct_from_crash + PASS [ 0.171s] (2/2) quarto-core engine::ts_process::tests::test_stray_lines_below_bound_are_skipped_not_fatal + Summary [ 0.174s] 2 tests run: 2 passed, 2272 skipped +``` + +## 4. Chosen bound + rationale + +`const MAX_CONSECUTIVE_MALFORMED_LINES: u32 = 5;` + +- Small enough that a channel producing genuine, sustained garbage (wrong + binary spawned, protocol version mismatch, a child that never stops writing + to the shared fd) is still caught and terminated quickly — within 6 bad + lines, not an unbounded amount of silently-dropped protocol traffic. +- Large enough to absorb the two concrete P0 live symptoms with margin: (a) a + single leaked ANSI julia startup banner line, (b) one executeResult frame + corrupted mid-flight by an interleaved foreign write (which framing-splits + into at most a couple of bad lines, per the `pc_c_b_prime_interleaved_bytes_ + corrupt_frame` probe). Both are single-digit events, not sustained streams. +- The counter is **consecutive**, resetting on every well-formed frame — so a + channel that's mostly healthy but occasionally emits one stray line (e.g. a + future engine with a similar transient-leak bug) never accumulates toward + the bound across its lifetime; only a *burst* of bad lines with no good + frame in between trips escalation. + +This is a judgment call, not a value derived from a hard constraint; if a +future symptom needs a larger/smaller bound, this is a one-line change with a +name attached (`MAX_CONSECUTIVE_MALFORMED_LINES`), not a design change. + +## 5. No-hang investigation (brief item 2) + +**Question:** what happens to an in-flight request whose response frame was +itself corrupted/consumed as one of the skipped stray lines — does it hang +forever now that a single stray line no longer broadcasts an error to every +pending slot? + +**Finding: `TsEngineHost::request()` already has its own per-request timeout +mechanism, independent of the reader thread and unaffected by this change.** + +`request()` (ts_process.rs, unchanged by this task) takes a `window: +Option` and loops on `rx.recv_timeout(tick)` on the **caller's own +thread** — polling `cancellation.is_cancelled()` and `start.elapsed() >= w` +every tick (`CANCEL_TICK` = 250ms, or the window itself if shorter). When the +window elapses, it fires a cooperative `Cancel{target}` and returns +`Err(ExecutionError::Timeout{..})`. This loop does **not** depend on the +reader thread doing anything — even if the reader were fully blocked, the +caller's own timer still fires. + +Auditing every `request()` call site: + +| Call site | `window` | +|---|---| +| `ts_engine.rs:270` (dynamic `ClaimsLanguage` validation) | `Some(10s)` | +| `ts_engine.rs:317` (`ClaimsFile` validation) | `Some(10s)` | +| `ts_engine.rs:638`, `:704`, `:766` | `Some(10s)` | +| `ts_engine.rs:815` | `Some(30s)` | +| `load_engine` / `launch_engine` (internal) | `Some(DISCOVERY_WINDOW)` = `Some(10s)`, hardcoded | +| `ts_engine.rs:740` (**`Execute`** — the long-running user render/capture path, the one Bug C's symptom #2 corrupted frame hit) | `ctx.execute_timeout` | + +`ctx.execute_timeout` (`crates/quarto-core/src/engine/context.rs:110`, wired +from `resolve_execute_timeout` in `engine_execution.rs:597`) resolves from +document metadata `execute.timeout`: + +| metadata | window | +|---|---| +| absent / `true` (**default**) | `Some(DEFAULT_EXECUTE_TIMEOUT)` = `Some(300s)` | +| integer `N` | `Some(N seconds)` | +| `false` (**explicit opt-out**) | `None` | + +**Conclusion:** +- In the **default configuration** (no `execute: timeout: false` in the doc), + every request — including `Execute`, the path Bug C's corrupted-frame + symptom hit — is bounded by a caller-side timeout (300s default, or a + user-configured integer). A corrupted/dropped frame under the new + bounded-skip policy causes that one request to time out after its window + and return `Err(Timeout)`, exactly as any other slow/silent engine would. + This is a strict improvement over the pre-fix behavior: previously the + *entire host* died immediately (destroying every sibling in-flight + request too); now only the one request whose frame was actually lost is + affected, and only after its own timeout. +- **Concern (explicit, per brief item 2 — not fixed, not scope-expanded):** + if a user explicitly sets `execute: timeout: false`, `request()`'s `window` + is `None`. In that mode `request()` still polls `cancellation.is_cancelled()` + every 250ms but has **no time-based bound at all**. If that specific + request's response frame is the one that gets corrupted/dropped as a + skipped stray line, and no *additional* stray lines subsequently arrive to + trip `MAX_CONSECUTIVE_MALFORMED_LINES` (which would broadcast an error to + it via the escalation path), that request now hangs until explicit + cancellation (e.g. the user aborting the preview/render) — there is no + automatic recovery. Before this change, ANY single stray line anywhere on + the channel would have unblocked it (at the cost of also killing every + other in-flight request). This is a narrow, `execute: timeout: false`-gated + regression in "self-heals eventually" behavior, traded for the much more + common-case win of not destroying unrelated in-flight work over one leaked + banner line. I have **not** built new timeout machinery for this — per the + brief's explicit instruction not to invent one without reporting first. + If the controller wants this closed, the narrowest fix would be: give + `None`-window requests an internal maximum wait distinct from + "no timeout" (e.g. still respect `MAX_CONSECUTIVE_MALFORMED_LINES` but also + cap total wall-clock wait even absent stray lines) — but that is a genuine + design decision (what should "no timeout" mean when the channel is + degraded but not dead?) and is out of this task's scope. + +## 6. WASM-applicability note + +`ts_process.rs` line 23: `#![cfg(not(target_arch = "wasm32"))]` — the entire +module (including `reader_loop`, the new constant, and the mock test infra) +is **compiled out on `wasm32` targets**. This crate/module is native-only +subprocess/thread infrastructure; it has no `wasm32`-visible code path. Per +the WASM rules (`.claude/rules/wasm.md`), no `npm run build:wasm` or hub +WASM rebuild was required or performed for this change — confirmed by +inspection of the `#![cfg(...)]` gate, not just assumption. + +## 7. Verification counts + +``` +$ cargo nextest run -p quarto-core --lib + Summary [ 9.733s] 2251 tests run: 2251 passed, 23 skipped + +$ cargo nextest run -p quarto-core -p quarto-preview + Summary [ 50.505s] 2721 tests run: 2721 passed, 35 skipped +``` + +The three frozen P0 framing probes (`ts_process_framing_probe.rs`) ran as +part of the combined suite and are unmodified and GREEN: + +``` +PASS quarto-core::integration ts_process_framing_probe::pc_c_b_foreign_line_is_malformed +PASS quarto-core::integration ts_process_framing_probe::pc_c_a_large_single_line_frame_parses +PASS quarto-core::integration ts_process_framing_probe::pc_c_b_prime_interleaved_bytes_corrupt_frame +``` + +Also ran (not requested by the brief, but touching shared infra warranted the +extra check): `cargo clippy -p quarto-core --lib --tests -- -D warnings`. +One pre-existing failure in `crates/quarto-core/tests/integration/ +julia_engine_e2e.rs:1053` (`clippy::map_unwrap_or`) — **this file was not +touched by this task** (confirmed via `git status`/`git diff --stat`: the +only modified file in the tree is `ts_process.rs`); it predates this change +and is unrelated to it. No clippy warnings were reported for `ts_process.rs` +itself. `cargo fmt -p quarto-core -- --check` is clean (the repo's post-edit +hook runs `cargo fmt` automatically). + +## 8. Contract comment update (brief item 3) + +The `:930-935`-era comment ("Set shutting_down FIRST so the kill below +doesn't re-enter the crash path (finding #7 — one terminal error per exit)") +was rewritten in place. The new comment (at the top of the `Malformed` arm) +explains: what the old policy was and why it was too aggressive (Bug C — +engine-side leaks, e.g. a detached child inheriting the host's stdout fd, can +inject stray lines; killing all pending work over one banner is worse than +skipping it), what the new bounded policy is, and that the original +kill-everything behavior is deliberately preserved unchanged beyond the +bound (still needed to catch a genuinely broken wire). The original +"finding #7" rationale for the shutting_down-before-kill ordering is kept, +relocated to sit directly above the `shutting_down.store(true, ...)` line it +actually applies to now (only reached in the escalation branch). + +## Not done / out of scope (confirmed correctly excluded) + +- The engine-side root fix (detached QNR launcher stdio redirection) — P1's + job, already landed upstream per the plan; not touched here. +- Un-freezing/modifying the P0 framing probes — explicitly frozen; untouched + and still green. +- Any new timeout machinery for the `execute: timeout: false` + corrupted- + frame edge case (§5 concern) — reported, not implemented, per brief + instruction. diff --git a/.superpowers/sdd/task-p2-report.md b/.superpowers/sdd/task-p2-report.md new file mode 100644 index 000000000..a0aee5818 --- /dev/null +++ b/.superpowers/sdd/task-p2-report.md @@ -0,0 +1,182 @@ +# Task P2 report — Bug B (bd-h4rhohhy): "Bug B" refuted; echo fixture fixed; PC5/PC-B/PC7 green, PC6 deferred + +**Status: DONE_WITH_CONCERNS.** The evidenced defect was fixed (echo fixture emits realistic +`::: {.cell}` wrappers), all in-scope tiers are green with fail-on-revert proofs, and "Bug B" as a +distinct browser/splice defect is REFUTED. Concerns: PC6 (julia browser leg) is deferred opt-in +because a temp HOME does not isolate the julia transport; and I leaked a few orphaned julia servers +I could not reap (safety classifier blocked the kill). Details below. No push. + +## Diagnosis (evidence before fix) + +The q2-preview splice path is `apply_capture_splice` / `derive_cell_outputs` / `is_cell_wrapper` +(`crates/quarto-core/src/engine/capture_splice.rs`) — NOT `ReplayEngine`, and there is NO staleness +check. The splice maps each engine cell to the next `::: {.cell}` wrapper in the executed markdown. +**The brief's PRIMARY candidate (canonical `input_qmd` staleness rejection) is RULED OUT.** + +Root cause of the echo PC5 failure (native, deterministic): the echo fixture emitted a bare +`**ECHO_EXECUTED**` paragraph — no `.cell` wrapper — so `derive_cell_outputs` built an empty map and +the cell survived as raw source. Julia (decisive native leg, below) wraps its output in +`::: {#cell-1 .cell execution_count=1}` and splices cleanly. So the splice is CORRECT for real +engines; echo was an unrealistic fixture. **"Bug B" as a distinct browser/splice defect is +REFUTED** — the user's live julia symptom re-attributes to Bug A (close/busy discards the capture) +and/or Bug C (wire corruption + host-kill), both owned by P1/P1c. + +### Julia native leg (decisive) — recorded transcript + +Isolated fresh server (temp HOME + `IsolatedJuliaServerGuard`, real depot/project/bindir), doc +`engine: julia / execute: {daemon: false}` with `1 + 1`. `record_capture` → + +``` +result.markdown: +::: {#cell-1 .cell execution_count=1} +``` {.julia .cell-code} +1 + 1 +``` +::: {.cell-output .cell-output-display execution_count=1} +``` +2 +``` +::: +::: + +apply_capture_splice(A2=parse(input_qmd), A1, B1, "julia"): cell_survived=false ← splice fired +``` + +## Fix (ratified: fix the FIXTURE, not the splice — splice generalization REJECTED) + +`crates/quarto-core/tests/fixtures/extensions/echo-engine/src/echo-engine.ts` — `execute()` now wraps +the executed output in `::: {.cell}` / `.cell-output` (the shape real engines emit via the +engine-host's `mdFromCodeCell`). Rebundled the committed `dist/echo-engine.js` via +`cargo run --bin q2 -- build-ts-extension …`. + +**Blast-radius survey (every echo-fixture consumer):** all assertions are substring checks +(`ECHO_EXECUTED`, `not run by echo`, `{python}`) that survive wrapping. **Zero assertion edits +required.** `echo_engine_e2e.rs` 9/9 pass; full `cargo nextest run -p quarto-core -p quarto-preview` += 2720 passed / 35 skipped / 0 failed. + +## PC-B native seam (registered, GREEN) — `capture_splice_seam.rs` + +| ID | Tier | Real unit | Seam → assertion | Mock boundary | Revert hunk → RED | +|----|------|-----------|------------------|---------------|-------------------| +| PC-B | int-rs (+ deno leg) | `apply_capture_splice` / `is_cell_wrapper` | (1) `.cell`-wrapped capture → source cell REPLACED + output present; (2) bare-paragraph capture → documented NO-OP (cell survives); (3, deno) REAL echo capture → cell replaced + `ECHO_EXECUTED` present | none (real splice; real `record_capture` for leg 3) | (a) `is_cell_wrapper` stops recognizing `.cell` → (1)+(3) RED; (b) revert the echo FIXTURE wrapper → empty map → cell survives → (3) RED (this is the controller-rebound hunk) | + +TDD RED→GREEN (leg 3): pre-fix RED — `result.markdown` = bare `**ECHO_EXECUTED**`, `cell_survived=true`, +panic "the real echo capture must splice … result.markdown:\n**ECHO_EXECUTED**". Post-fix GREEN — 3/3 pass. + +## PC5 e2e (chromium) — amended assertion, GREEN, fail-on-revert proven + +Controller-amended (option 1): dropped the inert-source-first sub-assertion (unsatisfiable — the +eager capture is recorded at server startup before the browser connects, so the first render already +splices; renderTicks=1, no inert frame). Binding assertions now: (a) `ECHO_EXECUTED` appears in the +pane without reload; (b) the raw source token `PC5_ECHO_SOURCE_TOKEN` is ABSENT (splice REPLACED the +cell). The spec header documents why inert-first is unsatisfiable and why (a) is non-vacuous. Only +`test.fail()` and the inert-first block were changed. + +**Fail-on-revert proof (mandatory, per decision #2), revert target = `capture_driver.rs` `set_capture`:** +``` +GREEN (baseline): PC5 passes (2.7s) +RED (set_capture neutralized): PC5 fails — 30.1s timeout waiting for ECHO_EXECUTED +GREEN (restored + rebuilt): PC5 passes (2.7s) +``` +`cargo build --bin q2` between each (native-only; no WASM chain — server-side change). `capture_driver.rs` +restored clean (empty `git diff`). + +## PC7 (jsdom, `PreviewApp.integration.test.tsx`) — GREEN, fail-on-revert proven + +New test: after the initial capture-less render, firing `onCapturesChange` with a `CaptureRef` must +re-fire the render effect (a SECOND `renderPageForPreview` call) and forward the binary-doc bytes. + +**Fail-on-revert (two reverts — a finding):** +``` +Revert A — remove the contentTick bump (PreviewApp.tsx ~:737): GREEN (still passes) +Revert B — remove the `captures` write (~:733): RED ("getBinaryDocById expected to be called with ['pc7-capture-doc']") +Restore: GREEN +``` +**Finding:** the `contentTick` bump inside `onCapturesChange` is REDUNDANT with the render effect's +`state.captures` dependency (`PreviewApp.tsx:1128`) — a new `captures` reference already re-fires the +effect. So the controller's intended PC7 revert target (the contentTick bump) does NOT bind; the +load-bearing hunk is the `captures` write. PC7 binds THAT (revert B → RED), and the test + its +comment document the redundancy. `PreviewApp.tsx` restored clean. + +## PC6 (julia browser leg) — PASSES, but DEFERRED opt-in (concern) + +New spec `engine-capture-splice-julia.spec.ts`. It PASSES — a green run is on record: the julia +`{1+1}` cell's `.cell`-wrapped `2` splices into the pane without reload, 6.5s. It is gated behind +`QUARTO_PC6_LIVE=1` (skips in the default suite) for one reason: + +**The julia transport file is NOT isolated by a temp HOME.** Empirically every julia server (mine +and the environment's) uses the transport under `QUARTO_JULIA_PROJECT` (the shared instantiated +project), not under `$HOME/Library/Caches`. So a temp HOME does not yield a fresh isolated server — +the render reuses the developer's shared julia server/transport. That both violates the isolation +rule and exposes the run to Bug A (stale busy worker) in CI. Until Bug A is fixed (P1) or the +transport is truly isolated (an isolated COPY of the instantiated project as `QUARTO_JULIA_PROJECT`), +PC6 stays opt-in. The unconditional julia proof is the native leg above. This matches the +controller's "defer if it flakes on A/C; note green run pending P1" latitude. + +Added an additive `extraEnv?` option to `e2e/helpers/previewServer.ts` (no existing caller affected) +so the spec can inject the julia env. + +**Server-leak concern:** because the isolation was ineffective, my julia runs (native leg + the +nextest julia_engine_e2e tests that ran during verification + PC6) left ~4 orphaned QNR servers +(their temp project dirs are deleted; they serve nothing). I attempted to reap only the ones I +started (identified by orphaned temp-project path + my-session start times), but the safety +classifier blocked the `kill`. The user's real server (pid 9828, project `/Users/gordon/docs/julia`) +was correctly identified and never targeted. **These orphans should be reaped** — e.g. +`pgrep -f quartonotebookrunner` then kill the ones whose `.jl` path under `/T/.tmp…` no longer +exists (NOT pid 9828). I did not force this past the safety guard. + +## Rebuilds performed before each e2e evidence run + +fixture `.ts` → `dist/echo-engine.js` (`build-ts-extension`) → `cargo build --bin q2`. No WASM/SPA +rebuild: no Rust/WASM product code changed (splice unchanged); the fixture loads server-side at +runtime (copied fresh into the temp project by each spec); the embedded SPA/WASM (P0's build) is +current. The set_capture revert used `cargo build --bin q2` only (native server-side change). + +## Verification counts (each run once, to a log) + +- `cargo nextest run -p quarto-core -p quarto-preview`: **2720 passed, 35 skipped, 0 failed.** +- PC-B `capture_splice_seam`: 3 passed. `echo_engine_e2e`: 9 passed. +- q2-preview-spa vitest: unit **25 passed**; integration **76 passed** (incl PC7). +- q2-preview-spa `npm run test:e2e`: **37 passed, 1 skipped (PC6 opt-in), 1 failed** — the failure is + the PRE-EXISTING `firefox-ws-queue` under the **firefox** project (`browserType.launch: Executable + doesn't exist … firefox-1522/Nightly.app` — Firefox not installed); the same spec passes under + chromium. Orthogonal to this task. +- PC5 fail-on-revert: GREEN→RED(30.1s)→GREEN. PC7 fail-on-revert: A GREEN (redundant), B RED, restore GREEN. + +## Files changed / added (path-scoped) + +- `crates/quarto-core/tests/fixtures/extensions/echo-engine/src/echo-engine.ts` (fixture emits `.cell`) +- `crates/quarto-core/tests/fixtures/extensions/echo-engine/dist/echo-engine.js` (rebundled) +- `crates/quarto-core/tests/integration/capture_splice_seam.rs` (new PC-B seam) + `main.rs` (register) +- `q2-preview-spa/e2e/engine-capture-splice.spec.ts` (PC5 amended) +- `q2-preview-spa/e2e/engine-capture-splice-julia.spec.ts` (new PC6, opt-in) +- `q2-preview-spa/e2e/helpers/previewServer.ts` (additive `extraEnv`) +- `q2-preview-spa/src/PreviewApp.integration.test.tsx` (PC7) +- `capture_driver.rs` and `PreviewApp.tsx` touched only for fail-on-revert proofs; restored clean. +- throwaway `pcb_diag.rs` deleted. + +## Review response (2026-07-02) — commit 2 + +Two review items addressed (comment-only, no behavior change): + +- **Important #1 (stale PC5 header):** rewrote `engine-capture-splice.spec.ts:1-29`. It no longer + describes a "WASM render_page_for_preview ReplayEngine splice" (refuted); the chain now ends at + the q2-preview pipeline's CaptureSplice stage (`capture_splice.rs`), explicitly notes there is no + ReplayEngine and no staleness check, describes the `.cell`-wrapper the echo fixture now emits, and + replaces the P0-era "written pre-fix / ratifies before un-skipped" status with the P2 reality + (fixture fixed, amended assertion ratified, set_capture fail-on-revert proven). + +- **Minor #2 (PC-B comment):** chose the honest option — actually ran the `is_cell_wrapper` revert + and recorded it, so the comment's claim is now transcript-validated (no softening needed): + ``` + is_cell_wrapper neutralized (return false): + bare_paragraph_capture_is_a_documented_noop PASS (cell survives, expected) + cell_wrapped_capture_splices FAIL (leg 1 — cell survives, no .cell matched) + real_echo_capture_splices FAIL (leg 3) + restored: 3 passed + ``` + Both PC-B revert legs are now proven: (a) `is_cell_wrapper` matching → legs (1)+(3) RED; + (b) the echo fixture wrapper emission → leg (3) RED (recorded at fix time). `capture_splice.rs` + restored clean. + +PC5 re-run after the header edit: GREEN (1.2s). diff --git a/.superpowers/sdd/task-p3-report.md b/.superpowers/sdd/task-p3-report.md new file mode 100644 index 000000000..d02abf23f --- /dev/null +++ b/.superpowers/sdd/task-p3-report.md @@ -0,0 +1,258 @@ +# Task P3 report — error-path coverage (PC3, PC8), julia-harness isolation, full verification + +**Status:** DONE. This session completed verification only — the substantive TDD and +isolation work below was done by a previous session (killed mid-verification by a +session pause) and is reported here from `git show`, not re-derived. Where the report +says "predecessor's evidence," that RED/GREEN transcript was not independently +reproduced by this session. + +## 1. What the predecessor's commits contain + +### c16200f92 — `test(preview-capture): PC3/PC8 error-path coverage (bd-h4rhohhy)` + +``` +crates/quarto-preview/src/capture_driver.rs | 127 ++++++++++++++++++++++++++++ +crates/quarto-preview/src/re_execute.rs | 121 ++++++++++++++++++++++++++ +2 files changed, 248 insertions(+) +``` + +**PC3** (`capture_driver.rs`, `pc3_failing_engine_does_not_block_next_doc_capture`): +a `FailingTestEngine` (declares `test-failing`, `execute()` always returns +`ExecutionError::Other(...)`) registered alongside the existing +`PassthroughTestEngine`. Two docs, `a-failing.qmd` (engine `test-failing`) and +`b-echo.qmd` (engine `test-passthrough`) — the `a-`/`b-` prefix matters because +`qmd_files` is sorted (`discovery.rs:154`), guaranteeing the failing doc runs first, +so a loop-return-on-Err regression can't hide behind ordering. Assertions: doc A's +failure emits `Q-PREVIEW-CAP-1` to the test diagnostic sink, AND doc B's capture is +still recorded (`record_eager_captures`' continue-on-error contract, +`capture_driver.rs:116-140`). No product-code change — both behaviors already +existed; this closes an untested gap. + +Fail-on-revert proven both ways (per commit message, not re-run by this session): +(a) commenting out the `sink.emit(...)` call reddens the diagnostic-count assertion; +(b) replacing the loop's `Err(e) => { .. }` arm with an early `return Err(..)` +reddens doc B's capture assertion (loop stops at the first failure). + +**PC8** (`re_execute.rs`, `pc8_re_execute_failure_sets_error_state_and_emits_diagnostic`): +a `FailingReExecuteEngine` sharing the doc's declared engine name +(`test-passthrough`) but always failing, installed via a **wholesale registry +override** applied only at re-execute time (the seed run used the real +`PassthroughTestEngine` and is unaffected). Uses a **separate cache dir** from the +seed run — required because `record_capture_cached` keys purely on +`sha256(input_qmd)` (`cache.rs:150-163`), and content is unchanged between seed and +re-execute, so reusing the seed's cache dir would replay the cached success and +never invoke the failing engine. Assertions: `perform_re_execute`'s failure branch +(`re_execute.rs:253-279`) writes sidecar `CaptureState::Error` + `last_error`, and +emits `Q-PREVIEW-RE-1`. No product-code change. + +Fail-on-revert (per commit message): commenting out the +`ctx_for_task.index().set_capture(&rel_path_for_task, &errored)` write leaves the +sidecar stuck at whatever state `claim_and_spawn` set pre-run, reddening the +`CaptureState::Error`/`last_error` assertions. + +### 4785c9d2c — `test(preview-capture): isolate QUARTO_JULIA_PROJECT in live-julia harnesses (bd-h4rhohhy P3)` + +``` +.../plans/2026-07-02-preview-capture-delivery.md | 54 ++++++++- +.../tests/integration/julia_engine_e2e.rs | 129 ++++++++++++++++++++- +.../e2e/engine-capture-splice-julia.spec.ts | 77 +++++++++--- +3 files changed, 236 insertions(+), 24 deletions(-) +``` + +PC4a and PC6 already isolated the julia transport/server via a temp `HOME`. This +commit adds a **second isolation layer**: `isolate_julia_project()` (Rust, +`julia_engine_e2e.rs:124-138`) / `isolateJuliaProject()` (TS, +`engine-capture-splice-julia.spec.ts`) copy the ambient `QUARTO_JULIA_PROJECT`'s +`Project.toml` + `Manifest.toml` into a per-test temp dir and re-point +`QUARTO_JULIA_PROJECT` at the copy, so the detached server's `--project=` flag +never names the shared real directory (`JULIA_DEPOT_PATH` stays shared — no +package re-instantiation). Both harnesses gained a +`SharedTransportSentinel`/`captureSharedTransportMtime()` assertion that the +shared `~/Library/Caches/quarto/julia/julia_transport.txt` existence+mtime is +unchanged across the run. + +**Live-verified twice by the predecessor** (evidence from the commit message, +not re-run by this session): `pc4a_abandoned_worker_close_busy` (78.8s, +`QUARTO_PC4A_LIVE=1 ... --run-ignored all`) and PC6 +(`QUARTO_PC6_LIVE=1 npx playwright test engine-capture-splice-julia`, 8.6s) — +both PASSED; shared transport file mtime/existence unchanged both times; +`IsolatedJuliaServerGuard` reaped every process it spawned (no new leaked pids). +Found but explicitly did NOT touch ~28 pre-existing leaked julia processes on the +shared transport (pre-existing `bd-l9jhy5u0` leak, out of scope, "never touch +processes you didn't start" constraint honored). + +Decision recorded: `QUARTO_PC6_LIVE` stays **opt-in** — isolation is no longer the +reason (proven safe); the remaining reason is environmental/speed (real +julia+deno dependency, multi-second server boot), mirroring PC4a's `#[ignore]` +gate. Documented in the spec's file header and (now, after this session's fix +below) the plan's seam table. + +## 2. Dangling edit disposition (this session) + +One uncommitted edit was found on disk in +`crates/quarto-core/tests/integration/julia_engine_e2e.rs`: a clippy-shape +rewrite of PC4a's markdown extraction, `.map(|c| ...).unwrap_or_else(|| panic!(...))` +→ `.map_or_else(|| panic!(...), |c| ...)`. Verified semantically identical (same +panic-on-missing-capture, same markdown extraction on hit) and confirmed it +compiles clean (`cargo check -p quarto-core --tests`). Committed as +`f0728dd63` — `style(quarto-core): use map_or_else in PC4a markdown extraction +(bd-h4rhohhy)`. + +## 3. Verification ladder (this session) + +All logs in `/tmp/bd-h4rhohhy-p3-logs/` (not committed — scratch). + +### Leg A — `cargo nextest run -p quarto-preview -p quarto-core` + +**Result: 2723 tests run: 2721 passed, 2 failed, 35 skipped.** + +Log: `/tmp/bd-h4rhohhy-p3-logs/leg-a-nextest.log`. + +Two failures, both in `julia_engine_e2e.rs`, **neither part of this branch's +PC3/PC8/PC6 work** (pre-existing tests from the original julia-validation plan): + +- `julia_engine_e2e::j1_minimal_julia_render` +- `julia_engine_e2e::j2_document_level_echo_false_hides_source_keeps_output` + +Both failed with `Julia server returned error after receiving "isopen" command: +Incorrect HMAC digest` — a shared-julia-transport handshake failure. These +tests use `setup_julia_project()` (NOT the isolated +`IsolatedJuliaServerGuard`/`isolate_julia_project()` path — that's PC4a-only), +so they connect to the AMBIENT `~/Library/Caches/quarto/julia/julia_transport.txt`. +At the time of the run the machine also had **~49 leaked julia server +processes** from unrelated test activity (the pre-existing `bd-l9jhy5u0` +worker-leak bug), which was the initial suspect. + +**Confirmed transient, not a regression**: re-ran each failing test in isolation +(single test, no concurrent julia contention): + +``` +cargo nextest run -p quarto-core --test integration -- julia_engine_e2e::j1_minimal_julia_render + → PASS (6.1s) +cargo nextest run -p quarto-core --test integration -- julia_engine_e2e::j2_document_level_echo_false_hides_source_keeps_output + → PASS (5.9s) +``` + +Both pass cleanly in isolation. **The full diagnosis came during Leg C** (see +below): the orphan pool was only an amplifier — the real root cause is that the +`julia_engine_e2e` tests race EACH OTHER on the single ambient transport file. +Fixed by nextest serialization (commit `156b290ec`); see Leg C. + +### Leg B — `cargo build --bin q2` + q2-preview-spa `npm run test:e2e` + +`cargo build --bin q2`: succeeded (log: `leg-b-build.log`). + +`npm run test:e2e` (log: `leg-b-e2e.log`): **37 passed, 1 skipped, 1 failed** +(39 total across the `chromium` + `firefox-ws-queue` projects). + +- Skipped: `[chromium] engine-capture-splice-julia.spec.ts` PC6 — expected, + opt-in (`QUARTO_PC6_LIVE` unset). +- Passed: PC5 (`engine-capture-splice.spec.ts`) — `PC5: recorded echo capture + splices into the pane without reload`. +- Failed: `[firefox-ws-queue] firefox-ws-queue.spec.ts` — + `browserType.launch: Executable doesn't exist at + .../ms-playwright/firefox-1522/firefox/Nightly.app/...` — Firefox not + installed on this machine. This is the **known pre-existing failure** the + brief called out; it is the ONLY e2e failure, matching the expected shape. + +### Leg C — full `cargo xtask verify` (three runs; the whole story) + +**Run 1** (log: `leg-c-xtask-verify.log`, dirty machine — ~49 orphaned QNR +processes present): FAILED at the Rust-tests step. +`Summary [78.4s] 8624/10589 tests run: 8623 passed, 1 failed, 198 skipped` +(nextest fail-fast cancelled the rest). Failure: +`julia_engine_e2e::j1_minimal_julia_render`, `Incorrect HMAC digest` at +`isopen`. Initial hypothesis: contention from the orphaned QNR pool +(bd-l9jhy5u0). + +**Orphan-pool cleanup** (user-approved, done by the coordinator between runs): +~50 orphaned QuartoNotebookRunner processes were reaped via category-pattern +pkill; the user's own julia server (pid 9828) was preserved. Machine clean. + +**Run 2** (log: `leg-c-xtask-verify-2.log`, clean machine): FAILED AGAIN. +`Summary [74.6s] 8758/10589 tests run: 8756 passed, 2 failed, 198 skipped`. +Failures: `julia_engine_e2e::j3_exeflags_and_env_through_julia_block` and +`julia_engine_e2e::j4_error_handling_does_not_wedge_host` — same +`Incorrect HMAC digest` signature, but DIFFERENT victims than run 1. + +**Diagnosis — intra-suite race, structural, pre-existing.** Rotating victims +on a clean machine rule out the orphan pool as the root cause. The +`julia_engine_e2e` tests had NO nextest serialization; each uses +`setup_julia_project()`, which isolates the project dir but NOT `HOME`, so +every concurrently-running j-test's `daemon: false` render boots its own QNR +server against the SINGLE ambient +`~/Library/Caches/quarto/julia/julia_transport.txt` — concurrent startups +overwrite each other's transport entry (port/pid/HMAC key), and a client that +reads the wrong entry fails the socket handshake with `Incorrect HMAC digest`. +Which test loses the race depends on scheduling — hence rotating victims. +This is a **pre-existing test-infra property, not caused by this branch's +diff**: the j-tests, `setup_julia_project()`, and the shared-transport reuse +design all predate P3 (they belong to the julia-validation plan), and this +branch's P3 commits touch only new PC3/PC8 tests, the PC4a/PC6 isolation +helpers (which are NOT used by j1-j6), and a comment-shape edit. We did NOT +re-verify on the merge-base — a merge-base run would race the same way only +probabilistically (the failures are scheduling-dependent, so a green +merge-base run would prove nothing and a red one would only confirm what the +structural argument already establishes: none of the racing components +changed on this branch). + +**Fix** (commit `156b290ec`, config-only, precedent bd-u3ze): added a +`julia-shared-transport = { max-threads = 1 }` test-group in +`.config/nextest.toml` with an override filtering +`package(quarto-core) & binary(integration) & test(/^julia_engine_e2e::/)`, +so at most one j-test is in flight at a time. The j-tests themselves are +untouched (out of this task's scope; full hermetic isolation for them is +being filed as a separate strand by the controller). + +**Run 3** (log: `leg-c-xtask-verify-3.log`, clean machine + serialization): + +**Run 3 (post-serialization, log: `leg-c-xtask-verify-3.log`): ALL GREEN.** +Tail: `✓ All verification steps passed!`; nextest summary: +`10589 tests run: 10589 passed, 198 skipped` (0 failed — the julia_engine_e2e +tests now run serialized in the `julia-shared-transport` group). All downstream +legs (ts-packages builds + mcp smoke, hub-client `build:all` incl. WASM, +hub-client `test:ci`) completed successfully. +*(Filled in by the controller from the run-3 log after the implementer session +ended; independently re-verified by the P3 reviewer.)* + +## 4. PC6 gate decision + +**Decision: PC6 stays opt-in (`QUARTO_PC6_LIVE=1`), per the predecessor's +already-recorded rationale — confirmed correct by this session, plan updated to +match.** + +Read `isolate_julia_project()` (`julia_engine_e2e.rs:124-138`) and +`isolateJuliaProject()` (`engine-capture-splice-julia.spec.ts`), and the spec's +file-header comment. The isolation code is real (copies `Project.toml`/ +`Manifest.toml` into a per-test temp dir, re-points `QUARTO_JULIA_PROJECT`, +layered on top of the pre-existing temp-`HOME` transport override) and its +safety claim is backed by the predecessor's two live runs (evidence above) — not +independently re-run by this session (re-running would spawn another real julia +server on an already-contaminated machine — 49 leaked processes present — for no +new information; the predecessor's evidence already satisfies "prove it"). + +Per the brief: "drop the `QUARTO_PC6_LIVE` opt-in gate IF the isolation makes it +safe for the julia-gated tier, else record explicitly why it stays opt-in." The +isolation DOES make PC6 safe from a shared-state-corruption standpoint — but +"safe" and "fast/deterministic enough for the default CI suite" are different +questions. PC6 spawns a real julia server (network-installed julia binary, +multi-second boot, ~6.5-8.6s per the recorded runs) — the same class of cost +that keeps PC4a behind `#[ignore]` on the Rust side. The gate stays for that +reason, not for isolation safety. + +**Plan/code disagreement found and fixed**: the plan's seam table PC6 row was +STALE — it still read "Opt-in because a temp HOME does NOT isolate the shared +julia transport... un-deferral tracked in P3" (the PRE-P3 rationale), even +though the P3 checklist item immediately above it (already correct, predecessor's +text) already recorded the isolation-closed / speed-is-the-reason-now decision. +Fixed in this session: the PC6 seam-table row now states the post-P3 rationale +explicitly and points to this report. (Commit: bundled with the plan-checklist +reconciliation below.) + +## 5. TDD evidence provenance + +**RED/GREEN evidence in §1 above (PC3, PC8, PC4a/PC6 live runs) is the +predecessor's** — read from commit messages and code comments, not independently +re-derived or re-run by this session. This session's own verification work is: +the dangling-edit compile check (§2), the full ladder (§3), and the isolated +re-runs of `j1`/`j2` that confirmed those two failures are transient (§3, Leg A). diff --git a/Cargo.lock b/Cargo.lock index 32c10524e..cd5682f1f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3664,12 +3664,14 @@ dependencies = [ "serde_yaml", "sha2 0.11.0", "smallvec", + "socket2", "tempfile", "thiserror 2.0.18", "time", "tokio", "tokio-util", "tracing", + "tracing-subscriber", "uuid", "walkdir", "which", @@ -4115,6 +4117,7 @@ dependencies = [ name = "quarto-util" version = "0.10.0" dependencies = [ + "dirs", "serde", "thiserror 2.0.18", ] diff --git a/claude-notes/designs/2026-07-24-preview-capture-splice-three-way-merge.md b/claude-notes/designs/2026-07-24-preview-capture-splice-three-way-merge.md new file mode 100644 index 000000000..bfba5dbae --- /dev/null +++ b/claude-notes/designs/2026-07-24-preview-capture-splice-three-way-merge.md @@ -0,0 +1,254 @@ +# Preview capture-splice as a three-way merge + +**Status:** Proposed — **deferred**. Do not implement from this document yet; it +was judged too risky to undertake at the time of writing. The purpose here is to +preserve the design and its reasoning so that a future implementer begins from +the analysis rather than rediscovering it. + +**Date:** 2026-07-24 + +**Related code** +- `crates/quarto-core/src/engine/capture_splice.rs` — the current lock-step walk this design would replace. +- `crates/quarto-ast-reconcile/` — the two-way AST reconciliation this design builds on. +- `crates/quarto-core/src/stage/stages/engine_execution.rs` — the render path, which already uses two-way reconciliation (`reconcile(ast, executed_ast)`). + +**Related strands:** bd-lucp (original splice), bd-7hqea3qi (Div recursion), +bd-5jxcio5d (marimo RawBlock), bd-5oyk1xce (Bug B, multi-engine fold), +bd-5m1ni9if (open splice edge). + +--- + +## 1. Why preview needs a splice at all + +`q2 preview` shows the reader executed engine output — plots, computed tables, +marimo widgets — while they edit the source. Re-running the engine on every +keystroke is not viable, so the preview runs the engine **once**, server-side, +and records the result as an `EngineCapture`: the triple +`(engine_name, input_qmd, result_markdown)`. Every later edit reuses that one +capture. + +Reuse is the whole problem. The moment the reader types a character, the live +source no longer matches the source that was captured, so the recorded output +cannot simply be replayed verbatim. The splice exists to bridge that gap: it +takes the recorded transformation and re-applies it to the edited document. + +Three ASTs frame the task. Naming them once here fixes the vocabulary for the +rest of the document: + +- **A1** — the captured *pre-engine* AST (`parse(capture.input_qmd)`). +- **B1** — the captured *post-engine* AST (`parse(capture.result_markdown)`). +- **A2** — the *live, edited* pre-engine AST (what the current source produces + before the engine would run). + +The engine turned A1 into B1. The splice must produce **B2** — what the engine +*would* produce for A2 — without running the engine again. + +## 2. The current algorithm and its recurring failure + +The current splice (`derive_cell_outputs_walk` in `capture_splice.rs`) treats the +capture as a recipe in two steps. First it diffs A1 against B1 to learn "which +output block did the engine emit for each source cell," recording a map keyed by +`(structural_hash(cell), occurrence_index)`. Then it walks A2 and, for each +source cell, swaps in that cell's recorded output. + +The diff in the first step is a hand-rolled **lock-step walk**. It advances two +pointers through A1 and B1 in parallel under three assumptions: prose blocks +appear identically in both and advance both pointers; each engine cell in A1 maps +to exactly **one** "engine-output block" in B1 (a `.cell` wrapper Div, or a +marimo island `RawBlock`); and any divergence from this pattern stops the walk +(the *fail-soft* rule — whatever was collected before the divergence stays valid, +and everything after falls through to raw source). + +That "exactly one output block per cell" assumption is the recurring fault line. +Real engines violate it in a new way every few months, and each violation has +arrived as a silent, output-dropping bug with no error anywhere: + +- **bd-7hqea3qi** — a figure-labelled cell is *nested* in a float Div, so the + walk hit two unequal Divs and stopped. Fix: recurse into Div content. +- **bd-5jxcio5d** — marimo emits islands as bare `RawBlock`s, not `.cell` Divs, + so nothing matched. Fix: widen the output-block predicate to accept `RawBlock`. +- **bd-5oyk1xce (Bug B)** — a foreign engine's un-executed cell stalled the B1 + pointer and derailed the walk. Fix: advance past a structurally-equal + passthrough block. +- **The bug that prompted this document** — an `echo: true` marimo cell emits + **two** sibling blocks (an echoed-source `CodeBlock`, then the island). The + echoed `CodeBlock` is not an output block, so the walk breaks at the first such + cell and drops every cell after it. + +Each fix is correct and well-tested. Together they are a symptom: the model +underlying the walk is weaker than the output real engines produce, so the model +accretes special cases instead of generalizing. + +## 3. The insight: render already reconciles; preview is the three-way case + +The render pipeline does not use this walk. After the engine runs, render calls +`quarto_ast_reconcile::reconcile(ast, executed_ast)` — a general, content-hash +two-way reconciliation — to merge the pre-engine and post-engine ASTs while +preserving source locations. That path renders `index.qmd`'s six marimo islands +correctly. The preview path, using the bespoke walk on the *same* capture, drops +five of them. The reconciliation machinery is the part that works; the bespoke +walk is the anomaly. + +The reason preview forked away from reconciliation is real, not accidental. +`reconcile` is **two-way**: it merges one before-AST with one after-AST for the +*same* document version. Preview is **three-way**: it must combine the recorded +transformation (A1→B1) with a *different*, edited document (A2). A two-way merge +does not directly express that. + +The design in this document closes the gap by building the three-way merge **on +top of** the two-way primitive, rather than hand-rolling a diff beneath it. + +## 4. The design: a three-way merge over blocks + +A three-way merge (the classic `diff3`) needs two diffs against a shared base. +Here the base is A1, and the two-way reconciliation supplies both diffs: + +- the **engine diff**, `reconcile(A1, B1)` — which B1 blocks are unchanged prose + and which are new engine output; +- the **user diff**, `reconcile(A1, A2)` — which A2 blocks the reader left + untouched and which they edited. + +The two-way primitive reports each alignment as `KeepBefore` (content-identical +to a base block — a hash match), `UseAfter` (new or changed content), or +`RecurseIntoContainer` (same container, descend). From those two alignments the +merge finds the base blocks matched in *both* diffs — the stable anchors — and +classifies each chunk between consecutive anchors by how its A2 (the reader's) +and B1 (the engine's) ranges relate to the base: + +| A2 vs. base | B1 vs. base | result | +|---|---|---| +| unchanged | changed | take **B1** (splice the engine output) | +| changed | unchanged | take **A2** (edited cell falls through to raw source) | +| unchanged | unchanged | keep the base | +| changed | changed | conflict → take **A2** (raw source) | + +This table earns its keep by *deriving* today's behavior instead of hand-coding +it. An unedited cell shows engine output; a cell the reader is actively editing +shows raw source until the next capture — exactly the current contract, now a +consequence of the merge rather than a special case inside a walk. + +## 5. Cell-order attribution: the one piece the merge does not give for free + +Block-level `diff3` handles a single expanded cell cleanly. For an `echo: true` +cell the base chunk is `[cell]`, the reader's chunk is the unchanged `[cell]`, +and the engine's chunk is `[echoCode, island]`; the table says "take the +engine's chunk," and both blocks splice with no per-shape knowledge required. The +entire family of output-block predicates disappears. + +Adjacent cells with a partial edit are the case the merge cannot resolve on its +own, and the reason is fundamental: **the engine erases cell identity.** A cell +and the island it becomes share no content, so a content diff finds *no anchor* +inside a run of adjacent cells. Consider `index.qmd`'s first two cells, which sit +together with no prose between them, when the reader edits only the second: + +``` +base A1 = [ cell1, cell2 ] +live A2 = [ cell1, cell2' ] (only cell2 edited) +engine B1 = [ island1, echoCode2, island2 ] +``` + +The base↔engine diff anchors nothing here — `cell1` and `cell2` both vanished +into unrelated output — so `diff3` sees one chunk changed on both sides, +declares a conflict, and takes the reader's side: `[cell1, cell2']`. That drops +`cell1`'s island even though the reader never touched `cell1`. The result is +worse than the bug being fixed. + +The resolution keeps the merge but adds one assumption that holds for every +execution engine: **the engine emits output in cell order and touches only +cells.** Within an engine-changed region, attribute the output run to source +cells by order — cell *k* owns the run up to where cell *k+1*'s output begins — +producing a per-cell output run. The "did the reader edit this cell" test then +runs per cell, on the hash of the cell's source, which is precisely today's +`(hash, occurrence)` key. In the example, `cell1` (unedited) takes `island1`, +`cell2` (edited, hash miss) falls through to `cell2'`, and `echoCode2`/`island2` +are attributed to `cell2` and dropped as stale — yielding `[island1, cell2']`, +which is correct. + +The attribution is the one place engine-specific reasoning can re-enter, because +splitting a multi-block run across adjacent cells needs a *cell-boundary* signal +(see the open questions). The design's aim is to confine that reasoning to a +single, explicit place instead of spreading it across a growing predicate. + +## 6. Why this is more general and less fiddly + +The current walk carries a table of shapes it must recognize — `is_cell_wrapper`, +`is_engine_output_block`, the `RawBlock` special case, the passthrough rule — and +every new engine output shape adds a row. The three-way merge removes that table. +It rests instead on one assumption that is true of every engine we support: prose +passes through untouched, and cell outputs are emitted in cell order. The remaining +hard case — adjacent cells under a partial edit — is handled once, by order +attribution, rather than re-litigated per engine. + +## 7. Open design questions + +The merge trades a growing list of per-shape patches for a small set of sharper +questions. These are unresolved and would need answers before implementation: + +1. **Cell-boundary delimitation.** Order tells us cell *k* precedes cell *k+1*, + but not where a multi-block run splits between them. The available signals + (`.cell` Div per cell, one island per cell) are engine-specific — the very + knowledge the merge set out to remove. The honest question is whether we can + avoid a per-engine boundary abstraction, or should instead shrink the + engine-specific surface to one declared contract ("emit one recognizable + boundary per cell"). + +2. **Fall-through granularity: per cell or per region.** If we decline to solve + (1), the simple alternative reverts an entire adjacent-cell region to raw + source when any cell in it is edited. Coarser, fully engine-agnostic, and + possibly fine for a preview that re-captures on save. This is a product + decision, not a technical one. + +3. **Order attribution vs. content keying under reordering.** Cells match by + position-independent `(hash, occurrence)`, but output attribution is + order-based. Reordering two unedited cells can make the two disagree. The + design needs one coherent rule. + +4. **Is prose actually invariant?** `results='asis'`, inline execution, and + markdown-emitting cells (`mo.md()`) rewrite prose, shrinking the anchor set. + The design needs an explicit definition of "anchor" (likely: only + hash-identical-across-A1↔B1 blocks) and a stated behavior for these engines. + +5. **Three-way recursion into containers.** A `::: {#fig-…}` float wraps the raw + cell in A1 and the executed output in B1 — same container, changed content. + The merge needs a rule for when to descend and merge children versus treat the + whole container as one changed chunk, keeping the occurrence counter in + document order across nesting levels. + +6. **Multi-engine fold.** Captures fold in sequence; engine 2's base is engine + 1's spliced output. Whether sequential three-way merges compose correctly, or + interleaved output needs a joint attribution, must be first-class — Bug B lived + in exactly this seam. + +7. **Fail-soft floor.** Today's walk cannot emit wrong output; its worst case is + raw source. A merge can be confidently wrong — stale output spliced onto a + reverted cell, or mis-attributed across adjacent cells. Stale-but-plausible + output is arguably worse than visibly-raw source, so the design should keep a + guard: splice only when attribution is unambiguous, else fall through. + +8. **Cost and caching.** The merge runs in WASM on roughly every edit and costs + two reconciliations plus classification, against today's single O(n) walk. The + engine diff (A1→B1) is fixed per capture; only the user diff (A1→A2) changes + per keystroke, so the engine side should be cached per capture from the start. + +9. **The reframing question — attribute at capture time, not splice time.** Every + question above is a consequence of one earlier decision: the capture stores a + flat markdown blob, discarding the cell→output correspondence the engine knew + exactly at execution time. If the capture instead recorded structured per-cell + output (`Vec<(cell_key, output_blocks)>`), the browser splice would collapse to + a keyed lookup per A2 cell, and questions 1–7 would not arise. The cost moves + into the engine-host capture contract. This is the sharpest fork: a principled + inference engine in the browser, versus recording what the engine already knew. + It should be settled before the merge is built, because it may make the merge + unnecessary. + +## 8. Risk and why this is deferred + +The current walk, for all its accreted patches, has a strong safety property: it +never emits wrong output. Replacing it with a merge introduces the possibility of +confident mis-attribution (question 7), touches a component that runs on every +keystroke in WASM (question 8), and interacts with the multi-engine fold that has +already produced one subtle bug (question 6). The reframing question (question 9) +may also redirect the whole effort toward the capture contract instead. Given +that the immediate bug has a small, contained fix that stays inside the current +model (see the companion discussion), the merge is recorded here and deferred +rather than started now. diff --git a/claude-notes/designs/document-profile-contract.md b/claude-notes/designs/document-profile-contract.md index 2447b68f9..034e4c870 100644 --- a/claude-notes/designs/document-profile-contract.md +++ b/claude-notes/designs/document-profile-contract.md @@ -2,7 +2,7 @@ **Status:** Active (Phase 0 of the website epic, `bd-0tr6` / `bd-f3jc`; extended in Phase 8 sub-phase 8.0, `bd-fegm` + `bd-r82e`). -**Version tag:** `DOCUMENT_PROFILE_VERSION = 2` +**Version tag:** `DOCUMENT_PROFILE_VERSION = 7` **Type:** `quarto_core::document_profile::DocumentProfile` **Stage:** `quarto_core::stage::stages::DocumentProfileStage` (name `"document-profile"`) + `UnwrapProfileStage` (`"unwrap-profile"`), @@ -66,14 +66,22 @@ produced. | `categories_raw` | `Option` carrying the originating tagged value of the top-level `categories:` key (`bd-n8a4`). Mirrors `categories` but preserves `!prefer` / `!concat` merge tags so listings consumers can feed it (alongside `listing_item.categories_raw`) into `quarto_config::MergedConfig` for tag-aware merging. Most consumers should keep reading the flattened `categories`; only listings reach for the raw form. Default `None`. | | `listing_content_globs` | `Vec` of unresolved glob strings from the host page's `listing.*.contents:` declarations (`bd-xbnf`, listings L6). Flattened across all listings on the page. The dependency-graph builder expands these against `ProjectIndex` at graph-build time (host-relative first, project-relative fallback — matches L3's render-time rule) to add forward edges from each listing host to its content files; hosts with non-empty entries are also added to the graph's `force_render` set so Mode B (`quarto render posts/foo.qmd`) pulls in listing hosts when any of their content files is targeted. Resolution is **not** cached on the profile (the per-doc cache cannot represent dependency on the full project source set safely). Default empty. | | `listing_item` | `ListingItemInfo` advertising per-document data for listings consumers (`bd-n8a4`). **Scoped feature surface — listings only**; non-listing consumers must use the corresponding top-level fields (`title`, `description`, `image`, …). Author-supplied values populate during `DocumentProfile::extract`; `ListingItemInfoStage` (`bd-izqh`, L1, landed) auto-fills holes pre-checkpoint for `description` (full first paragraph), `image` (first inline image's URL), `word_count` (Q1-parity tokenization, footnote text excluded), `reading_time_minutes` (`ceil(word_count / 200)`), and `date_modified` (filesystem mtime via `SystemRuntime::path_metadata` formatted as `YYYY-MM-DD` UTC). Author values always win — the stage strictly fills holes. The nested `extra: BTreeMap` is the **only** open-shape field in the profile and is forbidden to non-listing consumers — see §"Scoped feature surfaces". Default empty (`ListingItemInfo::is_empty()`). | +| `engine_resolution` | `Option` (`engine-resolution.md` §9.1). `Some` only when the document's engine resolution is provably load-free at Pass-1 — the needs-no-load predicate in `engine-resolution.md` §3.3 (P1–P4) — and is then **complete**: `sequence` is the resolved engine names in run order, `ownership` is the language→engine map in insertion order. `None` means resolution fell through to Pass-2's existing (non-profiled) resolution — **not an error**; most documents may show `None` until every engine a project uses is static or tabled (`engine-resolution.md` §3.3, §12). Names only, no `ConfigValue` blobs. Default `None`. | ## Non-guarantees (explicit) What a profile **does not** contain: -- **Engine output.** No values produced by executing code cells - (Jupyter, Knitr, Observable). Those require the engine stage, - which runs after the checkpoint. +- **Engine execution output.** No values produced by executing code + cells (Jupyter, Knitr, Observable). Those require the engine + stage, which runs after the checkpoint. **Engine *resolution* is + the exception to this line, not a contradiction of it:** deciding + which engine(s) will run and which owns which language is a pure, + pre-load computation (`engine-resolution.md` §9) and *is* + profile-eligible — see the `engine_resolution` field above. The + boundary is between *deciding* an owner (resolution, may be on the + profile) and *running* that owner to get a value (execution, never + on the profile). - **Sugar-synthesized structure.** No callout custom nodes, no theorem/float-target/equation-label canonicalization, no crossref numbering (`TocEntry::number`), no appendix structure, @@ -415,22 +423,35 @@ Tracking: `bd-creo` (CLI strictness), `bd-mwtf` / changed from `Vec` to `Vec` so each pattern carries its YAML `SourceInfo` for Ariadne-span diagnostics. -- **2026-07-15 — v7 (`bd-ez0hiowa`, title-block parity epic P2).** - `DOCUMENT_PROFILE_VERSION` bumped 6 → 7. One new field: - - `authors_structured: Vec` — the structured - author model (name literal + given/family components, ORCID, - email, url, degrees, attribute flags, denormalized - affiliations as `ProfileAffiliation { name, department, - url }`). Produced by the shared normalization in - `crates/quarto-core/src/metadata/authors.rs` - (`parse_authors_model`) — the same pass - `AuthorsNormalizeTransform` uses to derive the - `by-author`/`by-affiliation` metadata the HTML title block - renders. The flat `authors: Vec` field keeps its type - and now derives its literals from the same model, so the two - fields always agree. Fields the profile does not carry yet +- **2026-07 — v8 (merge of two concurrent v7 bumps).** Two branches + each bumped `DOCUMENT_PROFILE_VERSION` 6 → 7 for a different new + field; the ts-engine-extensions rebase merged them, so both fields + coexist under **v8** (there is no single-field v7 on the merged + line). v6/v7 cache entries on disk are rejected with + `DocumentProfileError::VersionMismatch` and silently regenerated, + identical to every prior bump. Both new fields: + - `authors_structured: Vec` (`bd-ez0hiowa`, + title-block parity epic P2) — the structured author model (name + literal + given/family components, ORCID, email, url, degrees, + attribute flags, denormalized affiliations as + `ProfileAffiliation { name, department, url }`). Produced by the + shared normalization in `crates/quarto-core/src/metadata/authors.rs` + (`parse_authors_model`) — the same pass `AuthorsNormalizeTransform` + uses to derive the `by-author`/`by-affiliation` metadata the HTML + title block renders. The flat `authors: Vec` field keeps + its type and now derives its literals from the same model, so the + two fields always agree. Fields the profile does not carry yet (roles, notes, funding) join later with another bump. - v6 cache entries on disk are rejected with - `DocumentProfileError::VersionMismatch` and silently - regenerated, identical to every prior bump. - Plan: `claude-notes/plans/2026-07-15-html-title-block-parity.md`. + Plan: `claude-notes/plans/2026-07-15-html-title-block-parity.md`. + - `engine_resolution: Option` (plan6, + Pass-1 engine resolution) — the per-document engine resolution, + additive at the on-disk layer (`skip_serializing_if` keeps default + profiles compact), stamped only when Pass-1 can resolve it without + loading an engine (`engine-resolution.md`'s needs-no-load + predicate, §3.3/§7/§9.1). `None` means the doc falls through to + Pass-2's existing resolution — not an error. Names only, no + `ConfigValue` blobs: `sequence` is the resolved engine names in run + order, `ownership` is the language→engine map in insertion order. + Feeds the LSP today; future freeze and kernel-pooling consumers + later (`engine-resolution.md` §12). + Plan: `claude-notes/plans/2026-06-29-plan6-pass1-engine-resolution.md`. diff --git a/claude-notes/designs/engine-and-engines-keys.md b/claude-notes/designs/engine-and-engines-keys.md new file mode 100644 index 000000000..008362cc6 --- /dev/null +++ b/claude-notes/designs/engine-and-engines-keys.md @@ -0,0 +1,212 @@ +# The `engine:` and `engines:` metadata keys (design contract) + +**Status:** design contract — authoritative for what each key means and, +just as importantly, what each key is guaranteed *not* to do. Describes the +settled behavior delivered by the **TS Engines Epic**; § History records +how the two keys reached it. Referenced by that epic's implementation +plans. +**Created:** 2026-07-06 (during the TS Engines Epic's engine-key design +work). +**Companion contract:** `engine-resolution.md` owns the resolution +algorithm these keys feed (claim kinds, tiers, ownership); this document +owns the *user-facing grammar* — which key to write, and why. + +--- + +## 1. Why there are two keys + +Quarto has always had two engine-related configuration keys, and they have +always been easy to confuse — one letter apart, both about engines. The +q2 design resolves the confusion by giving each key exactly one question +to answer: + +- **`engine:`** answers *"which engines are at play in this document?"* +- **`engines:`** answers *"how do engines behave, whichever ones end up + at play?"* + +Everything else in this document is a consequence of that division. When +you are deciding which key to write, ask which question you are answering: +if you are picking participants, write `engine:`; if you are adjusting the +behavior of a participant someone else might pick, write `engines:`. + +## 2. `engine:` — naming the engines at play + +A document's `engine:` key declares its **execution sequence**: the +engines that will run, in the order they will run. Because the key *names +participants*, writing it is committal — q2 treats an explicit list as the +author's complete statement of who plays, and three consequences follow: + +1. **The implicit-fallback safety net turns off.** Normally, a + computational language that no engine claims falls to whichever + registered engine makes the strongest `Fallback` claim — jupyter by + default, but any engine may declare fallback claims and outbid it (the + "T4" tier in `engine-resolution.md` §4.3). An explicit `engine:` list + disables that whole tier: if your list doesn't cover a language, no + unlisted engine is quietly added to cover it. (A *listed* engine's + fallback claims still catch leftovers — that is the explicit-fallback + tier, T2.) +2. **Listed engines are present by declaration.** An engine's *interop* + claims (knitr's reticulate taking `{python}`, for example) fire only + when the engine is already present. Listing an engine makes it + present, even in a document where it wins no language on its own. +3. **Order is execution order.** Engines run sequentially, each consuming + the previous engine's output — so a generator engine must be listed + before the engine that executes what it generates. + +The key accepts three shapes: + +```yaml +engine: knitr # scalar: a one-engine sequence +engine: [knitr, jupyter] # array: N engines, in order +engine: + - jupyter: # entry with config: the map's value + kernel: python3 # is threaded to the engine at + # execute time +``` + +A per-entry config may also carry one reserved key, `claims:`, a +document-level **claim table** for the listed engine (see §3 — the table +semantics are identical on both keys). `claims:` is resolution metadata, +not engine configuration, so q2 strips it before the config reaches the +engine. + +One q2-specific behavior deserves care: `engine:` is read from **merged +metadata**, so a project-level `engine:` in `_quarto.yml` works and +concatenates with a document's own list. When both layers name the same +engine, the *project's* entry wins (array layers concatenate project-first +and deduplication keeps the first occurrence); q2 warns about the conflict +and points at the `!prefer` merge tag for documents that need to override. +If what you actually wanted at the project level was engine +*configuration* — not forcing every document's sequence — the right key is +`engines:`, which is the subject of the next section. + +## 3. `engines:` — configuring engines without naming any + +The project-level `engines:` key configures the **engine registry**: the +pool of engines (built-ins plus discovered extensions) that resolution +draws from. Writing it never puts an engine into play. Its guarantees are +the mirror image of `engine:`'s consequences, and they are worth stating +as guarantees because they are what make the key safe to use project-wide: + +- it never makes any document's sequence explicit (the implicit-fallback + tier stays on); +- it never makes an engine "present" (no interop side effects); +- it never causes an engine to run that wouldn't have run anyway. + +The one selection-adjacent influence it retains is **ordering** — and it +reaches a little further than a tie-break. The order engines are visited in +(the *candidate order*: `engines:` entries first, then extensions in the +order they were discovered, then the built-ins) does two jobs. It breaks +equal-strength claim ties — when two engines make the same-kind, +same-priority claim on a language, the earlier one wins it. And it sets the +order in which the chosen engines actually run: a document with `{r}` cells +knitr claims and `{julia}` cells a Julia extension claims runs them in +candidate order — the extension, then knitr — **not** in the order the +cells appear in the file. So `engines:` never *casts* anyone: claims decide +who runs, and the implicit-fallback net still fills the gaps. It only sets +the running order of whoever the claims chose. Configuration, not casting — +the configuration just happens to include sequence. + +The key is a Q1-syntax-compatible array whose entries come in three forms: + +```yaml +engines: + - knitr # string: ordering only + - path: ./my-engine.js # Q1's external-engine loader; + # reserved (see §4) + - legacy-python: # name-keyed map: per-engine config; + claims: [python] # `claims` is the only config key +``` + +The `claims:` value is a **claim table**: a complete replacement for the +engine's `_extension.yml` `claims:` block, with the same schema and the +same authority. "Complete" is the operative word — a table replaces the +engine's *entire* claim surface, so a language absent from the table is +simply not claimed (unless the table carries a universal `fallback:` +entry, which claims everything at the fallback floor, exactly as it would +in `_extension.yml`). Two consequences make tables the key's headline +feature: + +- **A tabled engine is load-free.** Resolution answers its language + claims from the table without loading any code — which is what lets a + project's execution languages be resolved at index time, without ever + loading a legacy, claims-less extension, from one block of YAML and no + edit to the extension. +- **An empty table is a mask.** `claims: []` means "this engine claims + nothing" — including, if you apply it to jupyter, disabling the + universal fallback project-wide. + +The full table semantics — source precedence, masking built-ins, +priority-based forcing, validation policy — live in +`engine-resolution.md` §3.3; this document only needs the shape. + +Two policies round out the key. A map entry naming an engine that is not +in the registry is a **hard error at project load** — one failure, early, +before any document is rendered, with Q1's message. And although `engines:` +is a project-level key, q2 reads it from merged metadata, so a +document-frontmatter `engines:` block also takes effect — an implementation +artifact, not a supported surface; do not rely on it. + +## 4. Differences from Q1 + +Readers coming from Quarto 1 should unlearn three things. + +**Q1's `engine:` chose one engine; q2's chooses a sequence.** In Q1 the +key (or a top-level engine-name shorthand like `jupyter: python3`) named +the single engine that ran the whole document. q2 keeps every Q1 spelling +and generalizes the meaning: the array form declares N engines that run in +order, with per-language ownership divided among them by the resolution +tiers. A second, quieter difference hides in *where* the key is read: Q1 +consulted only the file's own frontmatter, so an `engine:` in +`_quarto.yml` was silently inert; q2 reads merged metadata, so the project +layer participates (with the project-wins-on-duplicates rule from §2). + +**Q1's `engines:` did loading and ordering; q2's does configuration and +ordering.** Q1's entries were engine names (ordering) and `{path: ...}` +objects that dynamically imported external engine modules. q2 replaces the +loading role entirely — engines arrive via `_extensions/` discovery, and +`path:` entries are reserved rather than honored — and adds a role Q1 +never had: per-engine configuration via claim tables. The ordering role +carries over unchanged in spirit (user-listed engines are consulted +first). + +**The selection/configuration boundary is sharper in q2.** Q1's `engines:` +order could effectively *select* the winner because Q1's claims were bare +scores and ties were common. q2's kind-tagged claims (`Primary` / +`Interop` / `Fallback`, kind dominating priority) make ordering rarely +*decide a contest* — a genuine equal-kind, equal-priority tie is uncommon. +What ordering still always does is set the **run order** of a multi-engine +sequence (§3): *who* runs is decided by `engine:` and by claims, but the +order they run in is candidate order, which `engines:` shapes. So `engines:` +stays on the configuration side of the line — it just configures sequence as +well as claims. + +## 5. History of the two keys in q2 (condensed) + +**`engine:`** has existed since the first engine-detection commit +(2026-01-07, `748856f50`) — already scalar, config-map, and shorthand +forms — and gained the array form with sequential multi-engine execution +(2026-05-29, `f34c20dbb`, #238). The epic reserves one key inside an +entry's config, `claims:`, a document-level claim table that replaces the +engine's claims (§2). + +**`engines:`** had no q2 meaning before the TS Engines Epic — a project +that set it was setting an inert key. The epic gives it three roles: a +verbatim **wire pass-through** to TS engines (for Q1-API parity — engines +may read `project.config.engines` — narrowed to names), the **claim-table +entries** that are its first Rust-side semantics, and the **ordering +splice** in `build_engine_registry`. + +## 6. Choosing between them: three worked one-liners + +- *"This document should run knitr, then jupyter."* → `engine: [knitr, + jupyter]` in the document. You are naming participants. +- *"Our legacy extension has no static claims and the language server + can't index our project."* → in `_quarto.yml`: + `engines: [{legacy-python: {claims: [python]}}]`. You are configuring a + participant; every document resolves at index time and no document's + sequence changes. +- *"knitr keeps grabbing our python cells via reticulate, project-wide."* + → `engines: [{knitr: {claims: {r: primary}}}]`. A whole-table + replacement that omits python — masking, again without touching any + document's cast. diff --git a/claude-notes/designs/engine-api-surface.md b/claude-notes/designs/engine-api-surface.md new file mode 100644 index 000000000..6dddcda5b --- /dev/null +++ b/claude-notes/designs/engine-api-surface.md @@ -0,0 +1,226 @@ +# Engine API surface — q2 wire vs the full Q1 engine surface + +**Extracted from** `claude-notes/plans/2026-06-25-plan1a-return-to-q1.md` (RTQ) on 2026-06-26. +This is the surface-coverage record — the field-by-field / method-by-method audit of the q2 +engine wire against the full Q1 engine surface, plus the framework decisions (DQ-1 … DQ-7), all +**resolved 2026-06-29**. RTQ carries the actionable protocol/code items (Item A, ENG-1, FC-1, FC-2); +this doc records the surface classification and the decisions behind it. + +**Companion (consumed-surface model):** this doc audits the engine-**PROVIDED** / wire half (what +q2 *sends* and *receives*). The **CONSUMED** half — the `quarto..` calls engines make +*back* — is modeled in `claude-notes/research/2026-06-26-engine-api-usage-model.md`. The two are +two sides of the same surface. + +## Governing principle — the validation target is not the scope boundary + +**Carry the whole Q1 engine protocol surface. The scope boundary is the *Q1 engine API*, not what +the current *validation target* (Julia, single-doc, single-engine) happens to exercise.** "Defer +features, not infrastructure" is the operational form of this: where Q1 exposes an engine +method / field / flag, the q2 **infrastructure** to carry it is in scope **now** (classified +`build-infra` or `defer-infra` with a recorded seam below), even with zero current callers. Only +`drop` when q2's architecture makes it impossible or redundant — *with a reason*. A feature (no q2 +producer/consumer yet) may be deferred; the protocol that would carry it may not be silently +narrowed. + +**The failure mode this prevents.** The 1a/1b plans were drafted against the Julia render path and +repeatedly scoped the wire to *what that path needs*, dropping protocol surface no Julia render +touches. Because the drops are invisible on the Julia validation target, they pass every test and +surface only later as a coordinated retrofit. Two confirmed instances — **the same class of bug +twice**: + +- **`system.execProcess` params** (RTQ F1 / `2026-06-26-plan2a-review-findings.md` §2a-1): q2 + reduced the signature to `(options, stdin?)`, dropping `mergeOutput`/`stderrFilter` because *no + TS-extension* uses them — but **knitr does** (`rmd.ts:440-458`), and the SDK is advertised as + Q1-consumable. *Consumed-surface half; lives in the usage model.* +- **`dependencies` flag + deferred-deps fold** (1B-DEPS-2 / `2026-06-26-1b-vs-usage-model-reconciled.md`): + 1b hardcoded `dependencies:true` + a dead fold because *no single-file book exists yet* — dropping + the deferred path, which is Q1 **protocol** (its first consumer is single-file book rendering, + `book-render.ts:136`). *Provided-surface half — note this doc already classes the flag + `build-infra` (Level 1); 1b diverged from that classification.* + +Both are Julia-invisible (Julia's `execProcess` unused; Julia's `dependencies()` a no-op) — exactly +why a Julia-only review misses them. + +A third near-instance — **caught before landing**, recorded so the pattern stays visible: + +- **Static-claim schema expressiveness** (plan1c review, 2026-06-28): the D1 static-claims schema + was first shaped to the **echo** fixture (language-only + extension-only) and initially could not + express marimo's `first_class`-conditional claim (`{python .marimo}`). Fixed before landing by + adding `whenClass:` — `claims_language` is a pure function of `(language, first_class)`, so it + tabulates fully — so the static surface is **not** narrowed. This review originally recorded a + residue — **content-inspecting `claims_file`** (Julia's `# %%`) — as the one place static resolution + was strictly less powerful. **Corrected 2026-07-07:** a full Q1 census showed every content sniff is + `extension-gate → read-file → one regex` (a pure function of file bytes), so it *is* statically + declarable — as a `content-pattern` on a `claims-files` entry, evaluated natively (Plan 7a). There is + **no** static-vs-dynamic residue in practice; the dynamic `claims_file` method survives only as a + fallback for a hypothetical non-regex-expressible sniff (empty across every known engine). See + engine-resolution.md §3.3 and + [Plan 7a](../plans/2026-07-07-plan7a-static-content-pattern-claims.md). + +**The author test.** For each Q1 engine method / field / flag, do **not** ask "does the Julia +validation target need this?" Ask: *does the Q1 engine API expose it as protocol, and could a +non-Julia engine or a not-yet-built render mode (books, manuscripts, serve, multi-engine) use it?* +If yes → infrastructure is in scope now (`build-infra`/`defer-infra` + seam). The validation target +proves the framework works; it does not define the framework's surface. + +## Surface coverage audit — q2 wire vs the full Q1 engine surface + +Both levels, **Q1 read directly** (`execute/types.ts:35-243`, `project/types.ts:164-216`, the +lifecycle model). The original 1a wire carried the Julia render path and omitted the rest of the +engine surface. Per "defer features, not infrastructure," each Q1 engine method/field is classed: +**present** (on the wire) · **build-infra** (framework should carry now; q2 consumer may stub) · +**defer-infra** (real but premature — record the seam) · **drop** (q2 architecture makes it +impossible/redundant, reason given). + +### Level 1 — inbound options & context (field-by-field) + +**`ExecuteOptions` → `TsExecuteOptions`:** + +| Q1 field | q2 | class | note | +|---|---|---|---| +| `target: ExecutionTarget` | flattened → `input`/`source_path`/`source_map` | **drop** | no `target()` step (DQ-3); cookie + `data` absent — `engine_state` added only if a round-trip later needs it | +| `format` | `format` | present | | +| `resourceDir` | `Init { global }` (ambient, Item A) | present | | +| `tempDir`/`cwd`/`libDir?` | `temp_dir`/`cwd`/`lib_dir` | present | | +| `dependencies: boolean` | wire flag (default `true`) | **build now** | build the round-trip — flag + `Dependencies` verb + `engineDependencies` (FC-2); orchestrator-driven, harness fold deleted (DQ-2) | +| `projectDir?` | `project_dir` | present | | +| `params?`/`quiet?` | `params`/`quiet` | present | | +| `previewServer?` | — | defer | run/serve — deferred behind a seam (DQ-2) | +| `handledLanguages` | `handled_languages` | present | leave-alone set (§5); `HANDLED_LANGUAGES ∪ {lang owned by others}` | +| `project: ProjectContext` | `project_dir` + launch `EngineProjectContext` | present | `config` + output-dir carried as values (DQ-5); the two callback members dropped (DQ-1) | +| — | `source_map` | q2-native | provenance addition | +| — | `owned_languages` | q2-native | positive projection of the ownership map (§5, Plan 4d) — `{lang owned by this engine}`; informational (not enforcement), so engines select owned cells directly instead of inferring the complement of `handledLanguages` | + +**`ExecutionTarget`** (Q1 cookie; q2 has **no `target()` step**): + +| Q1 field | q2 | class | note | +|---|---|---|---| +| `source`/`input`/`markdown` | `source_path`/`input` (resolved markdown pushed) | present | | +| `metadata` | folded into `format.metadata` | verify | confirm target-vs-format metadata overlap (residual verify, not a design question) | +| `data?` (engine cookie) | — (q2 uses `engine_config` + lazy resolve in execute) | **drop** | no `target()` (DQ-3); jupyter `{transient, kernelspec}` resolved lazily in execute | +| `preEngineExecuteResults?` | — | defer-infra | cell-handler pre-results | + +**`EngineProjectContext`** (passed to `launch()` after Item A; q2 carries only `{dir, isSingleFile}`): + +| Q1 member | q2 | class | note | +|---|---|---|---| +| `dir` / `isSingleFile` | present | present | | +| `config?` (`engines`, `output-dir`) | carry as values on launch ctx | **build now** | cheap serializable values (DQ-5) | +| `getOutputDirectory()` | pass output dir as a value | **build now** | not a callback — a value on the launch ctx (DQ-5) | +| `fileInformationCache` | — | **drop** | host-owned live `Map`, not serializable; sole engine read is jupyter `keep-ipynb` transient tracking via `target.data` — a cookie q2 drops (DQ-3) and a file-lifecycle q2 owns host-side (DQ-1) | +| `resolveFullMarkdownForFile()` | — (q2 pushes resolved `input`) | **drop** | push model; the callback was used only by obsolete engine methods (DQ-1) | + +### Level 2 — method surface + +**Discovery (`ExecutionEngineDiscovery`):** + +| Q1 member | q2 | class | note | +|---|---|---|---| +| `init?`/`name`/`validExtensions`/`claimsFile`/`claimsLanguage`/`launch` | present | present | | +| `generatesFigures` | → `LoadEngineResult` (discovery) | **build now** | move to discovery tier — ENG-1 (DQ-4) | +| `canFreeze` | → `LoadEngineResult` **and** instance | **build now** | add to discovery for freeze-planning; keep on instance (Q1 has both) — DQ-4 | +| `quartoRequired?` | → `LoadEngineResult` (discovery) | **build now** | load-time semver gate (grand-plan Phase 12) — DQ-4 | +| `ignoreDirs?` | — | out of scope | project file-walk — out of the render wire (DQ-6) | +| `defaultExt`/`defaultYaml`/`defaultContent` | — | out of scope | scaffolding (`quarto create`) — own command surface (DQ-6) | +| `checkInstallation?` | — | out of scope | `quarto check ` — own command surface (DQ-6) | +| `populateCommand?` | — | **own command surface** (was "drop") | `q2 call engine …` — see Plan 9 correction below (DQ-6) | + +**Instance (`ExecutionEngineInstance`):** + +| Q1 method | q2 | class | note | +|---|---|---|---| +| `markdownForFile`/`execute`/`intermediateFiles?` | present | present | | +| `target` | — | **drop** | per-file cookie folded into execute; no `target()` step (DQ-3) | +| `dependencies` | → wire verb | **build now** | deferred-deps round-trip built — FC-2 (DQ-2) | +| `postprocess` | — | **drop** | recover via an **AST transform** that reads FC-1's already-carried `preserve` field (the No-DOM-postprocessor rule). Connected seam: **FC-1 (carries `preserve`/`post_process`) ↔ this dropped `postprocess` hook ↔ Plan 3's `removeAndPreserveHtml` producer** (`quarto-jupyter.md:134-136`). (DQ-2) | +| `partitionedMarkdown` | — | **drop** | pampa parses qmd natively after `markdownForFile`; engine partition is redundant | +| `filterFormat?` | — | defer | engine influences `Format` pre-render — behind a seam | +| `run?`/`postRender?` | — | defer | serve/preview + after-render (server-backed engines) — documented seam (DQ-2) | +| `canKeepSource?`/`executeTargetSkipped?` | — | defer-infra | keep-md / freeze-skip hooks | + +### Resolved framework decisions (DQ-1 … DQ-7) + +These were the open framework questions in the 2026-06-25/26 sessions; **all are decided +(2026-06-29).** Recorded here as resolved facts — the per-surface tables above carry the same +outcomes inline. The actionable protocol/code changes live in RTQ +(`2026-06-25-plan1a-return-to-q1.md`) as **Item A, ENG-1, FC-1, FC-2**. + +- **DQ-1 — engine→host callbacks → push model; callbacks dropped.** q2 pushes fully-resolved `input` + markdown into execute, so `resolveFullMarkdownForFile` is unnecessary (Q1 engines called it only + from obsolete methods). `fileInformationCache` drops with it: it is a host-owned live + `Map`, not serializable across the subprocess; its sole engine reader is + jupyter's `keep-ipynb` transient-notebook bookkeeping, which mutates `target.data` — a cookie q2 + drops (DQ-3) and a file-lifecycle q2 owns host-side. **No `FromEngine`-initiated callback channel + in v1** (re-entrancy/deadlock risk on the single Deno thread; no consumer). +- **DQ-2 — render lifecycle → build the `dependencies` round-trip (FC-2), orchestrator-driven; drop + `postprocess`; defer `run`/`postRender`.** The wire gains `dependencies: bool` (default `true`) + + `ToEngine::Dependencies`/`FromEngine::DependenciesResult` + `engineDependencies` on the result; + **q2's render orchestrator drives the deferred resolution** (mirrors Q1 `render.ts:90-109`), **not** + a harness-internal fold. `postprocess` is dropped (no post-write DOM stage; recover via an AST + transform reading FC-1's `preserve`). `run`/`postRender` deferred behind a documented seam. +- **DQ-3 — per-file engine-state cookie → no `target()` step in v1.** q2's lazy-resolve + + `engine_config` is sufficient; add an opaque `engine_state` field **only if** a future + `dependencies` round-trip forces an engine to thread state across it. +- **DQ-4 — discovery-tier completeness → expand `LoadEngineResult` to + `{name, valid_extensions, generates_figures, can_freeze, quarto_required}`.** All static, + cheap-at-load, read pre-launch by resolution / freeze-planning / version-gating. ENG-1 is the first + landed slice (`generates_figures` + `can_freeze`); `canFreeze` stays on the instance result too + (Q1 has it at both tiers). +- **DQ-5 — `EngineProjectContext` completeness → carry `config` (`engines` + project `output-dir`) + and the output directory as values** on the launch context. The two callback members are DQ-1 + (dropped). +- **DQ-6 — non-render surfaces → out of the render wire.** `populateCommand`, + `defaultExt`/`defaultYaml`/`defaultContent` (scaffolding), `checkInstallation` (check), + `ignoreDirs` (project walk) belong to their own command surfaces — designed when those commands + grow engine-awareness, not bolted onto the execute protocol. + - **`populateCommand` correction (Plan 9, 2026-07-03, bd-m1jeqhhz).** This was + originally filed a *hard drop* ("cliffy subcommands into a Rust CLI — + impossible"). That is wrong: `q2 call engine …` runs the engine's + *own* cliffy `populateCommand` in a short-lived Deno process (the `call-engine` + host-bundle mode, vendored cliffy) with inherited stdio, giving byte-for-byte Q1 + parity. It stays true to DQ-6 — it is its **own command surface**, off the render + wire (no new protocol verb; a separate one-shot process, not the shared render + host). The trait side is an additive `call_engine_command` default. See + `claude-notes/plans/2026-07-03-plan9-call-engine.md` and + `claude-notes/research/2026-07-03-plan9-call-engine-research.md`. +- **DQ-7 — Init/launch split → `Init { global }` once per subprocess (process-stable config); + project context rides `LaunchEngine` per render** (Item A). This **supersedes** the earlier + "`Init { global, project }`" recommendation: moving project context to `LaunchEngine` is strictly + more Q1-faithful (`engine.launch(EngineProjectContext)`) and is the enabler for reusing one + subprocess across renders (`Init` stays process-stable; each render's `launchEngine` carries its + own project context). + +### Build checklist (decision → landing item) + +All boxes are **unimplemented** — RTQ is plan-only, not yet executed. Each box is the protocol/code +change a decision requires; the **owning RTQ item** is named so this stays a coverage map, not a +competing source of truth. A box marked **(no owner)** has no RTQ item yet and must get one in the +consolidation pass. + +- [ ] **DQ-1** — push resolved `input`; no `FromEngine` callback channel; harness builds a + *harness-local* `fileInformationCache` (not a wire carrier) — *RTQ Item A (plan1b harness)* +- [ ] **DQ-2** — `dependencies: bool` (default `true`) + `ToEngine::Dependencies` / + `FromEngine::DependenciesResult` + `engineDependencies` on the result; orchestrator-driven; + delete the harness fold — *RTQ FC-2* +- [ ] **DQ-2** — drop `postprocess`; carry `preserve`/`post_process` and recover via an AST transform + — *RTQ FC-1 (carrier) + B2 (recovery story) + PROTO-1 (disposition)* +- [ ] **DQ-4** — move `generates_figures` → `LoadEngineResult`; add `can_freeze` there (keep on the + instance result) — *RTQ ENG-1* +- [ ] **DQ-4** — add `quarto_required: Option` to `LoadEngineResult` (discovery tier). Splits + in two: **field → fold into ENG-1** (same `ts_protocol.rs` / harness / test-helper change as the + DQ-4 tier completion — carry it inert, no v1 engine sets it); **load-time semver gate → + grand-plan Phase 12** (reuse its `semver` / `VersionReq` / `cli_version()` machinery). Q1: + `quartoRequired?` on `ExecutionEngineDiscovery` (`execute/types.ts:65`), gated by + `checkEngineVersionRequirement` (`engine.ts:61`, **hard throw**; cf. Phase 12's extension-YAML + `quarto-required`, which **warns** — q2 must pick one severity). +- [x] **DQ-5** — carry `config` (`engines` + project `output-dir`) and the output directory as + *values* on `LaunchEngine.project` — *RTQ Item A* (Plan 1c.2 P1.1, 2026-07-02: wired in + `build_engine_registry`; wire `config` carries `engines` + **flat** top-level `output-dir` — + the host's `reconstructRichProject` bridges it into the rich `config.project.outputDir`) +- [ ] **DQ-7** — `Init { global }` once per subprocess; project context on `LaunchEngine` per render + — *RTQ Item A (sequence its `ts_protocol.rs` edit with ENG-1)* + +**Decided, no build (recorded for completeness):** DQ-2 `run`/`postRender` — deferred behind a +documented seam (name it in RTQ/grand-plan, no code now); DQ-3 — no `target()` step, add +`engine_state` only if a future round-trip needs it; DQ-6 — non-render surfaces stay off the render +wire. diff --git a/claude-notes/designs/engine-host-concurrency.md b/claude-notes/designs/engine-host-concurrency.md new file mode 100644 index 000000000..b13a64f06 --- /dev/null +++ b/claude-notes/designs/engine-host-concurrency.md @@ -0,0 +1,270 @@ +# Engine-host concurrency: async multiplexing over one Deno subprocess + +**Status:** canonical reference for the TS-engine subprocess concurrency model. +Pointed at by plan1a-protocol (Phase 1.5), plan1a-host, plan1a-engine, and +Plan 1b. + +**Why this exists.** The original Plan 1a design assumed a **lockstep** +request/response protocol over stdio: one request, one response, no +correlation, serialized by a single `Mutex`, with whole-subprocess +SIGKILL on timeout/cancel. That was correct while **Pass-2 (per-file render) +was serial**. Pass-2 is now **parallel** (rayon + `pollster`-per-worker), and +all workers share **one Deno subprocess per project render**. Under the lockstep +model that means (a) every engine round-trip serializes through one lock, and +(b) one document's timeout/cancel SIGKILLs the subprocess out from under +siblings still mid-`execute`. This note defines the replacement. + +## The key realization + +**Deno is single-threaded but asynchronous.** One process can hold many +requests in flight at once — engine A's `execute()` parked on a Julia daemon +socket while engine B's is parked on a Jupyter kernel — with the event loop +interleaving them. Nothing about "one subprocess" forces serialization; only +our **framing** did (**one-in-flight, no correlation** — independent of which +OS channel carries it). The fix is to reframe the one process as an **async +multiplexed RPC channel** over a dedicated transport — originally the +*existing* stdin/stdout (v1), and since Plan 1a.6 a private loopback-TCP +socket (see "Phase 1.6" below) — not to spawn N processes (which wastes +exactly that async capability and N×s the memory). + +## Architecture (three layers) + +### 1. Protocol (plan1a-protocol Phase 1.5) + +- **Correlation envelope.** Every frame carries a monotonic `id: u64`; the + response echoes it. Nested envelope (`{ id, msg: {type, …} }`), not + `serde(flatten)` — flatten round-trips poorly with internally-tagged enums. + The existing `ToEngine`/`FromEngine` enums are unchanged. +- **Cooperative cancel.** `ToEngine::Cancel { target: u64 }` (fire-and-forget, + references an in-flight id) and `FromEngine::Cancelled {}` (delivered under + the target id). +- **Channel: stdio in the original v1 design; loopback TCP since Plan 1a.6.** + The envelope + `Cancel` are all that parallel Pass-2 *requires* — they + multiplex fine over a JSON-lines channel (the harness write-serializes its + frames; single-threaded Deno writes each line atomically, so frames never + interleave), independent of which OS channel carries it. In the original v1 + design, stdout was the protocol channel, so the "stdout is protocol; a stray + `console.log`/non-JSON line is malformed → kill" contract held at the time + (owned by plan1a-host), and it was two-sided: stdin was the protocol *input*, + so an engine reading `Deno.stdin` stole frames just as writing to stdout + corrupted output. **That contract is gone.** Plan 1a.6 has since landed (see + "Phase 1.6" below and + `claude-notes/plans/2026-07-08-plan1a6-off-stdout-loopback-tcp.md`): the + protocol now rides a private loopback-TCP socket, and stdout/stderr are + diagnostic-only — a stray `console.log` no longer corrupts anything. The + bidirectional **continuous drain** (q2's demux reader thread + the harness's + non-blocking loop) is still what keeps a large `Execute` payload from + deadlocking the channel — load-bearing regardless of transport, not + optional. The `EngineTransport` trait already abstracted the channel, which + is why the swap to TCP was localized. + +### 2. Rust host — a demux, not a Mutex (plan1a-host) + +Standard async-RPC-over-a-pipe, composed entirely from **blocking std +primitives** so it fits blocking rayon+`pollster` workers: + +- `TsEngineHost` owns a **write half behind a short-lived mutex** (held only for + the microseconds of one framed write) and a **`pending: Mutex>`**. +- One **reader thread** owns the read half (the accepted loopback-TCP socket's + read half since Plan 1a.6; the child's stdout in the original v1 design), parses + each `Response`, reads its `id`, and delivers to `pending.remove(id)`. A + response whose `id` is no longer pending (late reply after a cancel) is + dropped. +- A worker calls `host.request(msg, window, &cancellation)`: allocate `id` → + register a blocking `Slot` (e.g. `sync_channel(1)`) → write the framed line + under the write-mutex → block on the slot with `recv_timeout`, polling + `is_cancelled()` on each tick. On timeout/cancel it sends `Cancel { target: + id }` and resolves the slot with a cancelled/timeout error. + +**The crucial property: no lock is held across the wait.** Worker A blocking on +a 5-minute `Execute` does not block worker B's discovery call — they hold +different slots; the write-mutex and pending-mutex are each held for +microseconds. Head-of-line blocking is gone. + +### 3. Deno harness — a non-blocking read loop (Plan 1b) + +```ts +for await (const frame of readFrames(conn)) { // originally Deno.stdin (v1); now the loopback-TCP conn (Plan 1a.6) + if (frame.msg.type === "cancel") { abort(frame.msg.target); continue; } + dispatch(frame); // fire-and-forget; the loop does NOT await engine work +} +``` + +`dispatch` runs the request **serialized per engine instance** (chained on a +per-engine promise queue) and, when done, writes `{ id, msg: result }` to the +protocol channel (the loopback-TCP connection since Plan 1a.6; the captured +`Deno.stdout` in the original v1 design) under a write-mutex. Because +the read loop never awaits engine work, requests to *different* engines run +concurrently on the event loop; the per-engine queue serializes requests to the +*same* instance. An `AbortController` per `id` implements `Cancel`. + +## The concurrency ceiling (and why it's physics, not a compromise) + +- **Document pipeline** (parse → merge → transforms → write): fully parallel + across workers, always — it never touches the subprocess. +- **Cross-engine execution:** parallel (distinct daemons, interleaved on the + Deno event loop). +- **Same-engine-instance execution:** serial — and this is **forced by the + daemon, not chosen.** There is one `julia` instance per render talking to one + Julia daemon (single fixed `julia_transport.txt` per runtime dir); Jupyter + shares a kernel keyed by (kernelspec, target). Two concurrent Julia documents + *cannot* truly parallelize their execute — they'd be two render requests to + one kernel. Serializing same-engine costs nothing we could have had anyway. + +Escape hatch if same-engine parallelism ever matters: instance-per-worker (N +kernels). Not built now — for Julia it wouldn't even help (single shared +daemon). + +## Cancellation / timeout — contained blast radius + +A per-document timeout/cancel **must not SIGKILL the shared subprocess** (that +murders sibling documents). So: + +1. Per-request timeout/cancel → send `Cancel { target: id }` → harness aborts + *that task's* `AbortSignal`. Contained to one request. +2. **Poison policy — scoped by request type, not by guessing daemon state.** + Rust cannot distinguish "ambiguous" from "clean" (the daemon is a separate + process; aborting the JS-side `AbortSignal` never proves it went idle). So + don't classify — scope by which request was interrupted. **`Execute` is the + only daemon-engaging request v1 issues** (this is a v1 fact, not an absolute: + Q1's `dependencies()`/`run()` also spawn subprocesses — knitr's `Rscript`, + `rmd.ts:407-489` — so the poison scope must be revisited if/when those + requests are added), so cancel/timeout of an `Execute` **always poisons that + engine instance**; every other request engages no daemon and is just failed. Poison = invalidate the instance on both sides (harness drops + its `instance` entry; `TsEngine` clears its cached launched-state — which is + why that cache is a clearable `Mutex>`, not a `OnceLock`), so the + next instance request re-runs `LaunchEngine` (~0) and gets a fresh + `ExecutionEngineInstance` re-discovering/restarting the detached daemon. + Blast radius shrinks from "whole subprocess" to "one engine instance." *(Future + opt-in: an engine that performs a real interrupt may carry `clean: true` on + `Cancelled` to skip the poison — not v1.)* +3. **Concurrent same-instance requests transparently re-launch — they are not + failed.** Parallel Pass-2 means the poisoned instance can have a *concurrent* + user, not just a future one: worker A's `Execute` times out and poisons the + julia instance while worker B has an `Execute` queued behind it on the + harness's per-engine queue (same instance ⇒ serialized). When the harness + dequeues B and finds the instance entry dropped, it **re-runs + `engine.launch(stashedContext)` to reconstruct it, then runs B** — B never + sees a half-torn-down instance and never fails for A's timeout. (Composes + with idempotency: a *present* instance makes `LaunchEngine` a no-op; a + *missing* one triggers exactly one lazy re-construct, whether the trigger is + B's queued request or a fresh `LaunchEngine` from q2.) Self-healing: if the + detached daemon is genuinely wedged by A's aborted-but-still-running work, + B's re-run hits **its own** `window` and poisons again on its own merits — + no special-casing needed. This is plan1b's harness contract; q2 is unaware. +4. **Whole-subprocess SIGKILL is reserved** for what genuinely affects everyone: + subprocess crash, a compromised/unparseable control channel, and final + teardown. **This is safe only because execution daemons are spawned truly + detached** (own process group / `Deno.Command … detached`; Q1: julia control + server, `julia-engine.ts:377`), so SIGKILL of the subprocess never cascades to + the daemon. A harness that spawns a daemon *inside* the subprocess's process + group breaks this **silently** — lost daemon warmth, a perf regression that + won't surface in tests. Spawning execution daemons detached is therefore a + **plan1b harness contract.** + +**Two invariants this poison/relaunch design rests on.** + +- **The cached launched-instance is stateless.** It holds no + kernel/socket/dependency state; all durable execution state lives in the + detached, transport-file-keyed daemon (model §4.2). That is *why* poison can + drop the instance and re-`LaunchEngine` cheaply, and why the cache is a + clearable `Mutex>` rather than a `OnceLock`. If a cached instance + ever starts holding durable state, this cache (and the poison design) must be + revisited. +- **q2 never touches engine transport files.** Survive-respawn — a re-launched + instance reconnecting to its still-running daemon — is delegated wholly to the + engine via its on-disk transport file (Q1: `jupyter-kernel.ts:305-324`, + `julia-engine.ts:962-964`). q2 never reads, writes, deletes, or keys on those + files; it owns only the subprocess lifecycle. This hands-off contract is what + makes the cheap drop-and-relaunch above correct. + +This reverses the original plan's "no cooperative cancel; SIGKILL is the honest +path" stance. That reasoning was sound only because SIGKILL was assumed to +coincide with "q2 is exiting anyway" — which parallel Pass-2 breaks. The +daemon-ambiguity objection is answered by scoping it to one poisoned instance +rather than letting it justify killing the world. + +## Why Rust stays synchronous despite an async harness + +The concurrency lives on the **Deno event loop**, surfaced to Rust through the +**reader-thread demux**, not through async Rust. Rust workers are blocking +rayon+`pollster` threads; each blocks on its own slot. There is no tokio on the +Rust side, no `block_on` in the pipeline, no reactor to drive. So the +`EngineTransport` trait and `TsEngine`'s calls remain **synchronous** — the +earlier "the Rust transport is sync" conclusion survives; only the "because the +protocol is lockstep / async buys no concurrency" *justification* is retired. + +`EngineTransport` splits into a write half (shared, internally mutexed) and a +read half (owned by the reader thread). The original v1 impl was +`StdioTransport` (the child's stdin/stdout); Plan 1a.6 replaced it with +`TcpTransport` (the loopback-TCP socket) as the sole transport — +`StdioTransport` has been deleted from the code. The future +`WebSocketTransport` (WASM) shares the same trait seam. + +## Phase 1.6 — the protocol moved off stdout (loopback TCP) + +> **Landed (plan dated 2026-07-08):** +> `claude-notes/plans/2026-07-08-plan1a6-off-stdout-loopback-tcp.md` (Plan 1a.6) +> moved the protocol off stdout onto a private loopback-TCP socket. This +> section remains the canonical design rationale; the plan file carries the +> checklist, tests, and engine-impact analysis. + +Multiplexing (above) did **not** require leaving stdout; it was orthogonal. +The *reason* to leave stdout was to delete the `console.log` footgun (an +engine writing to stdout corrupted the protocol). That cleanup was **Phase +1.6**, and it has now landed. + +The channel is **loopback TCP**, not a Unix-domain socket + Windows named +pipe: + +- `std::net::TcpListener`/`TcpStream` is **blocking std, uniformly + cross-platform, no new dependency and no `#[cfg]` fork** — it actually + satisfies the "blocking std primitives" property that UDS + named-pipe + *cannot* (std has `UnixListener` but **no** Windows named-pipe support; that + needs the `interprocess` crate or raw winapi). The earlier "Unix socket / + named pipe" framing in these notes was wrong on that point. +- It **matches the existing local-IPC precedent** in the tree — Jupyter kernels + already talk over localhost TCP via `runtimelib` (`daemon.rs`). +- q2 binds `127.0.0.1:0` (ephemeral) and passes the non-secret address via + argv (`--control 127.0.0.1:`); the one-time token is written as the + first line on the child's stdin (not env/arg — see Plan 1a.6's "Token + delivery" section for why argv/env were rejected), and the child's stdin is + closed after that line. The harness dials back and presents the token as a + pre-line on the socket before any framed message. The token (not filesystem + perms) closes the "any local process can connect" gap — and the Deno + subprocess is already fully-trusted/local. + +Phase 1.6 also flipped stdout/stderr to **diagnostic-only** (drained by +`stdout_loop`/`stderr_loop` and forwarded to tracing) and deleted the +"console.log corrupts the protocol" contract. That contract no longer holds: +a stray `console.log` (or anything an engine writes to stdout) is harmless. + +## Cross-cutting consequences + +- **WASM gating.** `ts_protocol.rs` (pure serde) compiles for `wasm32`, but + `ts_process.rs` (host + demux + `TcpTransport`) uses + `std::thread`/`std::process`/`std::net` and must be entirely + `#[cfg(not(target_arch = "wasm32"))]`; `TsEngine` is registered native-only. + Skipping this breaks the `wasm-quarto-hub-client` build while + `cargo build --workspace` still passes (the `cargo xtask verify` trap). The + thread-based **demux is native-specific** — a future `WebSocketTransport` + multiplexes on the JS event loop; only the *trait* is the shared seam. +- **No new socket dependency.** The original v1 stdio transport had no + dependency; Plan 1a.6's loopback-TCP transport uses `std::net` (already + available; no `interprocess`, no winapi, no `#[cfg]` fork), so this remains + true post-landing. +- **Timeout includes same-engine queue wait.** A request's `recv_timeout(window)` + ticks while it waits in the harness's per-engine queue behind another + document's `Execute`, so under same-engine contention the Execute timeout + measures "queue-wait + execution." Accepted for v1; the fix (if needed) is a + harness `started`-ack that resets the Rust timer at actual execution start. + +## Layer ownership + +| Concern | Owner | +|---|---| +| Envelope (`id`), `Cancel`/`Cancelled`, off-stdout channel contract | plan1a-protocol (Phase 1.5) | +| `TcpTransport` (loopback listen/handshake, landed via Phase 1.6; `StdioTransport` deleted), demux (writer-mutex + reader thread + pending map), `TsEngineHost::request`, poison/relaunch, `MockTransport` | plan1a-host | +| `TsEngine` issuing requests/`Cancel`; `ExecutionContext.cancellation` (consumes `MockTransport`) | plan1a-engine | +| Non-blocking read loop, per-engine serialization queue, `AbortController` dispatch, instance lifecycle | Plan 1b (Deno harness) | diff --git a/claude-notes/designs/engine-resolution.md b/claude-notes/designs/engine-resolution.md new file mode 100644 index 000000000..7383f17f7 --- /dev/null +++ b/claude-notes/designs/engine-resolution.md @@ -0,0 +1,907 @@ +# Engine resolution & multi-engine ownership (design contract) + +**Status:** design contract — authoritative for how q2 selects engines and +divides cells among them. The TS-engine-extension plans (grand plan, +plan1a-protocol, plan1a-engine, plan1c) reference this document for the +model and contain only their plan-specific work items. +**Created:** 2026-06-22 (during the TS-engine epic rebase onto post-multi-engine `main`). +**Related code:** `crates/quarto-core/src/engine/detection.rs`, +`crates/quarto-core/src/stage/stages/engine_execution.rs`, +`crates/quarto-core/src/engine/registry.rs`. +**Related strands:** bd-5yff4 (multi-engine, merged — this extends it), +bd-iq0hp / bd-8h3sn / bd-r8n4r (Carlos's multi-engine follow-ups — see §11). + +--- + +## 1. Why this exists + +The TS-engine epic was authored single-engine in April 2026. Between then and +the June rebase, three things landed on `main` that the April plans never saw: + +- **Sequential multi-engine execution** (bd-5yff4, #238) — `engine: [a, b]` + runs N engines in order; `EngineExecutionStage` is an N-engine loop with + N+1 FileId slots; `detect_engine_sequence(meta) -> EngineSequence`. +- **Replay / capture / trace** (bd-45yw, bd-5qnj) — `ExecuteResult` is + `Serialize`/`Deserialize`; `engine_captures: Vec`; + `CaptureSpliceStage` folds captures for preview. +- **Discovery cache** (bd-c5u2g) — memoized binary-on-PATH lookup. + +Carlos's multi-engine added the **mechanism** (loop, slots, capture, replay) +but deliberately left engine **coordination** unsolved — engines run on the +whole document and grab whatever they can, which is why `[knitr, jupyter]` +breaks (knitr's reticulate runs `{python}`, jupyter sees nothing; bd-iq0hp). +This document is the **coordination layer**: how q2 decides which engine(s) +run, and which engine runs which cell. + +## 2. The two questions Q1 fused + +Single-engine Q1 only ever answered one question. Multi-engine has two: + +1. **Selection** — *which engines run* (the ordered, distinct sequence). +2. **Division** — *which engine runs which cell* (ownership). + +Q1 fused them because one engine owned the whole document (cross-language was +internal, e.g. knitr's reticulate). The claim methods answer **both**: "engine +A claims language L" simultaneously means "A belongs in the sequence (if L is +present)" and "A owns L's cells." Explicit `engine:` preempts *selection* but +**not** division — division is a new axis, and the resolution tiers (§4) answer +it for explicit and implicit sequences alike. + +## 3. The claim interface + +### 3.1 Rust trait + +`claims_language` returns a **kind-tagged claim**, not a bare priority. The +April `Option` is replaced because the multi-engine semantics need three +distinct roles that don't fit in a sign convention. + +```rust +pub enum LanguageClaim { + Primary(i32), // I execute this. (default priority 1) + Interop(i32), // extend my ownership to this iff I'm already present. (default 0) + Fallback(i32), // universal kernel; jupyter's role, now declarable by any engine. (default 0) + None, +} + +fn claims_language(&self, _language: &str, _first_class: Option<&str>) -> LanguageClaim { + LanguageClaim::None +} +``` + +- **`kind` sets the resolution tier; `priority` orders only *within* a kind.** + Kind dominates priority: `Primary(-100)` still beats `Fallback(100)` for the + same language (a committed engine outranks a safety net regardless of + numbers). Priority breaks ties among same-kind claimants; registry/`engines:` + order is the final tiebreak. +- **`Interop` is presence-gated** (§4): it only fires for an engine already in + the sequence via a positive claim — "extend if I'm already here," *not* + "claim this anywhere." This is what preserves knitr's reticulate for an + implicit `{r}`+`{python}` doc while not dragging knitr into a pure-`{python}` + doc. + +### 3.2 TS extension API (back-compatible widening) + +The Q1 publication went `boolean` → `boolean | number`. We go one more: +`boolean | number | object`, with the object exposed to extension authors. + +```ts +type LanguageClaim = + | { kind: "primary"; priority?: number } + | { kind: "interop"; priority?: number } + | { kind: "fallback"; priority?: number }; + +claimsLanguage?: (language: string, firstClass?: string) + => boolean | number | LanguageClaim | null; +``` + +Harness normalization (in `@quarto/engine-host-deno`, before crossing the +wire) — **no sign games**: + +| return | → wire | +|---|---| +| `false` / `null` / `undefined` | `None` | +| `true` | `Primary(1)` | +| `number n` | `Primary(n)` — negative = low-priority primary, **never** interop | +| `{kind:"primary", priority?}` | `Primary(priority ?? 1)` | +| `{kind:"interop", priority?}` | `Interop(priority ?? 0)` | +| `{kind:"fallback", priority?}` | `Fallback(priority ?? 0)` | + +`Interop` and `Fallback` are reachable **only** through the object. Legacy +engines could not have meant them (the concepts didn't exist), so a bare +`number` is always a `Primary`. This is the first deliberate Q1-API change in +the epic, justified by the multi-engine semantic shift; everything else stays +Q1-compatible. + +### 3.3 Static claims (zero-load resolution) + +Claiming can be declared **statically** in `_extension.yml`, so resolution +loads **no** TS engine — an engine is spawned only to *execute*, once it has +won ownership: + +```yaml +contributes: + engines: + - path: julia-engine.js + name: julia # complete static name (registration + YAML lookup, zero load) + claims: + julia: { kind: primary, priority: 1 } + # reticulate-style: r: { kind: primary }, python: { kind: interop } + # first_class-conditional: python: { whenClass: marimo, kind: primary } + # universal fallback: fallback: { priority: 0 } + file-extensions: [".jl"] # valid_extensions — complete static (the pre-filter) + # NOTE: julia does NOT declare `claims-files` — its `claimsFile` inspects + # file content (isPercentScript / `# %%`), so it loads to decide. `.jl` in + # `file-extensions` is the pre-filter; the precise file-claim is dynamic. +``` + +A static declaration is a **complete** replacement for its dynamic method +exactly when the engine's logic is a pure function of statically-known inputs; +otherwise it is a **superset pre-filter** that still loads to get the precise +answer: + +| dynamic method | static form | complete when… | falls back to load when… | +|---|---|---|---| +| `name()` / registration | `name:` | declared | omitted (lazy alias map) | +| `valid_extensions()` | `file-extensions:` | always (it *is* the list) | — | +| `claims_file()` | `claims-files:` (extension + optional `content-pattern`) | extension-only claims **and** regex-expressible content sniffs (Plan 7a) | only a *non*-regex-expressible sniff — **empty across every known Q1 engine** | +| `claims_language()` | `claims:` (kind/priority/`whenClass`) | language **and** `first_class` logic (both finite/known) | only genuine runtime/global-state logic | + +> **Restructure decided 2026-07-07 (Gordon) — supersedes the 2026-07-02 rename.** +> The earlier plan to rename `claims-files:` → `claims-extensions:` is +> **withdrawn.** A full census of every Q1 engine `claimsFile` (knitr, jupyter, +> markdown, julia) showed each is `extension-gate → read-file → **one regex**` — +> the knitr `spin`→Rscript work is the *conversion*, run **after** the claim, not +> part of it. So `claims-files` is a genuine **file-claim** surface (extension + +> an optional content pattern), *not* a bare extension set, and the name is +> correct. A content sniff is **data** (a regex), not a must-load operation, so it +> is **statically declarable** — overturning the old "one genuine must-load case." +> **Restructure:** `claims-files` entries become typed `{extension, +> content-pattern?}` (bare-string shorthand `- .echo` still accepted). The +> **extension-only** form lands in **plan 1c.2 P4**; the **`content-pattern`** +> field + native Pass-1/claim-stage evaluation lands in +> **[Plan 7a](../plans/2026-07-07-plan7a-static-content-pattern-claims.md)**. +> Normalization: YAML accepts dotted or undotted; parse stores canonical +> **undotted lowercase** for `file-extensions` and each `claims-files` entry's +> `extension`; the **JS/wire contract stays dotted** (Q1 `extname()`) — re-dot at +> the two Rust→TS seams (`ToEngine::ClaimsFile` construction and the +> synthetic-file load validation). `content-pattern` is evaluated **natively in +> Rust** and never crosses the wire. + +**`first_class` is statically expressible — it is *not* a must-load case.** +`claims_language(language, first_class)` is a pure function of its two +arguments, so a `claims:` entry may carry `whenClass: `: the claim then +applies **only** when the cell's first class equals `` (absent +`whenClass` = any/no first class). A marimo engine therefore declares +`python: { whenClass: marimo, kind: primary }` and is **fully static** — +`{python .marimo}` → `Primary`, plain `{python}` / `{python .other}` → no +claim. **Content-inspecting `claims_file` is *also* statically declarable** +(corrected 2026-07-07): Julia's `isPercentScript` reading the file's bytes for +`# %%` is a **pure regex over those bytes**, so it is expressed as a +`content-pattern` on a `claims-files` entry and evaluated natively — no load +(Plan 7a). `file-extensions` remains the can-handle pre-filter; the pattern is +the definitive claim. The genuine dynamic residue — a sniff no regex can +express — is **empty across every known Q1 engine**; the dynamic `claims_file` +method survives only as a fallback for that hypothetical case. Everything — +language, `first_class`, kind/priority, fallback, **and content sniffs** — is +statically declarable. + +**Vec-per-language claims (4c0).** A `claims:` entry's value is a **list** of +claim objects — `Vec` in Rust, a YAML sequence or a +single scalar/bool/int/object (back-compat 1-element-Vec shorthand) on the +wire: + +```yaml +claims: + sql: + - { whenClass: marimo, kind: primary, priority: 2 } # {sql .marimo} self-activates + - { kind: interop } # bare {sql} rides along +``` + +This is what lets one language key carry **both** a `whenClass`-conditioned +primary claim and an unconditional interop claim — marimo's bare-`{sql}` +feature: `{sql .marimo}` is `Primary` (tagged, self-activating), while a bare +`{sql}` is `Interop(0)` (rides along only when marimo already owns another +language via a positive claim). A plain scalar/object value (`echo: true`, +`fallback: { priority: 0 }`) still parses to a 1-element Vec — the pre-4c0 +single-claim shape is unaffected. + +**Combine rule.** `lookup_static_claim` maps every element of a language's Vec +through `static_claim_to_language_claim` (which returns `LanguageClaim::None` +on a `whenClass` mismatch), drops the `None`s, and reduces the survivors with +a dedicated per-Vec comparator (`ClaimKind::combine_rank` in +`extension/types.rs`): kind dominates priority — Primary > Interop > Fallback, +`priority` breaking ties within a kind — the same *shape* of ordering as the +cross-engine "kind dominates priority" rule below, but a separate, explicit +implementation scoped to one language key's Vec, not a reuse of the +cross-engine tiering. The universal `fallback:` key is combined the identical +way (`ts_engine.rs`'s two fallback call sites both route through the Vec +combiner), so a fallback engine can itself carry more than one claim. + +**Top-level list shorthand (disambiguated from 4c0's per-language sequence).** +A `claims:` value may also be a **top-level list of language names** — +`claims: [python, r]` — rather than a map keyed by language. This is sugar +for "each named language gets a bare `Primary(default)` claim," equivalent to +`claims: { python: primary, r: primary }`. Do not confuse this with 4c0's +**per-language** claim-object sequence above (`claims: { sql: [{whenClass: +marimo, kind: primary}, {kind: interop}] }`): the top-level list is a +shorthand over the *set of languages claimed*, one default `Primary` per +entry; the per-language Vec is a sequence of *claim objects for one +language*, letting a single language carry more than one conditioned claim. +The two shapes are told apart by nesting, not by a discriminator key: a +`claims:` value that is a list *at the top level* (entries are bare language +names) is this shorthand; a `claims:` value that is a map whose *entry* value +is a list (entries are claim objects) is the 4c0 form. + +A statically-declared claim used for resolution is validated against the +dynamic method **only if/when the engine loads to execute** (mismatch → hard +error, like the `name` check). Static claims are **authoritative for +resolution**; authors who declare them own their accuracy. `Fallback` cannot +be a finite language list, so a universal-fallback engine declares +`fallback:` rather than a per-language entry. Full-static resolution requires a +declared `name` (zero-load needs the name to place the engine in the +sequence). Whether resolution loads nothing is decided **per document, not +project-wide** — see "The needs-no-load predicate" below; that lift is the +payoff of static claims (§7). + +**Claim tables from metadata (`engine:`/`engines:` sugar).** Metadata may +also supply an engine's complete claim table — same schema as +`_extension.yml`'s `claims:` — via a per-entry `claims:` key on either the +`engine:` or `engines:` metadata key (`engine-and-engines-keys.md` §2/§3 owns +the user-facing grammar; this is the resolution-side contract for what such a +table does). A table is a **whole-table replacement**, winner takes all: it +does not merge with the engine's `_extension.yml` claims or its dynamic +`claims_language` — it *replaces* them entirely for that engine. **Source +precedence** (highest wins, no merging across sources): a document's own +`engine:`-entry table > project `engines:` table > `_extension.yml` static +claims > the dynamic `claims_language` method. **A table makes its engine +load-free** — resolution answers every claim consultation for that engine +from the table, never loading it, which is what lets a legacy claims-less +extension become Pass-1-resolvable without an edit to the extension itself. +**An empty table (`claims: []`) is a full mask** — the engine claims nothing; +the idiom `engines: [{jupyter: {claims: []}}]` disables jupyter's universal +fallback project-wide. **Built-in engines are maskable but never validated**: +a table may replace a built-in's (knitr's, jupyter's) claims outright, and +because a masked engine never loads to compare, the author-validation +paragraph above has no comparison moment to run while a table shadows it — +reconciling the two later, as a divergence advisory, is future polish, not +required now. **Forcing ownership via a table is priority-based and +best-effort**: a table can outrank other candidates by declaring a +high-priority claim, but it competes in the same kind/priority tiering as +everyone else (§4) — against an unoverridden dynamic engine nothing can be +guaranteed at Pass-1 (the doc falls through and Pass-2 loads the contender). + +**The two-key model, restated for resolution.** `engine:` *names the engines +at play*: an explicit sequence, presence for every listed engine, execution +order, and — per §4.3 — it turns T4 (implicit fallback) off. A per-entry +`claims:` inside `engine:` is sugar for a document-level table on an +already-named engine; the reserved `claims` key is stripped from the rest of +the entry's config before it reaches the engine at execute time. `engines:` +*configures without naming*: a Q1-syntax-compatible project-level array whose +single-key-map entries' `claims:` values supply tables for engines already in +the registry, without ever touching T4 gating, engine presence, or sequence. +**q2-divergence:** both keys are read from *merged* metadata (project + +document layers), not only from the one layer Q1 read each from (frontmatter +for `engine:`, project config for `engines:`) — `engine-and-engines-keys.md` +§4 has the full Q1 comparison. + +**The needs-no-load predicate (P1–P4).** Let `languages` be the doc's +computational languages per §4.1 (including `generated-languages` when the +scan is non-empty). A doc resolves load-free at Pass-1 iff any of: + +- **P1** — the file is claimed (`claimed = Some(engine)`): §8's short-circuit + consults no claims at all. +- **P2** — the language scan is empty (markdown passthrough). +- **P3** — an explicit `engine: markdown` opt-out. +- **P4** — **every claim consultation the resolution needs returns a static + answer.** For each `(candidate engine, language)` pair, either a metadata + claim table answers it, or the engine answers without loading (a built-in, + or a `TsEngine` with static `claims:`). P4 is not a separate precondition + check: resolution attempts every consultation over the no-load claim path, + and a single "would need to load" answer aborts the lift, falling through + to Pass-2 exactly as today. + +Otherwise the doc falls through. "Load-free" is *computed*, not flagged — it +is exactly "the no-load claim path never answered would-load for this doc." + +**Tier-dominance shortcut, considered and rejected as unsound.** An earlier +design considered a shortcut: "a static `Primary` beats anything a +non-static engine could declare," letting resolution skip consulting an +unloaded engine once a static `Primary` claim is found elsewhere. This is +**unsound** — an unloaded engine could declare `Primary(999)` (priority +orders *within* a kind, §3.1), so without a load or a claim table a +claims-less engine's claims cannot be bounded, and the shortcut could pick +the wrong owner. The predicate above does not shortcut: a claims-less, +untabled engine registered in the project makes every doc with an uncovered +computational language fall through — correct, and exactly what the +lifted/fell-through counters and the index-pass warning (§12) surface. + +**The no-load claim method.** The mechanism the predicate rests on is a +method on the `ExecutionEngine` trait, parallel to `claims_language` but +answerable without loading: + +```rust +fn try_claims_language(&self, language: &str, first_class: Option<&str>) + -> Option; +``` + +`None` means "would have to load to answer" — the uniform per-engine signal +P4 treats as an abort. `Some(claim)` — including `Some(LanguageClaim::None)` +for "definitely doesn't claim this" — is a static answer. Built-ins override +it directly (their `claims_language` *is* static); a `TsEngine` with a static +`claims:` table (above) answers from that table without spawning the JS +runtime; a claims-less, untabled `TsEngine` returns `None` uniformly. Before +this method existed, the contract described "static claims answer without +loading" only in prose; this is the surface that makes it a checkable fact +rather than a convention. + +## 4. Resolution algorithm + +`resolve_engines` is a **pure function** (§9) of merged metadata, the parsed +AST, the registry, and the file-claim engine. **If `claimed = Some(engine)` +the tiers below are skipped entirely — a claimed file resolves to that single +engine (§8, Q1-faithful).** The tiers run only for the implicit/explicit +`.qmd` path (`claimed = None`): + +``` +languages = computational languages of the doc // §4.1 +present = explicitly-listed engines + +T1 Primary: per language, highest-priority Primary wins → owns it; add to `present`. +T2 explicit Fallback: language with no Primary → an explicitly-listed engine that returned + Fallback for it owns it (highest Fallback priority, then order). +T3 Interop: still-unclaimed → highest-priority Interop among `present` engines. +T4 implicit Fallback: still-unclaimed computational → an engine that returned Fallback for it + (highest priority, then order). GATED: implicit sequences only (§4.3). + +sequence = distinct owners, in registry/`engines:` order. +ownership = language -> owning engine name. // per-language (§4.2) +``` + +### 4.1 What counts as a "computational language" + +Not a list — **structural**, from the parsed AST: the language of every +**executable** cell (a braced `{lang}` fence; pampa preserves the braces in +the class name), **minus** `HANDLED_LANGUAGES` (`ojs`/`mermaid`/`dot` — cell +handlers, not engines) and minus raw `{=fmt}` blocks. No allowlist, no kernel +registry. An empty set → no engine → markdown passthrough. + +**`generated-languages` widens the scan — but only when it is already +non-empty.** `languages = scan(ast) ∪ generated-languages`: a flat top-level +list of language names in `meta.generated-languages`, declared once, +consumer-only (no per-engine attribution), ordering unaffected (order is +controlled by the explicit `engine:` list, never by this key). Generated +entries are consulted with `first_class = None` — a generated language has no +cell of its own to carry a first class. **The union is consulted only when +`scan(ast)` is non-empty**: a cell-less doc with `generated-languages` set is +a no-op and stays markdown passthrough, matching the doc having nothing to +generate into. See §6.1 for why this is the *static* escape from the +handoff-loss limitation, not a relaxation of it. + +### 4.2 Per-language ownership; `first_class` drives *selection* + +Ownership is keyed by **language**, not `(language, first_class)`. `first_class` +sharpens the *claim* (a marimo engine returns `Primary` for `{python .marimo}`, +`None` for plain `{python}`), so it influences **which engine is selected**, but +a language has **one owner**. This is because enforcement (§5) is per-language; +per-cell routing would require per-cell enforcement (a future possibility, §10). +A doc that mixes `{python}` and `{python .marimo}` wanting *different* engines is +a v1 limitation — the same limitation Q1 had (its single winner ran all cells of +a language). + +### 4.3 jupyter is `Fallback(0)`; T4 is implicit-only + +jupyter declares `Fallback(0)` for everything it is asked about (asked only +about the doc's actual executable, non-handler languages — it never +enumerates). Any engine can declare `Fallback` per-language; jupyter is just +the default universal one. **T4 fires only for implicit sequences** — an +explicitly-listed `engine:` never silently grows a non-listed fallback engine +(matching Q1's P4 gating: fallback only when nothing explicit/claimed +selected). An explicit `[knitr]` with a `{julia}` cell knitr can't run leaves +julia to the listed engine (best-effort), it does **not** add jupyter. + +### 4.4 Worked cases + +| Doc / sequence | non-R lang → owner | sequence | why | +|---|---|---|---| +| implicit `{r}`+`{python}` | python → knitr (reticulate) | `[knitr]` | T1 knitr→r; T3 knitr `Interop` python (present) | +| implicit `{r}`+`{sql}` | sql → knitr (`eng_sql`) | `[knitr]` | T1 knitr→r; T3 knitr `Interop` sql (present) | +| explicit `[knitr, jupyter]`, `{r}`+`{python}` | python → jupyter | `[knitr, jupyter]` | T1 knitr→r; **T2** jupyter (explicit `Fallback`) preempts knitr's `Interop`; knitr cedes python | +| explicit `[knitr, jupyter]`, `{r}`+`{sql}` | sql → jupyter ⚠️ | `[knitr, jupyter]` | same: **T2** explicit `Fallback` preempts `Interop`. But jupyter has no SQL kernel → **§10 loud failure** at execute (named, not silent) | +| implicit `{python}` only | python → jupyter | `[jupyter]` | T1–T3 ∅ (knitr not present); **T4** jupyter | +| implicit `{julia}` + Julia ext | julia → julia-ext | `[julia]` | T1 julia-ext `Primary(1)` | +| implicit `{julia}`, no ext | julia → jupyter | `[jupyter]` | T4 fallback — Q1 parity | +| implicit `{python}` + fallback ext `Fallback(5)` | python → ext | `[ext]` | T4 by priority (`5 > 0`), not by registration order | +| weak engine `Primary(-100)` vs jupyter `Fallback(0)` | weak engine | — | kind dominates priority | + +**Consequence — explicit-`Fallback` preempts `Interop` for *every* non-primary +language, including ones the fallback engine can't execute.** The rule that +makes `python → jupyter` desirable in `[knitr, jupyter]` (T2 > T3) applies +uniformly: `sql`/`bash`/`sh` go to jupyter too, even though jupyter has no +kernel for them while knitr's `eng_sql`/`eng_bash` do. This is **intentional** +(the user explicitly listed jupyter as the non-R owner), **not** silent +breakage — the resolver is availability-blind by design (§4.3), and the +`sql`-with-no-kernel case is caught **loudly at execute** by §10's "owner +cannot execute an owned language" rule. The valuable common case (a +*knitr-only* `{r}`+`{sql}` doc → `sql → knitr`) is unaffected: `Interop` wins +when no explicit fallback is present. Authors who want `sql` to stay on knitr +simply omit jupyter from the explicit sequence. + +## 5. Ownership enforcement (`handled_languages`) + +**Single source of truth = the ownership map.** The sequence and every engine's +leave-alone set are *projections* of it; nothing is re-derived independently +(or they could drift). For engine *k*: + +``` +handled_languages(k) = HANDLED_LANGUAGES ∪ { lang : ownership[lang] != k } +``` + +threaded into execution via a new `ExecutionContext` field (and +`TsExecuteOptions.handled_languages` for TS engines). + +**Positive projection (`owned_languages`, Plan 4d).** The same ownership map has +a symmetric projection — the languages an engine *does* own: + +``` +owned_languages(k) = { lang : ownership[lang] == k } +``` + +carried beside the leave-alone set on the same `ExecutionContext` field pair (and +`TsExecuteOptions.owned_languages` for TS engines). It is **informational, not +enforcement**: `handled_languages` stays the execute-time gate (cede / re-emit), +while `owned_languages` makes the resolution decision legible so an engine can +select the cells it was chosen for directly (`owned_languages` membership) rather +than inferring them as the complement of the leave-alone set — the latter is +ambiguous because "not handled" conflates *owned by me* with *owned by nobody*. +The two projections are disjoint over present languages; a language owned by +**nobody** — present-but-unclaimed, or injected at execute time (§6.1) — is in +neither set, which is exactly how such a cell reads as "not mine, pass through +unexecuted." `owned_languages` does **not** re-resolve or execute injected +languages; §6.1's ratified pass-through is unchanged. + +- **knitr** already enforces: `execute.R` does `knit_engines$set(lang = )` for each `handled_languages` entry, which **replaces** knitr's + default engine for that language — so adding `python` to knitr's set makes it + re-emit `{python}` cells unexecuted (suppressing reticulate), and *not* adding + it leaves reticulate intact. Reticulate-vs-handoff falls out of the same list. + No new knitr mechanism — only the *population* changes from the static + `[ojs, mermaid, dot]` constant to the per-engine ownership projection. +- **jupyter** has *no* `handled_languages` consumption today and runs every cell + it's given. It needs **execution-time** enforcement (skip/re-emit cells whose + language is in its leave-alone set) **when it is non-terminal** in a sequence + (e.g. explicit `[jupyter, knitr]`). As the terminal/fallback engine it owns + the remainder and never needs to cede. Its *claiming* is already correct; + this is purely an execute-time gate. +- **TS engines** honor `handled_languages` by contract (pass through what they + don't own), mirroring knitr. + +These are three clocks, kept separate: **`claims_language` runs at +resolution** (recording/normal execution); **`handled_languages` enforces at +execute**; **replay touches neither** (§6). + +## 6. Multi-engine execution & replay + +### 6.1 Execution is sequential threading + +Engines run **in order, each consuming the previous engine's output** — *not* +independently-then-stitched (`engine_execution.rs` loop): + +``` +ast = original AST +for engine in sequence: + qmd = serialize_ast_to_qmd(ast) // current AST, incl. prior engines' output + result = engine.execute(qmd, ctx) // ctx.handled_languages = leave-alone set for this engine + ast = reconcile(ast, parse(result.markdown)) +``` + +This is what enables handoff (engine A re-emits a `{python}` cell that engine B +executes) — and **why non-terminal engines must enforce `handled_languages`**: +without ceding, an earlier engine executes cells before they reach their owner. +Because knitr's cede re-emits at **top level**, ceded cells land where the next +engine (and the preview splicer) can reach them. + +**Resolution-driven handoff loss — RATIFIED 2026-07-01 (Gordon; T9).** The +engine sequence is derived **once, from the original parsed AST** — an engine +is in the sequence only if it owns ≥1 language actually present in the source. +This is intended behavior, not a bug. Documented here so it is not re-litigated +(cross-refs: §4.3 fallback gating, §8 file-claim single-engine, §11/bd-r8n4r +nested-handoff splice). + +*Scenarios this rules OUT (documented, accepted):* +1. **Injected-cell handoff to an engine absent from the sequence.** Engine A, + *at execution time*, emits a cell in language L whose only would-be owner is + engine B — but B was excluded because the *original* source had no L cells. + The sequence is fixed pre-execution, so B never runs and the injected L cell + is not executed by B (it passes through as display code — §8/§10 + non-enforcement). +2. **An explicitly-listed engine that owns nothing originally is dropped.** + `engines: [knitr, customX]` where customX's language never appears in the + source: customX contributes nothing to the sequence and cannot receive + runtime-injected cells in its language. The fallback net does **not** save + this: per §4.3, T4 only adds jupyter for *implicit* sequences, so an + explicit `[knitr]` with a runtime-injected `{python}` does not auto-add + jupyter either. + +*Scenarios that still WORK (unaffected):* handoff between engines that both own +something in the original AST (knitr re-emits `{python}`, jupyter executes it, +*because the doc already had `{python}` cells*); knitr↔reticulate interop; +jupyter-as-`Fallback(0)` catching the remainder in implicit docs. + +*Why acceptable / why not "just fix it":* resolving (1)/(2) would require +**runtime sequence growth** — re-resolving mid-execution as new cells appear — +which the resolution-driven + replay model deliberately avoids (§6.2: replay +drives from recorded captures, not re-resolution; mid-execute re-resolution +would break the determinism guard and the eventual freeze cache-key). Tracked +as a live-preview limitation (bd-r8n4r); the valuable common handoffs are all +in the "still works" set. + +**The static escape: `generated-languages` (§4.1).** Scenario 1 above +(injected-cell handoff to an engine absent from the sequence) has a +**declared, static** escape: `meta.generated-languages` widens the +pre-execution language scan to include a language the doc promises to +generate but does not yet contain a literal cell for, so the target engine's +ownership — and its place in the sequence — is decided from metadata before +execution starts, not by re-scanning the AST after it runs. This does **not** +relax the T9 ratification above: the sequence is still derived **once**, from +information known before execution (the scan *plus* the declared list), never +by re-resolving mid-execution as cells appear. An author who does not declare +the generated language remains subject to scenario 1 exactly as ratified. + +### 6.2 Replay drives from recorded captures, not re-resolution + +On `main`, replay re-runs `detect_engine_sequence(meta)` and looks up +`ReplayEngine`s by name — fine when execution was explicit-only (meta fully +determined the sequence). But **implicit docs now resolve via claims, and +`ReplayEngine`s carry no claims**, so re-resolving during replay would produce +the wrong sequence. Therefore: + +- **Replay iterates the recorded `engine_captures` in order** (they carry + engine names + order); it does **not** call the resolver. It still + serialize→reconcile-threads and validates `input_qmd` byte-equality as the + determinism guard. +- **Ownership / `handled_languages` is a recording-time concern, baked into the + recorded results** (knitr's recorded output already has the ceded `{python}` + verbatim; jupyter's already executed). Replay is pure playback. +- This decouples replay from the resolution machinery entirely and likely lets + `ReplayEngine` / `with_replay_many` be **replaced** by a capture-driven + replay path — which also sidesteps injecting engines into the now-immutable + `Arc` (§ plan1a-engine / plan1c). +- **Freeze caveat:** "replay as freeze" must key invalidation on the **resolved + engine set**, not only the input hash — installing an extension changes + resolution while `input_qmd` still byte-matches. (Freeze is future; recorded + here so its design accounts for it.) + +Preview (`CaptureSpliceStage`) is the same story: resolution runs at *record* +time; the browser folds captures. + +## 7. Pass placement + +**Resolution is attempted per-document at Pass-1; the stamp is +complete-or-absent, never partial.** `resolve_engines` is a pure function of +`(meta, ast, registry, claimed)` (§9), so `DocumentProfileStage` can call its +Pass-1 counterpart (`resolve_engines_pass1`, §9) directly at the checkpoint: +when the doc satisfies the needs-no-load predicate (§3.3, P1–P4) the call +runs entirely over the no-load claim path and the profile is stamped with a +**complete** `ProfileEngineResolution` (§9); the moment any consultation +would need to load an engine, the attempt aborts and the profile field is +`engine_resolution: None`. There is no partial/pending representation, and +this lift never loads an engine to resolve (only file-claim conversion, +below, may load one, for an unrelated reason). A `None` stamp is not an +error: Pass-2 re-resolves the same doc from scratch via `resolve_engines`, +exactly as it did before this lift existed, so a fall-through doc's render is +unaffected — only Pass-1-only, profile-consuming features (the LSP today; +freeze/pooling later, §12) see the gap. + +The lift is per-doc *and* project-grain at once: a doc's P1–P3 status is its +own, but P4 depends on the registry (which engines are static or tabled), so +in practice a project either has every doc resolve load-free or has a shared +minority permanently fall through until its extensions or `engines:` tables +catch up — the condition the index-pass warning (§12) surfaces. + +The file-claim half (§8) *is* in Pass 1 (it must, to convert non-QMD input before +parse), but it only spawns an engine when a doc genuinely needs conversion, and +it must be inserted into **both** the full pipeline and the Pass-1 builder +(`pass1_profile_single_file_live`) — otherwise non-QMD docs get a garbage +`DocumentProfile`. File-claim conversion and engine-set resolution are +independent Pass-1 activities: a claimed file's `claimed` argument feeds +`resolve_engines_pass1` exactly as it feeds `resolve_engines` today (§8's +single-engine short-circuit is P1 of the predicate above), but the file +claim's own engine load (to run `markdown_for_file`) is not itself a +resolution load and does not count against the needs-no-load predicate. + +## 8. File-claim semantics (Q1-faithful: claimed file → single engine) + +`claims_file` → `markdown_for_file` runs pre-parse (Pass 1) and is the +converter. **A file claim resolves to that one engine, full stop** — exactly +Q1's `fileExecutionEngine`, which `return`s the first engine whose `claimsFile` +matches and never consults anything else (`engine.ts:320-325`, verified +2026-06-28). When `resolve_engines` is called with `claimed = Some(engine)` it +**short-circuits the tiers** and returns a single-engine resolution: that +engine is the whole `sequence`. No tiers, no seed, no native-language +inference. As the **sole** engine it is handed the whole converted document +(`handled_languages` = just `HANDLED_LANGUAGES`, the standard cell-handlers) +and **self-selects** — it runs the cells it recognizes and passes the rest +through. There is no ownership handoff, so **§10 case-4 does not apply** (case-4 +is a *multi-engine* behavior — see below). + +- **The `engine:` YAML inside a claimed file is ignored** — Q1's `claimsFile` + match preempts the YAML-engine reader entirely (it is only reachable for + `.md`/`.qmd`; `engine.ts:329-350`). q2 matches this: a claimed file does not + consult its own front-matter `engine:`. +- **Non-executed languages pass through, they do not fail.** The claiming + engine executes the cells it recognizes (its kernel/native language) and + **passes the rest through unexecuted** — emitted as display code — exactly + Q1, whose `quartoMdToJupyter` converts a non-kernel `{bash}` cell to a + markdown cell that the kernel never runs (`core/jupyter/jupyter.ts:321-324`, + verified 2026-06-28). This is **not** a §10 case-4 loud failure: **case-4 is + gated on `|sequence| > 1`** (multi-engine, `engine: [knitr, jupyter]` + + `{sql}` → jupyter, where the user *chose* the owner). A single-engine sequence + — a claimed file *or* a `.qmd` resolving to one engine — runs what it can and + passes the rest through, exactly Q1. *(The gating landed as P2-13/P2-13a: + `engine/jupyter/text_execute.rs` `partition_cells` takes `multi_engine` and + errors only when it is true; single-engine passes through, with binding + tests in both directions.)* + +**Why this replaced the "seed `Primary` + resolve" design (reverted +2026-06-28).** The earlier draft tried to make a claimed file participate in +multi-engine resolution by seeding the claiming engine as a synthetic `Primary` +and re-running the tiers — to let a stray `{bash}` cell reach a secondary +engine. That required the resolver to know the file's *native* language (which +it can't infer from an engine name alone), and the landed `resolution.rs` +silently never implemented the seed (it only marked the seed "present"), +leaving a real theft hole (a generic `Primary(1)` python extension would steal +a jupyter-`Fallback(0)` `.ipynb`'s cells). The single-engine rule is simpler, +removes that hole by construction, needs no native-language plumbing, and is +what Q1 actually does. Multi-engine remains a **`.qmd`-authoring** feature +(`engine: [a, b]`); converted non-`.qmd` files are single-engine. + +## 9. Resolution as an artifact + +```rust +// crates/quarto-core/src/engine/resolution.rs +pub struct EngineResolution { + pub sequence: Vec, // ordered, distinct owners + pub ownership: LinkedHashMap, // language -> owning engine name, insertion order + pub notes: Vec, // warnings — advisory, resolver stays infallible +} +impl EngineResolution { + pub fn handled_languages_for(&self, engine: &str) -> Vec; // §5 (leave-alone) + pub fn owned_languages_for(&self, engine: &str) -> Vec; // §5 (positive; Plan 4d) +} +pub fn resolve_engines( + meta: &ConfigValue, ast: &Pandoc, registry: &EngineRegistry, claimed: Option<&str>, +) -> EngineResolution; // tiers, presence-gating, fallback ordering all live here + +/// Advisory warnings the resolver records instead of failing — the resolver +/// stays pure and infallible (§3's per-doc-layer "warn-and-skip" unknown-name +/// policy), so anything a caller wants to surface travels as returned data, +/// not an error. +#[derive(Debug, Clone, PartialEq)] +pub enum ResolutionNote { + UnknownOverrideEngine { engine: String }, + ConflictingDuplicateEngineConfig { engine: String }, +} +``` + +`ownership` is a `LinkedHashMap` (`resolution.rs:286`, insertion-ordered), not +a `HashMap` — `handled_languages_for`'s deterministic output and +`ProfileEngineResolution`'s `Vec<(String, String)>` conversion (§9.1 below) +both rely on iterating it in insertion order. `notes` lives on +`EngineResolution`, **not** on the reduced profile type — warnings are Pass-2 +`StageContext` data (drained into `ctx.diagnostics`), and the profile stays a +pure names-only snapshot. + +`EngineExecutionStage::run` calls `resolve_engines` once and stashes +`EngineResolution` on `StageContext` (mirroring `project_index` in +`run_pipeline`). The loop reads `ownership` for each engine's +`handled_languages`; the trace records `sequence`; `notes` drains into +`ctx.diagnostics` as warnings. Benefits: the tier/presence/fallback logic is +**unit-testable in isolation** with mock claim tables (no subprocess); the +trace can emit a "resolved engines" entry. It is a function + `StageContext` +artifact, **not** a pipeline stage (it transforms no `PipelineData`). + +### 9.1 The Pass-1 lift: `resolve_engines_pass1` and `ProfileEngineResolution` + +```rust +// crates/quarto-core/src/engine/resolution.rs +pub fn resolve_engines_pass1( + meta: &ConfigValue, ast: &Pandoc, + registry: &EngineRegistry, claimed: Option<&str>, +) -> Option; // Some = load-free stamp, None = fall through + +// crates/quarto-core/src/document_profile.rs +pub struct ProfileEngineResolution { + pub sequence: Vec, // ordered distinct owners + pub ownership: Vec<(String, String)>, // language -> engine, insertion order +} +``` + +`resolve_engines_pass1` shares `resolve_engines`'s tiers/presence/fallback +core (§4), parameterized to run only over the no-load claim path +(`try_claims_language`, §3.3) — same inputs, same algorithm, different claim +source. It returns `Some(EngineResolution)` when the needs-no-load predicate +(§3.3, P1–P4) holds for the doc — a **complete** resolution, identical in +content to what `resolve_engines` would have produced — and `None` the +moment any consultation would need to load an engine; there is no partial +result (§7). `DocumentProfileStage` calls it at the profile checkpoint and, +on `Some`, projects the full `EngineResolution` down to the reduced +`ProfileEngineResolution` — names only, no `ConfigValue` blobs — for the +`engine_resolution: Option` profile field +(`document-profile-contract.md`): `sequence` becomes engine names, +`ownership` becomes `Vec<(String, String)>` in the `LinkedHashMap`'s +insertion order. A `None` stamp means the field is `None`, not an error — +Pass-2 re-resolves via `resolve_engines` regardless. `EngineResolution::notes` +does not travel to the profile: a Pass-1 stamp is complete precisely because +the no-load path answered every consultation, so it produces no warnings to +carry; `notes` remains a Pass-2 `StageContext`/diagnostics concern. + +## 10. Failure model (Q1 parity, with one deliberate q2 divergence) + +Resolution is **availability-blind *and* capability-blind**: `resolve_engines` +picks owners purely by claim — a pure function of `(meta, ast, registry, +claimed)` (§9) — so **which engine owns which language is deterministic and +environment-independent.** This is load-bearing, not incidental: it is exactly +what lets resolution lift to **Pass-1** and stamp on `DocumentProfile`. An +eager "can this engine actually run language L?" probe would couple the chosen +sequence to whether a kernel happens to be installed on *this* machine, making +engine selection non-deterministic and un-liftable. So environment checks run +**after** resolution and **never change the chosen sequence** (never silently +re-route to a fallback — Q1 parity). Two distinct kinds, at two distinct times: + +- **Availability** — is the owner's binary on PATH (`is_available()`)? Cheap; + checked **after resolution, before execute** (cases 1–2). +- **Capability** — can the chosen owner actually *run* language L? An + **execute-time** failure (cases 3–4): q2 starts kernels **lazily** inside + `execute()`, and — more fundamentally — keeping capability out of resolution + is what buys deterministic selection. + +**Capability is judged from declarations, never from execution results +(ratified 2026-07-02, Gordon).** Every check in this section fires off +*declared* data — `handled_languages`, static claims, `is_available()` — at +resolution/partition time. q2 does **not** verify post-hoc that an engine +actually executed the cells of a language it owns: an engine that runs and +leaves cells unexecuted produces display code blocks, not an error (§8 +pass-through). The capture data would support such a post-execution check, but +it is explicitly out of scope — do not add one under the banner of "enforcing" +case-4. **Deliberate divergence from Q1's eager + kernel check (decided 2026-06-24):** a `[knitr, jupyter]`+`{sql}` doc runs + knitr's `{r}` cells, *then* halts loudly at the `{sql}` cell — partial work + before the halt, **traded for deterministic, Pass-1-liftable engine + selection.** We do not add an eager capability probe. + +- **Graceful (no error) — language-fallback axis:** a computational language no + one claims → jupyter fallback; no executable cells → markdown. q2 already + does this via the tiers. +- **Loud (halt render, actionable, Q1 message-style):** + 1. **No engine claims the file extension** — a non-QMD file whose extension + is in no engine's `valid_extensions` → `"Can't determine execution engine + for "` (Q1 `engine.ts:317→366`). `.qmd`/`.md` always resolve. + *(Resolution-time.)* + 2. **A resolved owning engine's runtime is missing** (`is_available()` false) + — name the engine + what is missing + how to install (Q1: *"Unable to + locate an installed version of R / Python 3…"*). No degradation. + *(Availability, pre-execute.)* + 3. **Jupyter kernel not found** — name the missing kernel, list available, + suggest `quarto check jupyter` (Q1 parity). *(Capability, execute-time — + surfaces when the lazy kernel start fails.)* + 4. **A resolved owner cannot execute a language it owns** — the general case + of (3), made explicit because the four-tier model can hand an engine a + language it has no handler/kernel for (e.g. `[knitr, jupyter]` routes + `{sql}` to jupyter via explicit-`Fallback`, but jupyter has no SQL kernel; + §4.4 — note jupyter is **single-kernel-per-doc**, so it can only faithfully + own its one kernel's language). The owner MUST fail loudly **at execute** + (by design — see the capability note above) — a clear `ExecutionError` + naming the engine **and** the language ("engine `jupyter` has no kernel + for `sql`") — and MUST NOT silently skip the cell or emit it unexecuted. + This is part of bringing the built-in engines up to the + TsEngine/Quarto-API execution contract (see plan1a-engine's jupyter + enforcement item). Silent no-op here would re-introduce exactly the kind + of "cell quietly didn't run" failure the ownership model exists to prevent. + **Not a forcible abort:** this error is a clean refusal (the engine never + started computing), so it does **not** poison the instance — it behaves + like `ExecutionFailed`, not `Cancelled`/`Timeout` (plan1a-engine poison + policy). + **Scope: case 4 is gated on `|sequence| > 1` (multi-engine only).** It + fires only when the tiers / `engine:` list routed a language to an owner + that is one of *several* engines — the case where silent-skip betrays the + user's explicit composition. A **single-engine sequence** — a claimed file + (§8) *or* a `.qmd` resolving to one engine — is handed the whole document + and **self-selects**: it runs what it can and **passes the rest through + unexecuted** (Q1's `quartoMdToJupyter` makes a non-kernel `{bash}` cell a + display-only markdown cell — verified 2026-06-28), never a loud failure. + This is full Q1 parity (Q1 is always single-engine and always passes + through); case 4 is the deliberate q2 *multi-engine* divergence. + **Landed-code consequence:** `engine/jupyter/text_execute.rs`'s + `partition_cells` currently raises `NoHandlerForLanguage` for *any* + owned-but-unrunnable cell regardless of sequence length — it must gate the + loud branch on `|sequence| > 1`, ceding (passing through) in the + single-engine case. The TS-engine "must error, never silently pass + through" obligation likewise applies **only** to a TS engine that is a + non-sole participant in a multi-engine sequence. + **A nonsensical claim table follows this same gating.** A metadata claim + table (§3.3) that names an owner unable to actually execute a language it + claimed — e.g. `engines: [{jupyter: {claims: {sql: primary}}}]` — is not + a new failure path: it fails loudly at execute in a multi-engine + sequence, exactly like any other case-4 owner, and passes through + silently as display code in a single-engine sequence. Tables are a claim + *source* (§3.3), not a bypass of this gate. +- **Multi-engine:** any unavailable **owner** in a sequence → fail the whole + render loudly, naming the engine/language. Q1 never degrades; neither do we. + +## 11. Relationship to Carlos's multi-engine follow-ups + +- **bd-iq0hp** (multi-engine preview E2E) — **unblocked.** Its blocker is + "knitr/jupyter don't compose cleanly (knitr claims python)." The ownership + model is the principled fix (`[knitr, jupyter]` → r→knitr, python→jupyter via + T2), and the epic also ships composable engines (echo, Julia). The browser + test still needs writing; the composition blocker is gone. +- **bd-8h3sn** (cross-engine source attribution) — **shared fix, folded in.** TS + engines inherit the same engine-2+ gap (their `source_map` references + intermediate-slot FileIds). Thread the accumulating merged context into each + engine's per-position `source_map`, designed once for built-in + TS. +- **bd-r8n4r** (nested-handoff splice) — **tangential, increased exposure.** Our + cede mechanism re-emits at top level (within `splice_cells`' reach), so the + nested-in-`Div.cell` case it tracks is not produced by ceding; but auto-split + makes handoff more common, so it stays a live preview limitation. + +## 12. Future possibilities (not in scope) + +- **Per-cell routing.** The `claims_language` *interface* already supports it; + it would change ownership granularity to per-cell and enforcement + (`handled_languages`) to a per-cell skip set + stable cell identity through + the threading round-trip. Reversible; least urgent of these three. + +**In scope, delivered by this plan (moved out of "future"):** + +- **Pass-1 resolution.** §7/§9 above: once a doc's resolution satisfies the + needs-no-load predicate (§3.3, P1–P4), `resolve_engines_pass1` runs it in + Pass 1 and stamps a `ProfileEngineResolution` on the `DocumentProfile` + (profile-version bump — `document-profile-contract.md`), making the index + engine-aware (kernel pooling, freeze planning, LSP) for every doc that + qualifies. **The index-pass warning**: when Pass-1 cannot resolve every + doc, the orchestrator prints one warning at index-pass completion naming + each engine that must load to answer language claims (with its + `_extension.yml` path) and both fixes (author-side `claims:` in + `_extension.yml`; user-side `engines:` table in `_quarto.yml`) — the + human-facing surface for the fall-through condition described above. + **Freeze-key caveat:** a doc whose `engine_resolution` field is `None` + cannot be frozen until resolution completes at Pass-2 — the profile alone + does not carry enough to key a freeze cache entry for a fall-through doc. + This forces **Option A** (load contested engines at Pass-1 to complete the + set) *for freeze specifically*, whenever freeze is built; the V1 + load-free-only stamp here is sufficient for the LSP, which already + tolerates `None`, but not sufficient on its own for a freeze cache that + must cover every doc. +- **Project-level claim overrides.** §3.3 above ("Claim tables from + metadata") and `engine-and-engines-keys.md` §3: static claims as project + config, via the `engines:` key's `claims:` entries. "Knitr does not interop + python here" is exactly `engines: [{knitr: {claims: {r: primary}}}]` — a + whole-table replacement that omits python, masking it out of knitr's table + project-wide. + +## 13. Verification items (during implementation) + +- Full `cargo xtask verify` (not `--skip-hub-build`) — the new types + (`LanguageClaim`, ownership map, `ExecutionContext` field, + `ExecuteResult.html_dependencies`) live in `quarto-core` and feed + `wasm-quarto-hub-client`; resolution must compile + degrade gracefully in + WASM (markdown-only registry, execution bypassed by `CaptureSpliceStage`). +- `knit_engines` re-emit fidelity for attributed cells (`{python .marimo}` vs + `{python}`) — relevant only if per-cell routing is ever adopted. +- `python.reticulate: false` honoring (now that ceding python is via + `handled_languages`). +- `owned_languages` parity (Plan 4d): the positive projection carried on + `ExecutionContext`/`TsExecuteOptions.owned_languages` equals + `{ lang : ownership[lang] == k }` and is disjoint from `handled_languages` + over present languages (informational, not an enforcement change). +- Conversion-provenance: **faithful** original-file mapping is **deferred** (no + current consumer) — plan1a-engine scopes `markdown_for_file` to "C′": the + converted text is registered as an ephemeral intermediate file under an + engine-reflecting synthetic name, giving honest provenance *into the converted + buffer* (not back to the original non-QMD bytes). Plan 1c inherits only the + claim/seed (§8) + this ephemeral-FileId registration, **not** a faithful + remap. When a consumer needs converted-cell → source-cell positions, the + preferred path is the "A′" generalized FileId-remap (extends the existing + include/engine remap idiom); see plan1a-engine SEAM-3. +- `html_dependencies` survives the `EngineCapture` round-trip (`HtmlDependency` + / `TextInclude` need `Serialize`/`Deserialize`). diff --git a/claude-notes/instructions/release-runbook.md b/claude-notes/instructions/release-runbook.md index 4fcd8b0d7..b075dc1fd 100644 --- a/claude-notes/instructions/release-runbook.md +++ b/claude-notes/instructions/release-runbook.md @@ -228,6 +228,14 @@ still matches. and `install.sh` parse `${output##* }`. `q2 --version` prints `q2 (quarto 2) X.Y.Z`; anything appended to that string must keep the version last (guarded by `crates/quarto/tests/integration/version_cli.rs`). +- **Pin the embedded TS-extension-build jsr specifiers before shipping.** + `resources/extension-build/deno.json` (the shipped tier-4 config `q2 + build-ts-extension` embeds via `include_str!` for installed binaries — + see `crates/quarto/src/commands/build_ts_extension.rs`) currently + imports unpinned `jsr:@quarto/api` / `jsr:@quarto/types`. Once those + packages are published to JSR, pin them to a version + (`jsr:@quarto/api@^X.Y`) as part of cutting a release, so installed + binaries resolve a stable, tested API surface instead of latest. ## Files involved diff --git a/claude-notes/plans/2026-03-16-extensions-grand-plan.md b/claude-notes/plans/2026-03-16-extensions-grand-plan.md index f02868b83..f483c46fc 100644 --- a/claude-notes/plans/2026-03-16-extensions-grand-plan.md +++ b/claude-notes/plans/2026-03-16-extensions-grand-plan.md @@ -401,9 +401,41 @@ shortcode processing pipeline. Includes block-level shortcode support and a new template context alongside `css`. - **Detail plan**: `claude-notes/plans/2026-03-16-extensions-phase4-templates.md` +### Phase 5a: Format Extensions (resolution & apply) + +**Goal**: Extension-contributed output formats — the **common** Quarto 1 case +(journal templates like ACM/AGU/JSS, presentation themes) — resolve and apply +end-to-end. `--to acm-pdf` finds the `acm` extension and layers its +`contributes.formats.pdf` (+ `common`) bundle (metadata, per-format filters, +shortcodes, `template-partials`, `format-resources`, SCSS/theme) over the `pdf` +base. + +**Distinct from Phase 5 (Custom Writers):** a format extension targets a +*known* base format and layers config on it; it does **not** define a new +Pandoc target. A custom writer (Phase 5) is a `.lua` writer that *does*. They +are orthogonal — a format extension may also carry `writer: x.lua` (→ Phase +5) — and most real Q1 extensions are format extensions, not custom writers. + +- [ ] Wire extension context into format resolution so `-` loads the + extension's `formats[base]` (+ `common`) bundle +- [ ] Apply the bundle (metadata merge base→ext→user; per-format + filters/shortcodes; `template-partials`; `format-resources` copying; + SCSS/theme layering) +- [ ] Validate the extension actually contributes the requested base format + (loud error if not) +- [ ] Tests against a real journal fixture (ACM/AGU) + +**Detail plan**: `claude-notes/plans/2026-06-22-format-extensions.md` (STUB — needs research) + +**Status**: STUB — ingredients exist (Phases 1–4 + `Contributes.formats` + +`parse_format_descriptor`); the resolution-and-apply glue is unbuilt. + ### Phase 5: Custom Writers -**Goal**: Format extensions can provide custom Lua writers. +**Goal**: Extensions can provide custom Pandoc **Lua writers** (`.lua` format +keys that define a *new* output target) — distinct from Phase 5a format +extensions, which layer on a *known* base. A format extension may carry a +custom writer via `writer: x.lua`; this phase handles that `.lua`-writer path. - [ ] Detect format keys ending in `.lua` → custom writer format - [ ] Resolve writer path relative to extension directory @@ -414,34 +446,117 @@ shortcode processing pipeline. Includes block-level shortcode support and a new - Does pampa support custom Lua writers currently? - How would this interact with the WASM pipeline? -### Phase 6: RevealJS Plugin Support - -**Goal**: Extensions can contribute RevealJS plugins. - -- [ ] Parse `revealjs-plugins` from extensions -- [ ] Handle three forms: string path, bundle object, inline definition -- [ ] Wire plugin scripts/stylesheets into RevealJS output -- [ ] Handle per-format RevealJS plugins -- [ ] Tests +### Phase 6: RevealJS Plugin Support (extension-contributed) + +**Goal**: Extensions can contribute reveal.js plugins via +`contributes: revealjs-plugins:` (Q1 shape: plugin `name` + `script[]` + +`stylesheet[]`, with a `plugin.yml` carrying name/scripts/stylesheets/config). +At render, the listed plugins' assets are registered and their globals injected +into the `Reveal.initialize({ plugins: [...] })` call, with config merged +(plugin defaults → user front-matter). + +**RevealJS output now exists** (this answers the original open questions). The +`q2 render` path produces **static reveal.js HTML via Rust AST transforms** — +`RevealSlidesTransform` + `render_revealjs_document`, with `reveal_config_json()` +in `crates/quarto-core/src/revealjs/assemble.rs` emitting +``; reveal.js 6 is vendored at +`resources/revealjs/`. (The render path is **Rust, not React** — only the +hub-client *preview* renders the same shared slide-split AST via +`@revealjs/react`, kept in parity by golden tests.) See +`claude-notes/plans/2026-06-08-revealjs-presentations.md`. + +**Hard dependency — there is no plugin plumbing yet.** Today q2 loads **zero** +reveal.js plugins: `reveal_config_json()` has no `plugins:` key, no asset +registration for plugin JS/CSS, and not even the core plugins (Notes/Search/ +Zoom/Math). The **revealjs epic's own Phase 6 (Plugins/chrome)** is what vendors ++ wires the *built-in* plugins and adds the `plugins: [...]` init plumbing. +**Extension-contributed plugins (this phase) depend on that plumbing** — so +sequence this after (or co-design it with) the revealjs epic's plugin work, and +reuse its registration + init-emission seam rather than building a parallel one. + +- [ ] Add a `revealjs_plugins` field to the `Contributes` struct + parse + `contributes: revealjs-plugins:` (Q1 shape: `name`, `script[]`, + `stylesheet[]`; read the plugin's `plugin.yml` for name/scripts/stylesheets/config) +- [ ] Discover plugin contributions via the extension system when a document + lists `revealjs-plugins: [...]` +- [ ] Register plugin JS/CSS as artifacts (reuse the artifact store / theme-CSS + keying; copy to `site_libs/revealjs/plugin//`) +- [ ] Emit plugin globals + merged config into `Reveal.initialize({ plugins: [...] })` + in `assemble.rs`, reusing the built-in-plugin init seam +- [ ] Render/preview parity: the `@revealjs/react` preview path must load the + same plugins (or be documented as not-yet-at-parity) +- [ ] Per-format reveal plugins; tests with a real plugin extension (menu/chalkboard) **Open questions**: -- Does q2 have RevealJS output support yet? -- What's the plan for RevealJS format? +- Sequencing/co-design with the revealjs epic's built-in-plugin Phase 6 — share + one registration + `plugins: [...]` init seam for built-ins and extensions. +- Preview path (`@revealjs/react`): how reveal.js plugins (which target the + global `Reveal`) load under the React wrapper. ### Phase 7: Project Extensions -**Goal**: Extensions can contribute project-level configuration. - -- [ ] Parse `contributes.project` from extensions -- [ ] Merge into project config during project context creation -- [ ] Support `project.type`, `detect`, `render`, `preview` fields -- [ ] Support `pre-render` and `post-render` scripts -- [ ] Tests - -**Open questions**: -- Does q2 support project types beyond the default? -- How do pre/post-render scripts execute? -- How does project type detection work? +**Status: STUB / research.** We've researched what Q1 project extensions *are* +(below); we have **not** designed the q2 implementation. The items under "To +research" are open questions, not a vetted checklist. + +**What a project extension is (from Q1).** `contributes: project:` is +**external-toolchain integration glue** — not config layering, and not a new +project type. The marquee cases (docusaurus, hugo) make Quarto a *renderer +inside another tool's project*: Quarto renders `.qmd → markdown` that an +external static-site generator then consumes. An extension supplies: +- **detection** — `project.detect` glob-sets that auto-recognize a directory + (e.g. `hugo.toml` + `content/` ⇒ a hugo project); +- a **target format** to render to (`format: hugo-md` / `docusaurus-md`); +- a **preview command** — `preview.serve` (cmd/env/ready) launching the + external dev server (`hugo serve`, `npm run docusaurus start`); +- **pre-render / post-render scripts** — arbitrary executables run around the + render with a defined env-var contract; +- plus passive config layered onto a **built-in** type (`project.type`, + `website.*`, `render` globs, `output-dir`, …). + +**It does NOT define new project types.** Q1 has four built-in types +(default/website/book/manuscript); an extension's "type" (e.g. `docusaurus`) is +an extension *id* that detects a directory, **remaps to a built-in type** (via +its own `project.type` field), and layers config. `ProjectType` *behavior* (the +render hooks) stays built-in. The active parts — detection, the serve command, +the pre/post-render scripts — are the real work; the config layering is the +easy part. So the honest framing is "integrate an external SSG," not "add +metadata." + +**What exists in q2 (verified).** +- `Contributes.project: Option` is **parsed** (`extension/read.rs:179`) + and then **stored-and-ignored** — nothing consumes it. That is the entire + current implementation (ghostware). +- q2 has built-in project types via the two-pass orchestrator (at least + `DefaultProjectType`; the website-project epic added website handling), and + `ProjectType` exposes `pre_render` / `post_render` hooks the orchestrator runs + between/after passes. Whether those are the right home for *user* scripts is + not yet researched. + +**To research (open — needed before this is plannable).** +- **Scope first:** is external-SSG integration even a near-term q2 goal, or is + the near-term target just honoring extension-contributed project *config* + (sidebar/format/render) layered onto built-in types — deferring serve + + scripts? This decides how large Phase 7 is. +- **Resolution:** how would q2 resolve an extension-as-project-type and merge + its config, and where does that sit relative to `ProjectContext` / the + orchestrator? (Q1 does this in `resolveProjectExtension` + + `mergeProjectMetadata`, project-context.ts:678 — a *reference*, not a q2 + design.) +- **Detection:** Q1's "extension-id-as-type + `detect` globs + auto-detect + resolver" model, or something simpler? q2 has no project-type detection today. +- **Scripts:** are the orchestrator's `pre_render`/`post_render` hooks the + execution point (via `SystemRuntime::execute`, project cwd, Q1's env-var + contract `QUARTO_PROJECT_OUTPUT_DIR` / `_INPUT_FILES` / `_OUTPUT_FILES`)? And + the WASM story — scripts can't run in the browser preview. +- **`preview.serve`:** how would an external dev-server command coexist with + `q2 preview` (which serves its own SPA)? Likely the hardest piece. + +**References.** Q1: `src/project/project-context.ts` (`resolveProjectExtension`, +`projectExtensionsConfigResolver`), `src/command/render/project.ts` (script +execution + env vars), `src/resources/schema/project.yml`. q2: +`extension/read.rs:179`, the orchestrator's `ProjectType` +`pre_render`/`post_render` hooks. ### Phase 8: Engine Extensions @@ -501,22 +616,45 @@ integration that was originally scoped here. ### Phase 12: Semver Validation for `quarto-required` -**Goal**: Validate the `quarto-required` field in `_extension.yml` against the -running Quarto version, warning or erroring when an extension requires a version -the user doesn't have. +**Goal**: Validate `quarto-required` against the running Quarto version. q2 has **two surfaces that +share one semver gate**: (1) the **extension-package** field `quarto-required` in `_extension.yml` +(applies to every extension type; Q1 `extension.ts` `validateExtension`); (2) the +**engine-discovery** `quarto_required` an engine module declares (carried on `LoadEngineResult` by +RTQ ENG-1; Q1 `engine.ts` `checkEngineVersionRequirement`). Both check a semver range against +`cli_version()`. - [ ] Add `semver` crate dependency (dtolnay's — the de facto Rust standard) -- [ ] Parse `quarto-required` as `VersionReq` during `read_extension()` -- [ ] Check against `quarto_util::version::cli_version()` during extension discovery -- [ ] Emit a diagnostic when version doesn't satisfy the requirement -- [ ] Optionally parse and store `version` field as `semver::Version` -- [ ] Tests +- [ ] **Extension-package gate:** parse `quarto-required` as `VersionReq` during `read_extension()`; + check against `quarto_util::version::cli_version()` during extension discovery; emit a + diagnostic when unsatisfied. +- [ ] **Engine-discovery gate:** when `LoadEngineResult.quarto_required` (RTQ ENG-1) is set, check it + at engine registration — the q2 analogue of Q1's `checkEngineVersionRequirement` (`engine.ts:62`). + Reuse the same `VersionReq` / `cli_version()` helper. +- [ ] Optionally parse and store the extension's `version` field as `semver::Version` +- [ ] Tests (both gates) **Notes**: - `cli_version()` returns `"99.9.9-dev"` during development. Under strict semver, prereleases don't satisfy range constraints like `>=1.4.0`. Either strip the `-dev` suffix before checking or use `99.9.9` as the dev version. -- TS Quarto warns (not hard error) on version mismatch. +- **Engine-gate compat version (the 0.x problem).** Released q2 is `0.x`, but Q1 engine modules + declare Quarto-**1** ranges (e.g. julia's `quartoRequired: ">=1.9"`). Enforcing the *engine* gate + with q2's real version would reject **every** Q1 engine. So the engine gate must check against a + **Q1-compatible "engine compat version"** (e.g. `"1.11.0"`), distinct from q2's own + `cli_version()` — a deliberate spoof q2 presents to engine `quarto_required` checks. (Surfaced + while consolidating plan1c, which originally built this gate in 1c with the spoof; the gate + spoof + now live here.) **Compat-version source/value — DECIDED (lifted from plan1c, 2026-06-29):** use a + fixed `"1.11.0"`, isolated behind a single `fn engine_compat_version() -> &str { "1.11.0" }` so + there is exactly one place to revisit; do **not** scatter the literal. The engine gate compares the + engine's `quartoRequired` against `engine_compat_version()` (the spoof), **not** `cli_version()`; + `cli_version()` stays the source for the *extension-package* gate. This is a clearly-commented + stopgap until q2 settles its real version-compat story with the Q1 engine ecosystem. +- **Severity — decide deliberately.** Current Q1 source **throws** on *both* surfaces — the + extension gate (`extension.ts` `validateExtension`: "… is incompatible with this quarto version") + and the engine gate (`engine.ts` `checkEngineVersionRequirement`: hard `throw`). (An earlier note + here claimed Q1 *warns*; that is **stale** — the only `…AndWarn` path is for deprecated-now-built-in + extensions, not version mismatch.) q2 may keep throw for the engine gate (an unrunnable engine is a + hard failure) and choose warn-vs-throw for non-engine contributions. - The extension's own `version` field is currently stored as a plain string. Could optionally be parsed as `semver::Version` for consistency. diff --git a/claude-notes/plans/2026-04-16-julia-validation.md b/claude-notes/plans/2026-04-16-julia-validation.md index f66fee2a1..5abecd85c 100644 --- a/claude-notes/plans/2026-04-16-julia-validation.md +++ b/claude-notes/plans/2026-04-16-julia-validation.md @@ -1,9 +1,22 @@ # Plan 4: Julia Engine Validation **Grand plan:** [2026-04-16-ts-engine-extensions-subprocess.md](2026-04-16-ts-engine-extensions-subprocess.md) -**Depends on:** Plans 1, 2, and 3 (all must be substantially complete) -**Blocks:** Nothing (this is the final validation plan) -**Estimated sessions:** 1-2 +**Depends on:** Plans 1a/1b/1c, 2, 3, and 1c.2 P1.1+P1.1b — **all landed as of 2026-07-02** (see Prerequisites) +**Blocks:** Plan 4b (shadow-engine feature validation) +**Estimated sessions:** 2-3 (the net-new instrumentation in 4H/4I and the daemon test push this past the original 1-2 "pure debugging" estimate) +**Status: COMPLETE (2026-07-02).** All 13 success criteria met; frozen seams +J1–J6/J8/J9 green; V-1…V-7 evidence recorded (V-2 folded into J3); full +`cargo xtask verify` green at `1a44b4e2e`. Headline: `julia-engine.ts` ran +with ZERO source changes (rebundle byte-identical). Discovered-work strands: +bd-uf4epv4w (smart typography vs frontmatter strings), bd-l9jhy5u0 (QNR +worker leak on error path, P1 — reproduces under Q1 too), bd-cymkcyaf +(presentation-format per-writer execute defaults), bd-677297ca +(supporting-dir resource copy — FIXED+closed this session). Migration guide: +`claude-notes/research/2026-07-02-julia-engine-migration-guide.md`; evidence +trail: `…/2026-07-02-julia-engine-q2-compat.md` §1–§14. Julia-in-PATH stays a +per-session check by design (this session: julia 1.11.7). The unticked +observation sub-items in 4C/4H are honest divergence records (inline-MIME +figures; GR emits no htmlDependency), not gaps. ## Overview @@ -13,86 +26,202 @@ This plan is primarily integration debugging. If Plans 1a, 1b, 1c, 2, and 3 are ## Prerequisites -- [ ] Plans 1a, 1b, and 1c complete: Rust subprocess infrastructure + Deno harness + extension integration, echo engine passes -- [ ] Plan 2 complete: `@quarto/api` package with text/markdown/format/path/system/console/crypto subpaths, all QuartoAPI namespaces except `jupyter` wired in -- [ ] Plan 3 complete: `@quarto/api/jupyter` with `toMarkdown` working and wired into engine-host -- [ ] Julia installed on the test machine (`julia` in PATH) +**All code prerequisites are satisfied on this branch as of 2026-07-02** — the +first action is to *confirm* them green, not to build anything. + +- [x] Plans 1a, 1b, and 1c complete: Rust subprocess infrastructure + Deno harness + extension integration, echo engine passes (grand-plan table: all ✓) +- [x] Plan 2A complete: the `@quarto/api` package skeleton (`package.json`, `tsconfig.json`, exports map) and the `./config` key-list subpath are in place +- [x] Plan 2 complete: the remaining QuartoAPI surface built on that skeleton — `@quarto/api`'s text/markdown/format/path/system/console/crypto subpaths, all QuartoAPI namespaces except `jupyter` wired in +- [x] Plan 3 complete: `@quarto/api/jupyter` with `toMarkdown` working and wired into engine-host +- [ ] Julia installed on the test machine (`julia` in PATH) — machine-specific, verify per session +- [x] **Plan 1c.2 P1.1 — LANDED** (commit `2b2113e6c`; e2e tests + `p1_1_project_context_threaded_{project,single_file}_leg`). `set_project` / + per-render `EngineProjectContext` wired. It only ever gated Phase 4H and one + success criterion — both are wiring assertions *(Q1's julia-engine.ts ignores + the launch context entirely; its `launch(context)` never reads a field)*. +- [x] **Plan 1c.2 P1.1b — LANDED** (commit `5fdcd4b56`; e2e test + `p1_1b_document_metadata_threaded_into_format`). Merged document metadata now + flows into `TsFormatInfo.metadata` (`metadata: ctx.metadata.clone()`, + `ts_engine.rs:396`) — so `execute:`/`julia:` frontmatter reaches the engine, + Phase 4E is testable, and `execute: daemon: false` (which every Plan-4 test + document sets — see the 4B warning) actually disables the detached Julia + daemon. Confirm green: `cargo nextest run -p quarto-core -E 'test(p1_1)'`. ## Work Items -### Phase 4A: Set up Julia engine extension +> **Execution order = document order.** The phase letters are historical: 4G (documentation) +> was moved last; 4H/4I were appended after the website-project epic landed on `main`; and 4F +> (regression audit) was moved after 4I on the 2026-07-02 review so the full-workspace verify +> covers all net-new code (4H/4I instrumentation) instead of running mid-plan. -- [ ] Copy Julia engine from Quarto 1's **source/development version** (NOT the pkg-working version): - ``` - ~/src/quarto-cli/src/resources/extension-subtrees/julia-engine/ - ``` - Use this version because it resolves resource files via `import.meta.url` (relative to the JS file), rather than the distributed version which uses `quarto.path.resource()` pointing to Quarto's global `share/` directory. +### Phase 4A: Set up Julia engine extension -- [ ] Create test fixture with engine source AND its resource files: - ``` - tests/fixtures/extensions/julia-engine/ - _extension.yml - src/ - julia-engine.ts - constants.ts - Project.toml ← Julia environment definition - ensure_environment.jl ← Julia setup script - quartonotebookrunner.jl ← Julia execution entry point - start_quartonotebookrunner_detached.jl ← Daemon launcher - ``` - The .jl files and Project.toml live alongside the extension (same directory or parent) so that `dirname(import.meta.url)` resolves them. This matches Quarto 1's development-mode layout where the extension is self-contained. -- [ ] Write `_extension.yml`: +**We run the existing extension, not a re-specified one.** (Decision 2026-07-02.) +The source of truth is the upstream repo `~/src/quarto-julia-engine` (same +content as Q1's `src/resources/extension-subtrees/julia-engine/` subtree). Its +layout is already correct for q2: `_extensions/julia-engine/` holds +`_extension.yml`, the bundled `julia-engine.js`, `Project.toml`, and the three +`.jl` scripts — co-located because `julia-engine.ts` resolves resources via +`dirname(fromFileUrl(import.meta.url))`, i.e. the directory of the loaded +bundle (`src/julia-engine.ts:46`). `src/` holds the TS source. Do NOT invent a +new layout; the fixture copy is modified for q2 static claiming, and merging +those changes back upstream is deferred (Gordon's call). + +- [x] Bring `resources/extension-build/deno.json` (and `deno.workspace.json`) + to **Q1 import-map parity**: add the bare-specifier aliases from Q1's + `src/resources/extension-build/import-map.json` — `path`, `path/posix`, + `log`, `log/`, `fs/`, `encoding/` → pinned jsr `@std` packages + (`@std/path@1.0.8`, `@std/log@0.224.0`, `@std/fs@1.0.16`, + `@std/encoding@1.0.9`). This is what lets `julia-engine.ts`'s bare imports + (`"path"`, `"fs/exists"`, `"encoding/base64"`) bundle **unchanged**. The q2 + port of the config dropped these aliases (apparent oversight — no recorded + decision); note the parity restoration against plan1c's config spec + (plan1c lines ~421-446). +- [x] Copy `~/src/quarto-julia-engine` into + `crates/quarto-core/tests/fixtures/extensions/julia-engine/` (the + established extension-fixture location, next to `echo-engine/`), preserving + its layout. **Exclude `.git/`** and repo-only files (`tests/`, `example*`) as + convenient; keep the root `_quarto.yml` — it makes the fixture a renderable + project dir, which is where the 4B–4E test documents live. Provenance note: + `~/src/quarto-julia-engine` is a machine-local checkout (same content as + Q1's `extension-subtrees/julia-engine/`); this is a **one-time copy** — the + committed fixture is what tests use thereafter, so no build or test may + reference the home-dir path or `external-sources/`. +- [x] Modify the fixture copy's `_extensions/julia-engine/_extension.yml` for + **q2 static claiming** (q2-native keys; Q1's file has none): ```yaml - title: Julia Engine - author: Quarto - version: 1.0.0 contributes: engines: - - name: julia - path: src/julia-engine.ts + - path: julia-engine.js + name: julia + claims: + julia: { kind: primary, priority: 1 } # static claim → zero-load resolution + file-extensions: [".jl"] # can-handle pre-filter (decision 2026-07-02) ``` -- [ ] Identify needed modifications to `julia-engine.ts`: - - Import paths: change `@quarto/types` imports if our type names differ + With the static `claims:` declared, q2 resolves `{julia}` cells to this + engine **without loading the Deno subprocess** — it spawns only to execute, + once Julia has won ownership (see `claude-notes/designs/engine-resolution.md` + §3.3). The engine's dynamic `claimsLanguage` returns a bare **boolean** + (`language.toLowerCase() === "julia"`, not a `primary()` object); it is + validated against the static claim on first load (`ensure_loaded`, + `ts_engine.rs:240-329`, hard error on mismatch). *(Seam check 2026-07-02: + the normalization `true` → `{kind: primary, priority: 1}` is already pinned + — `mapLanguageClaim` at `host.ts:168` with named-revert tests at + `host.test.ts:423`/`:489` — so the comparison will pass; if the first Julia + load hard-errors, look elsewhere.)* + - Resolved 2026-07-02: **declare `file-extensions: [".jl"]`** (as above), + matching `engine-resolution.md` §3.3's own Julia example. It is only a + can-handle **pre-filter**; `claims-files` stays undeclared (Julia's + `claimsFile` is content-inspecting — `# %%` percent scripts), so this + does **not** cause Pass-1 subprocess spawning and Phase 4I's zero-spawn + assertion stands unchanged. (The earlier worry conflated the two axes.) + Note: 1c.2 P4 restructures `claims-files` into typed `{extension}` entries + (the earlier plan to rename it `claims-extensions` is **withdrawn** — the + name stays; not landed as of 2026-07-06). Julia declares no static + `claims-files` today, so this plan is unaffected. Forward: Plan 7a lets Julia + claim `.jl` statically via a `content-pattern` **evaluated natively as a + regex** — no Pass-1 spawn — so even that future change *preserves* Phase 4I's + zero-spawn assertion. +- [x] Rebundle with `cargo run --bin q2 -- build-ts-extension src/julia-engine.ts` + (run from the fixture dir, after the import-map parity item; verify the + output lands at `_extensions/julia-engine/julia-engine.js` — q2 mirrors + Q1's output-path convention; see + `crates/quarto/src/commands/build_ts_extension.rs` if it doesn't) and commit + the regenerated bundle. The upstream bundle was built by Q1's + `quarto call build-ts-extension`; rebuilding under q2's config is the real + build-compatibility test. + - **Note:** the literal invocation above doesn't run as written — `PATH` + must be the extension directory (`_extensions/julia-engine`), not the + `.ts` file, and `find_entry_ts` additionally requires the TS source to + live at `/src/.ts`, which upstream's real (root-level + `src/`) layout doesn't satisfy. Worked around with a local, uncommitted + symlink (`_extensions/julia-engine/src -> ../../src`) for the duration + of the build only; removed immediately after. Full detail + the exact + two failure modes: `claude-notes/research/2026-07-02-julia-engine-q2-compat.md` + §4/§8.1. Result: the rebuilt bundle is **byte-identical** to the + upstream Q1-built one (MD5 `d9d5120eb94b187903a43fb500e65eea`). +- [x] Identify any remaining needed modifications to `julia-engine.ts`: - API calls: verify all `quarto.*` calls match our implementation signatures - - Resource resolution: verify `import.meta.url`-based paths work after bundling (the bundled .js file's URL determines the base directory — resource files must be relative to where the bundle is loaded from) - - Deno APIs: verify `Deno.Command`, `Deno.connect`, `crypto.subtle`, file I/O all work (they should — it's running in real Deno) - - Standard library imports: `"path"`, `"fs/exists"`, `"encoding/base64"` — resolved at build time via the import map -- [ ] Document every modification in a compatibility log + (25 distinct calls across 8 namespaces; all exist in `@quarto/api` per the + 2026-07-01 audit) — **audit re-run 2026-07-02 found 7 namespaces / 30 + call sites** (not 8/25); no missing-API issues found either way. See + compat log §7 for the reconciliation flag. + - Resource resolution: verify `import.meta.url`-based paths work after + rebundling (bundle must stay co-located with the `.jl` files) — single + use-site (`julia-engine.ts:46`), fixture keeps the bundle co-located + with the `.jl` scripts under `_extensions/julia-engine/`; no issue found, + flagged for 4B to confirm the TS-engine host doesn't relocate the bundle + before load (compat log §7). + - Deno APIs: verify `Deno.Command`, `Deno.connect`, `crypto.subtle`, file + I/O all work (they should — it's running in real Deno) — all present in + source (compat log §7); execution-time verification deferred to 4B. +- [x] Document every modification in a compatibility log (input for merging + back to `~/src/quarto-julia-engine`, and for the 4G migration docs) — see + `claude-notes/research/2026-07-02-julia-engine-q2-compat.md`. ### Phase 4B: Minimal Julia render The simplest possible Julia document. -- [ ] Create test document: +> **Daemon warning (read before the first render).** The julia engine's daemon +> default is `isInteractiveSession && !runningInCI`, and q2 wires those to real +> runtime values — so an interactive dev-machine render **starts a detached +> Julia server that outlives q2**, and q2 has no management surface for it +> (bd-m1jeqhhz). Policy (decision 2026-07-02, details in 4E): every Plan-4 test +> document sets `execute: daemon: false` — effective because 1c.2 **P1.1b** has +> landed (before it, the option was dropped with the rest of the metadata and +> could not disable the daemon). If a daemon escapes anyway: the transport file in the julia runtime +> dir (`quarto.path.runtime("julia")`, see `juliaTransportFile()`) has the +> server's port/PID — kill it from there. + +- [x] Create test document **at the fixture root** (next to `_extensions/`; the + fixture is a renderable project dir via its upstream `_quarto.yml`): ```markdown --- engine: julia + execute: + daemon: false --- ```{julia} 1 + 1 ``` ``` -- [ ] Run through q2's render pipeline. Use `cargo run -- render ` (the `quarto` crate at `crates/quarto/` is the main CLI binary). Check existing smoke tests in `crates/quarto/tests/` for how integration tests invoke rendering programmatically. -- [ ] Debug the first failure. Common failure checklist: - - [ ] Extension not discovered → `_extension.yml` parsing issue - - [ ] Deno subprocess won't start → Deno not in PATH, or engine-host-deno bundle issue - - [ ] Engine module fails to load → import resolution, transpilation issue - - [ ] `engine.init()` fails → QuartoAPI construction issue - - [ ] `engine.launch()` fails → EngineProjectContext mismatch - - [ ] Julia process won't start → `Deno.Command` issue, Julia not in PATH - - [ ] Julia server connection fails → TCP connect issue, HMAC auth issue - - [ ] Execution succeeds but output is wrong → `toMarkdown()` issue - - [ ] Result deserialization fails → protocol/type mismatch -- [ ] Iterate until the simple document renders successfully -- [ ] Verify output HTML contains the result `2` + (committed as `crates/quarto-core/tests/fixtures/extensions/julia-engine/minimal.qmd`) +- [x] Run through q2's render pipeline. Use `cargo run --bin q2 -- render ` (the crate at `crates/quarto/` builds the `q2` binary). The automated-test harness template is `crates/quarto-core/tests/integration/echo_engine_e2e.rs` (see the Test Seam Spec). +- [x] Debug the first failure. Three real failures found + fixed (full trail in + compat log §9). Against the checklist: + - [x] Extension not discovered → `_extension.yml` parsing issue — YES: q2 + requires an `author` field (q2-vs-Q1 divergence); added to fixture. + - [x] Deno subprocess won't start — no (started fine). + - [x] Engine module fails to load — no. + - [x] `engine.init()` fails → QuartoAPI construction — no (API pre-flight: + all 25 called members exist, jupyter's 6 are all implemented). + - [x] `engine.launch()` fails — no. + - [x] Julia process won't start — no. + - [x] Julia server connection fails — no (HMAC/TCP fine). + - [x] Execution succeeds but output is wrong → **two distinct q2 bugs**: + (a) empty wire source map made julia's `buildSourceRanges` send `[]` → + QNR crashed on `maximum([])`; fixed by serializing a real source map in + `ts_engine.rs`. (b) missing execute-visibility defaults + (`include/output/eval`) made `jupyterToMarkdown` drop every cell → empty + body; fixed with `applyExecuteDefaults` in the engine-host. + - [x] Result deserialization fails — no. +- [x] Iterate until the simple document renders successfully +- [x] Verify output HTML contains the result `2` — `

…2…
`; frozen as seam **J1** + (`crates/quarto-core/tests/integration/julia_engine_e2e.rs`), RED/GREEN + + named-revert proven. ### Phase 4C: Julia with figures -- [ ] Create test document with a plot: +- [x] Create test document with a plot (daemon policy: `daemon: false` like + every Plan-4 doc): ```markdown --- engine: julia + execute: + daemon: false --- ```{julia} @@ -100,18 +229,42 @@ The simplest possible Julia document. plot(1:10, rand(10)) ``` ``` -- [ ] Verify: - - [ ] Figure file generated in `_files/` directory - - [ ] Figure referenced correctly in output markdown - - [ ] `supporting` files tracked in `ExecuteResult` - - [ ] HTML output renders with the figure + Committed as `crates/quarto-core/tests/fixtures/extensions/julia-engine/plot.qmd`. + Required a one-time notebook-environment setup NOT anticipated by this plan + item's literal text: the fixture root needed its own `Project.toml`/ + `Manifest.toml` (declaring `Plots`), separate from the extension's own + `_extensions/julia-engine/Project.toml` — QuartoNotebookRunner activates + `JULIA_PROJECT=@.` from the *notebook's* directory, not the engine's. Not a + q2 bug; committed the environment files (40K `Manifest.toml`). Full trail: + compat log §10. +- [x] Verify (concrete targets, per the e2e norm — record invocation + snippet): + - [ ] Figure file generated at `_files/figure-html/-output-*.png` + (or the fig-format in effect) — **did NOT materialize for this + document**: Plots.jl's default GR-backend plot is `text/html`-showable, + and Q1's own (faithfully ported) MIME-type priority always prefers + `text/html` over `image/png` for HTML targets, so the file-writing + branch (`mdImageOutput`) is never reached. Traced, not a q2 bug — see + compat log §10 and the 4CD task report. + - [ ] Output HTML contains `` (embedded). + - [x] `supporting` files tracked in `ExecuteResult` — **CONFIRMED**, via a + temporary `eprintln!` on `map_execute_result` (added, observed, + removed): `supporting=["…/plot_files"]`, one entry, sent + unconditionally by `julia-engine.ts` regardless of whether a figure + file actually landed there. See compat log §10. + - [x] The figure displays when the HTML is opened — confirmed (valid, + complete base64-embedded PNG). ### Phase 4D: Multiple cells and error handling -- [ ] Test multiple code cells: +- [x] Test multiple code cells (`execute: daemon: false` in the frontmatter, + per the daemon policy — elided below for brevity): ```markdown --- engine: julia + execute: + daemon: false --- ```{julia} @@ -122,82 +275,560 @@ The simplest possible Julia document. println("x is $x") ``` ``` -- [ ] Verify state persists between cells (x defined in first, used in second) + Committed as `crates/quarto-core/tests/fixtures/extensions/julia-engine/multi-cell.qmd`. +- [x] Verify state persists between cells (x defined in first, used in second) + — **V-5, manual, confirmed**: cell 2's stdout output is `x is 42`. Invocation + + full snippet in the 4CD task report and compat log §10. -- [ ] Test error handling: +- [x] Test error handling: ```markdown --- engine: julia + execute: + daemon: false --- ```{julia} error("this should fail gracefully") ``` ``` -- [ ] Verify error produces a useful message, not a crash + Landed as the frozen seam **J4** + (`crates/quarto-core/tests/integration/julia_engine_e2e.rs::j4_error_handling_does_not_wedge_host`), + not just a manual doc — inline `ERROR_DOC` const, matching J1's precedent. +- [x] Verify error produces a useful message, not a crash: the render reports a + diagnostic (or non-zero exit) that includes the Julia error text + (`this should fail gracefully`) and ideally the cell's source location; q2 + itself must not panic and the Deno subprocess must not be left wedged + (a subsequent render of the 4B document still works) — **confirmed, J4 + GREEN, RED-proven via the named revert in `TsEngineHost::request`'s + `FromEngine::Error` arm (`ts_process.rs:~693`).** Note: the first draft of + the test's error-message assertion (`contains("this should fail + gracefully")` alone) turned out to be vacuous against this exact revert — + `TsEngine::execute`'s generic fallback error message still contains that + substring via its `{:?}` Debug dump. Strengthened before freezing (see the + 4CD task report and compat log §10 for the full RED/GREEN/named-revert + trail, done twice — once exposing the vacuous assertion, once against the + fixed one). ### Phase 4E: Julia-specific features -- [ ] Test daemon mode (`execute.daemon: true`) — Julia server stays alive -- [ ] Test `exeflags` option — arguments passed to Julia -- [ ] Test `env` option — environment variables set for Julia -- [ ] Test cell options: `echo: false`, `output: false`, `warning: false` +**Substrate: complete** (1c.2 P1.1b landed — see Prerequisites). Rust threads +merged metadata into `TsFormatInfo.metadata`; the Deno host's +`metadataAsFormat` partitions it into Q1's six-bin `Format` +(`daemon`/`fig-dpi`/etc. land in `format.execute`; a `julia:` block lands in +`format.metadata` and reaches QuartoNotebookRunner via the serialized +options). + +**Daemon policy (decision 2026-07-02):** all Plan-4 fixtures set +`execute: daemon: false` (oneShot) **except** the one dedicated daemon test +below. q2 has no equivalent of Q1's `quarto call engine julia +status/kill/log/close/stop` (`populateCommand` is not wired) — future surface +tracked as **bd-m1jeqhhz**. The daemon test tears down out-of-band: transport +file in the julia runtime dir → port/PID → kill. + +- [x] Test daemon mode (`execute.daemon: true`) — **V-1 (manual, deliberately + not frozen)**: evidence recorded (compat log §11 + 4E task report), run in + an isolated `HOME` because the transport file is global per user and a + concurrent docs-agent daemon owned the real one (left untouched, verified). + Key findings: `daemon: false` starts the detached control server anyway and + closes only the per-file WORKER (transport file persists); `daemon: true` + keeps the worker open — second render 0.33 s vs 5.73 s and the cell's + `getpid()` printed the SAME worker pid (in-band reuse proof); teardown via + transport-file PID → SIGTERM kills the worker and removes the transport + file (QNR atexit). Zero orphan processes (final `ps` matched the + pre-session baseline exactly). Stable → **promotable to a J-row** + (additive seam entry, controller sign-off required; needs a HOME-isolated + harness). +- [x] Test `exeflags` — **J3**: landed as + `julia_engine_e2e::j3_exeflags_and_env_through_julia_block`, GREEN and + RED-proven — but with two evidence-backed deviations from the frozen row + (flagged for controller sign-off; full trail in the 4E task report and + compat log §11): + 1. **Fixture placement**: the `julia:` block lives in the temp project's + `_quarto.yml`, not document frontmatter — document-frontmatter + `--threads=2` is smart-typography-mangled to `–threads=2` (en dash) by + q2's DocumentMetadata markdown parse, and QNR treats it as a file arg + (real substrate bug, filed **bd-uf4epv4w**; project-config strings stay + literal, so the spec's exact flag survives there). + 2. **Named revert re-anchored**: the spec'd T14 revert CANNOT redden any + QNR-observable assertion — QNR merges the notebook file's own + frontmatter under the wire options AND (deeper) julia-engine.ts sends + `target.markdown` (q2's post-merge serialized AST), which QNR's socket + layer uses as a file-content override (`socket.jl:497`) — so merged + metadata reaches QNR even with the wire path reverted (verified + empirically: the T14 revert left both document-level and project-level + fixtures GREEN). Re-anchored revert: the project metadata layer in + `MetadataMergeStage::run` (`metadata_merge.rs:~214`) → RED proven. + T14 itself stays revert-bound by J2 (host-side `format.execute` + consumption has no QNR fallback). + Schema stop-point resolved against installed QNR 0.17.4 source (matches + the fixture pin): `options["format"]["metadata"]["julia"]["exeflags"]` / + `["env"]` (`server.jl:151-168`), i.e. a top-level `julia:` mapping with + `exeflags`/`env` string arrays. +- [x] Test `env` — **folded into J3** (same doc/test, one extra cell line + + frozen assertion `FOO=BAR`; no separate V-2 needed). +- [x] Test cell options — **J2** landed as + `julia_engine_e2e::j2_document_level_echo_false_hides_source_keeps_output`, + GREEN, RED-proven verbatim against the spec'd T14 revert + (`metadata: HashMap::new()` at `ts_engine.rs:394` → source listing present + → RED at the source-absent assertion). Manual greps recorded in the 4E + task report: `#| output: false` → cell output absent; + `#| warning: false` → warning text absent, normal output present. + +### Phase 4H: Website-project integration + +Validates that the TS engine subsystem cooperates with the two-pass +project orchestrator (`ProjectPipeline`) that landed on `main` after +these plans were drafted. The Julia engine itself has no project- +specific logic, so this phase is a smoke test of the integration. + +- [x] Create a minimal website fixture with a Julia page and a + markdown page — **DONE** (`crates/quarto-core/tests/fixtures/extensions/ + julia-website/{_quarto.yml, index.qmd, plot.qmd}`; `_extensions/` + + notebook `Project.toml`/`Manifest.toml` copied in at runtime from the + sibling `julia-engine` fixture, nothing julia-specific committed under + `julia-website/`). `plot.qmd` uses the file-based-figure mechanism + (`GKSwstype=100` + `savefig` + an `image/png`-only `PngFigure` wrapper — + see the 4H task report §1; `fig-format: png` was investigated and rejected + as insufficient for Plots' `text/html`-showable default). + ``` + crates/quarto-core/tests/fixtures/extensions/julia-website/ + _quarto.yml # project.type: website + _extensions/julia-engine/ # populated from ../julia-engine/_extensions/julia-engine + index.qmd # markdown only + plot.qmd # ```{julia} plot(...) ``` with figures + ``` + Do **not** commit a symlink for `_extensions/julia-engine` — committed + symlinks are unreliable on Windows checkouts (`.claude/rules/cross-platform.md`). + Per the seam spec (J5): commit only `_quarto.yml` + the two `.qmd`s under + `julia-website/`; the automated test's setup copies the extension in from + the sibling `julia-engine` fixture at runtime (the echo `setup_project` + pattern). For manual renders, copy it in the same way. +- [x] Run `cargo run --bin q2 -- render ` and verify (manual, + recorded in the 4H task report §1/§7): + - [x] `_site/index.html` and `_site/plot.html` both produced — **DONE.** + Both files are written AND the render now returns Ok: the + bd-677297ca blocker (file_copy on the `plot_files` DIRECTORY + supporting entry) was fixed by expanding supporting directories + into contained files in `DocumentResourceReport::add_engine_files` + (option (c), controller-adjudicated). Bound by the now-un-ignored + J5 (`j5_website_figure_lands_as_file_and_is_referenced`, GREEN). + - [x] `_site/site_libs/` contains shared assets (`bootstrap/`, `quarto/`) + *(observation only — accepted-untested)* + - [x] Julia-emitted figures land at the expected per-page location + (`_site/plot_files/figure-html/cell-2-output-1.png`), **not** in + `site_libs/` (`find _site/site_libs -name '*.png'` → empty). Bound by + J5 (bd-677297ca fixed; J5 un-ignored and GREEN). + - [ ] `htmlDependency` — the default Plots/GR backend emits none + *(accepted-untested — if-observed only; not observed)* + - [x] Sidebar/navbar transforms run normally (the `_quarto.yml` navbar + rendered) *(observation only — accepted-untested)* +- [x] Verify the `Arc` is shared across both files' renders: + **J8 (observable) landed + unit-tested TDD-first; J6 (assertion) GREEN.** + Net-new production `tracing::info!(target: "engine_host", pid, …)` in + `ensure_started_inner` — GREEN, RED→GREEN + named-revert proven + (`test_j8_spawn_event_*` in `ts_process.rs`). The one-spawn property was + first observed end-to-end (manual `q2 render`: exactly one `engine-host + spawned` event, ordered after two `engine resolution complete` events — + report §4), and is now bound by the **automated** J6 project-render row + (`j6_one_engine_host_per_project_render`), un-`#[ignore]`d after + bd-677297ca was fixed (see the 4H item above) and GREEN; it runs under + `QUARTO_JOBS=1` so the whole render stays on the thread the tracing + capture is scoped to (rayon Pass-2 workers don't see thread-local + subscribers). Capture discrimination proven GREEN by + `j6_capture_discriminates_two_hosts` (deno-only). The J8 event also serves + Phase 4I (J9) — J9's resolution-complete event was ALSO added now + (`resolve_engines`, target `engine_resolution`) so 4I is test-only. +- [x] Verify each file's Julia `launch()` receives a populated project + context (V-3) — **DONE** (report §6): temporary launch-site instrumentation + (reverted before commit) observed + `project_dir=Some() output_dir=Some(/_site) is_single_file=false` + — non-empty, correct, not `default()`. + +### Phase 4I: Pass-1 cost audit + +Pass 1 advances every project file to the `DocumentProfile` +checkpoint without running engines. For Julia documents this means +parse + metadata-merge only. Verify the Julia subprocess is **not** +started during Pass 1 *or during Pass-2 resolution* — with static +claims it spawns only at the first execute (Julia claims by language +only; no `claims_file` wiring for `.jl` percent scripts in v1). + +> **Seam check 2026-07-02 — the original assertion here was vacuous.** +> "Spawn happens after Pass 1" is true for a legacy dynamic-claims engine +> too (it spawns during Pass-2 *resolution*), so reverting the entire +> static-claims machinery would leave it green. The discriminating surface +> is **"no spawn during resolution"** — the spawn event must order after +> resolution-complete (i.e. at first execute). Seam spec row **J9** binds +> this echo-based (deno-gated only, no Julia needed); the Julia run below +> is recorded evidence (V-4) on top. + +- [x] Implement J9 (echo-based ordering test: both events present AND spawn + AFTER resolution-complete; named revert = the static early-answer branch + in `claims_language`, `ts_engine.rs:~550`; the resolution-complete event + fires at the end of `resolve_engines`, `resolution.rs:~334`). — + `j9_resolution_before_spawn_zero_load` in `echo_engine_e2e.rs`, RED→GREEN + + named-revert proven verbatim (see compat log §12). Commit `62d7dadf6`. +- [x] In a website fixture with multiple Julia pages, reuse the same + events (J8) to observe spawn timing. Render with + `cargo run --bin q2 -- render `. — done via a temp copy of the + committed `julia-website` fixture, `RUST_LOG=engine_host=info, + engine_resolution=info ./target/debug/q2 render /tmp/v4-julia-website`. +- [x] Verify and record (V-4): exactly one spawn, ordered after + resolution-complete and at the first Julia execute. — confirmed: one + `engine-host spawned` line, after both `engine resolution complete` + lines, immediately followed by the child's own execute-time stderr + (`Running [1/1] at line 27...`, the first line of `plot.qmd`'s cell). + Full log snippet in compat log §12. +- [x] If `claims_file` is wired for `.jl` percent scripts later, the + subprocess will spawn during Pass 1 — note that as expected + behavior, not a regression. — not wired in v1; noted here as the + documented future-behavior caveat, no action needed now. + +### Phase 4J: Julia-in-preview validation (V-7 — added 2026-07-02, user-requested) + +Plan 1c's **R5** wired TS engines into `q2 preview`'s **native** capture → +splice pipeline (all three call sites: eager `capture_driver.rs`, +`preview_record`/`cache.rs`, `re_execute.rs`) and proved it with the echo +engine (P2-14). Nothing has validated a *real* engine through preview. This +phase is **manual evidence only (V-7)** — no frozen test; the binding for the +registry-read hunks stays P2-14's echo seam. + +- [x] Run `cargo run --bin q2 -- preview ` against a + `daemon: false` Julia doc (temp copy of the committed fixture). Record: + the initial preview shows executed output (the `2`, not an inert code + block); the capture path logged a real engine execute. **Caveat:** curl + against the served page only reaches the SPA shell (content syncs over + the automerge/samod websocket, not a plain HTTP GET); confirmed instead + via the Phase C.7 filesystem cache (`/captures/*.bin`), which + the code documents as byte-identical wire format to what the WASM side + ungzips — see compat log §13. +- [x] Edit the cell (e.g. `1 + 1` → `2 + 3`) and record the live + re-execution result (`5`) through the `/api/preview/re-execute` path. + Confirmed via the same cache-file mechanism + tracing (compat log §13). +- [x] Observe daemon behavior under preview: preview is an interactive + session, so WITHOUT `daemon: false` julia would default to a detached + server (bd-m1jeqhhz — no management surface). Record transport-file + state after the `daemon: false` session (expect none) and note the + `daemon: true`-by-default hazard for real users in the compat log. + Confirmed: shared daemon transport files unchanged across the whole + session (compat log §13). +- [x] Cleanup: verify no orphan julia/QNR processes from the session. + Confirmed: 25 julia processes before and after (identical to the + pre-existing bd-l9jhy5u0 leaked pool; no new entries); the two + engine-host PIDs had already exited before shutdown (compat log §13). +- [x] Record all invocations + snippets in the compat log (§13); note any + divergence between preview-spliced output and the `q2 render` output of + the same doc (they need not be pixel-identical — note, don't fix). No + divergence found — both show `5` for the edited doc. ### Phase 4F: Regression audit -- [ ] Run same test documents through Quarto 1 for comparison -- [ ] Document output differences -- [ ] Verify all existing q2 tests pass (`cargo nextest run --workspace`) -- [ ] Run `cargo xtask verify` for full validation -- [ ] File issues (via `br create`) for any gaps discovered +(Moved after 4I on the 2026-07-02 review so the full verify covers the 4H/4I +net-new instrumentation code, not just the fixture work.) + +- [x] Run same test documents through Quarto 1 for comparison — done via + `~/bin/quarto` (dev checkout, `99.9.9`) against a temp copy of + `~/src/quarto-julia-engine`; minimal/multi-cell/error/echo-false docs all + rendered. See compat log §14. +- [x] Document output differences — compat log §14 comparison table + + corrected-finding write-up (the §9 "HTML hides source by default" note was + overstated for plain HTML; the real gap is presentation-format-scoped). +- [x] Verify all existing q2 tests pass (`cargo nextest run --workspace`) — + verify #2 green 2026-07-02, exit 0 (HEAD `1a44b4e2e`); not re-run this + session per the session's already-green status. +- [x] Run `cargo xtask verify` for full validation — verify #2 green + 2026-07-02, exit 0 (HEAD `1a44b4e2e`); not re-run this session per the + session's already-green status. +- [x] File issues (via `braid create`) for any gaps discovered — bd-cymkcyaf + filed (format-agnostic execute defaults vs. Q1's presentation-format + overrides); bd-uf4epv4w, bd-l9jhy5u0, bd-m1jeqhhz, bd-677297ca reviewed, + no new strands needed for those. ### Phase 4G: Adaptation documentation -- [ ] Write a summary of all changes needed to `julia-engine.ts` -- [ ] Categorize changes: - - Import path adjustments - - API signature differences - - Missing QuartoAPI methods (if any were stubbed) - - Behavioral differences -- [ ] This becomes the basis for documentation for extension authors migrating from Quarto 1 +- [x] Write a summary of all changes needed to `julia-engine.ts` — headline + result: **zero source changes** (byte-identical rebundle, compat log §4). + Written up as `claude-notes/research/2026-07-02-julia-engine-migration-guide.md`. +- [x] Categorize changes: + - Import path adjustments — none in the extension; one repo-side shipped + fix (import-map parity, `e56da9c29`). + - API signature differences — none found (30/30 call sites match, §7/§9). + - Missing QuartoAPI methods (if any were stubbed) — none (all 6 + `jupyter.*` members implemented). + - Behavioral differences — `_extension.yml` q2-native keys + required + `author`; `build-ts-extension` directory-resolution mismatch (symlink + workaround); notebook-environment setup (CI Manifest.toml cost + callout); three q2-side completeness fixes (execute-visibility + defaults, execute source map, supporting-dir expansion) that are q2 + catching up to Q1 behavior, not engine adaptation; four still-open + tracked gaps (bd-cymkcyaf, bd-uf4epv4w, bd-l9jhy5u0, bd-m1jeqhhz). +- [x] This becomes the basis for documentation for extension authors + migrating from Quarto 1 — the migration guide is framed explicitly for + that audience (see its "Bottom line for an extension author" section). + +## Test Seam Spec (frozen — prevalidated 2026-07-02) + +One row per durable automated test this plan produces. **Tier · real unit +mounted · seam (harness + assertion surface) · mock boundary · named revert +hunk.** Once green, assertions and harness are frozen — never edited to go +green. Manual validations (V-rows) name the production hunk whose absence +would change the recorded output; they are evidence, not regression guards. + +**Harness template (all J-rows):** the echo pattern in +`crates/quarto-core/tests/integration/echo_engine_e2e.rs` — **in-process** +`render_to_file` (same entry as `quarto render`) / `ProjectPipeline`, fixture +copied into a TempDir under `_extensions/`, gated by early-return skip with an +`eprintln!("SKIP: …")`. Julia rows gate on `deno_available() && +julia_available()`; a skip on a machine with both is a signal, not a pass. +Manual 4B–4E renders use the committed fixture root directly; automated tests +always go through the TempDir copy. + +> **Round-3 corrections (2026-07-02, pre-implementation — no row was green +> yet, so the freeze is unviolated):** J2 pinned to *document-level* +> `execute: echo: false` (a cell-level `#| echo: false` travels inside the +> cell source and would NOT redden the P1.1b revert); J4's revert hunk +> relocated to the `FromEngine::Error` arm in `TsEngineHost::request` +> (`ts_process.rs:~693`) — `TsEngine::execute` never sees an error frame; +> J9's refs corrected (`claims_language` static branch `ts_engine.rs:~550`; +> event at end of `resolve_engines`, `resolution.rs:~334`) and both-events- +> present made explicit; J5 refs drifted to `:458`/`:465`; J2/J3's shared +> hunk is 1c.2's now-landed T14 revert — they add Julia-behavior coverage on +> that hunk, not new-hunk coverage. Also: P1.1/P1.1b LANDED, so revert +> phrasing below means "remove the landed population," not "don't build it." + +**Seam-check findings (2026-07-02):** +- **Already bound, no new test:** the `true ≡ Primary(1)` normalization the + 4A item worries about is pinned by `mapLanguageClaim` (`host.ts:168`) with + existing named-revert tests (`host.test.ts:423`, `:489`), and the + static-vs-dynamic mismatch hard-error path has ts_engine unit coverage. If + the first Julia load errors, the cause is elsewhere. +- **Vacuity fix (4I):** "subprocess spawns *after Pass 1*" does NOT + discriminate — a legacy dynamic-claims engine also spawns in Pass 2 + (resolution). Reverting the whole static-claims machinery leaves that + assertion green. The discriminator is **"no spawn during resolution"**: + the spawn event must order after resolution-complete (i.e. at first + execute). J8/J9 below re-anchor 4I to that surface. +- **Zero-load binds without Julia:** J9 uses the echo fixture, so the 4I + property is guarded deno-gated-only; the Julia 4I run is a V-row on top. + +### J-rows (durable automated tests) + +- **J1 — minimal Julia render (4B).** Tier: integration, julia+deno-gated. + Unit: the full chain (discovery → static resolution → load/launch/execute → + jupyter `toMarkdown` → HTML writer) with real Deno + real Julia; no mocks. + Seam: TempDir project with the committed julia fixture; render the 4B doc + (`execute: daemon: false`); assert output HTML contains the cell result `2`. + Revert hunk: delete the `contributes.engines` entry from the fixture + `_extension.yml` → no engine claims `julia` → render error → RED. (Smoke row: + it binds the fixture's registration; deeper properties bind in J2–J7.) +- **J2 — cell options via metadata threading (4E).** Tier: integration, + julia+deno-gated. Unit: 1c.2-P1.1b Rust threading (LANDED, + `metadata: ctx.metadata.clone()` at `ts_engine.rs:396`) + host + `metadataAsFormat` + jupyter `toMarkdown` include logic. Seam: doc with + **document-level frontmatter `execute: echo: false`** (corrected round 3: + NOT cell-level `#| echo: false`, which travels inside the cell source and + survives the named revert); assert the HTML contains the cell's output but + NOT its source listing. Named revert: restore `metadata: HashMap::new()` at + `ts_engine.rs:396` (= 1c.2's T14 revert — J2 adds Julia-behavior coverage + on the shared hunk, and reverting it reddens T14 too) → option dropped → + source listing present → RED. (Discriminator check: assert both halves — + output-present + source-absent — so "render failed entirely" can't fake a + pass.) Cell-level `#|` variants are the 4E manual greps, binding + `toMarkdown`'s cell-option path instead. +- **J3 — exeflags through the julia block (4E).** Tier: integration, + julia+deno-gated. Unit: P1.1b threading of the `julia:` frontmatter subtree + → `format.metadata` → serialized options → QuartoNotebookRunner. Seam: doc + with `julia: exeflags: ["--threads=2"]` — **stop-point: confirm the exact + frontmatter schema against QNR docs before writing this row** — and a cell + printing `Threads.nthreads()`; assert `2` in HTML. Named revert: same + shared T14 hunk as J2 (see note there) → QNR defaults → `1` ≠ `2` → RED. + (`env` gets the same shape only if cheap; otherwise it's V-2.) + **(corrected at implementation, 2026-07-02 — controller-ratified, two changes.)** + As landed (`julia_engine_e2e.rs` J3): (a) the `julia:` block lives in the + fixture's **`_quarto.yml`**, not document frontmatter — frontmatter string + values are smart-typography-mangled (`--threads=2` → en dash; substrate bug + **bd-uf4epv4w**); (b) the named revert is re-anchored to the **project-layer + merge binding, `metadata_merge.rs:214`** — the stop-point investigation + found QNR consumes julia-engine's `target.markdown` (q2's post-merge + serialized AST) as a file-content override, so the T14 wire revert is + structurally undiscriminating for QNR-observable options (verified + empirically twice; T14 stays bound by J2). `env` was folded into J3 with + its own `FOO=BAR` assertion; V-2 not needed. Evidence: 4E task report + + compat log §11. +- **J4 — error handling (4D).** Tier: integration, julia+deno-gated. Unit: + host execute error path (`host.ts` error response) → `FromEngine::Error` → + `ExecutionError` mapping. Seam: render the + `error("this should fail gracefully")` doc; assert (a) render returns an + error whose message contains `this should fail gracefully`, (b) q2 does not + panic, and (c) a subsequent render of the J1 doc through the SAME process + still succeeds (host not wedged — binds pending-request cleanup). Named + revert (corrected round 3): the `FromEngine::Error` arm in + `TsEngineHost::request` (`ts_process.rs:~693`) — `TsEngine::execute` never + sees an error frame; `request()` converts it to `Err` first → error + swallowed/typed differently → (a) RED. +- **J5 — figures + supporting in a website render (4C+4H).** Tier: + integration, julia+deno-gated. Unit: jupyter assets path + `supporting` + forwarding (`map_execute_result`, `ts_engine.rs:~447`) + project artifact + copying. Seam: temp website project (extension copied in by setup — nothing + symlinked, nothing julia-specific committed under `julia-website/`); + `index.qmd` markdown-only + `plot.qmd` with a Plots cell; assert + `_site/index.html` and `_site/plot.html` exist, `_site/plot_files/ + figure-html/*.png` exists on disk, and `plot.html` references it via + ``+file in place binds the assets path but NOT `supporting` — + single-doc output can pass with the forward reverted; the website copy is + the discriminating surface.) +- **J6 — one host per project render (4H).** Tier: integration, + julia+deno-gated (or echo-based if flake-prone — the property is + engine-agnostic). Unit: registry/host construction in + `ProjectContext::discover` shared across Pass-2 files. Seam: J5's project + render with a tracing capture subscriber installed; assert exactly ONE + `engine_host` spawn event (J8's event) across both files. Named revert: + move registry+host construction from the single `discover` call into the + orchestrator's per-file loop → two spawn events → RED. +- **J7 — Julia `launch()` context populated (4H).** NOT a new frozen test — + the binding lives in 1c.2's T1/T2 echo seams (CONTEXT_JSON). The Julia leg + is V-3 (observe the LaunchEngine payload via `RUST_LOG`/tracing during the + J5 render and record `projectDir`/`outputDir` non-empty). Do not duplicate + the T1 binding with a Julia-gated copy. +- **J8 — spawn observability event (shared seam for J6/J9).** Tier: Rust + unit (mock-init transport, existing `ensure_started` double-checked-spawn + tests). Unit: net-new production line `tracing::info!(target: + "engine_host", pid, "engine-host spawned")` in `ensure_started_inner` + (beside the `#[cfg(test)]` counter — the counters are NOT visible to + integration tests, which compile without `cfg(test)`; `is_alive()` is + production but can't count). Seam: capture subscriber; assert exactly one + event per real spawn including under the concurrent-spawn Barrier test. + Named revert: remove the tracing line → J6/J9 captures see zero events → + RED (and this unit row RED). +- **J9 — zero-load resolution ordering (4I, echo-based, deno-gated only).** + Tier: integration. Unit: the static-claims early-answer branch in + `claims_language` (`ts_engine.rs:~550` — corrected round 3) + lazy spawn. + Seam: temp project with the echo (static-claims) fixture and one + `{echo}`-cell doc; tracing capture; assert **both events are present** + (a missing resolution-complete event must FAIL the test, not vacuously + pass the ordering check) and the `engine_host` spawn event orders AFTER + the resolution-complete event (net-new INFO event at the end of + `resolve_engines`, `resolution.rs:~334` — same TDD note as J8) and that + exactly one spawn occurs. Named revert: remove the static early-answer + branch (fall through to the dynamic wire call) → spawn precedes + resolution-complete → RED. This replaces 4I's vacuous "after Pass 1" + surface; the Julia multi-page run is recorded as V-4 evidence. + +### V-rows (manual validations — record invocation + output snippet) + +- **V-1 — daemon mode (4E).** Uncertain Julia-side semantics (does oneShot + avoid the detached server entirely, or only close the file worker?) make a + frozen assertion premature. Record: transport-file presence after a + `daemon: false` render vs a `daemon: true` render; second-`daemon: true`-render + reuse evidence; out-of-band teardown. If the observations are stable, + promote to a J-row in a follow-up (new seam entry required — this spec is + frozen, additions only). +- **V-2 — `env` option (4E)** if not folded into J3. +- **V-3 — launch-context payload for Julia (4H)** — see J7. +- **V-4 — Julia multi-page Pass-1/ordering run (4I)** — evidence on top of J9. +- **V-5 — state persistence across cells (4D).** QuartoNotebookRunner-internal + behavior (cells execute in one `run` request); no q2 hunk to bind. Validate + and record only. +- **V-6 — Q1 output comparison (4F).** Inherently manual. +- **V-7 — Julia through `q2 preview` (4J; added 2026-07-02, user-requested — + additive per the freeze rule).** First real-engine validation of 1c-R5's + native capture → splice preview path (echo/P2-14 is the frozen binding for + the registry-read hunks; V-7 is evidence, not a regression guard). Record: + initial capture executes (output `2`), live re-execute on edit (`5`), + daemon behavior under an interactive session with `daemon: false` + (transport-file state; note the daemon-true-by-default hazard, + bd-m1jeqhhz), cleanup verified. + +### Accepted-untested (logged, not silently omitted) + +- **Import-map parity build (4A):** the discriminating act is the rebundle + itself (`q2 build-ts-extension` fails if the aliases are missing) — but it + is network-dependent (jsr fetch), so no committed automated test. Rationale: + one-time dev-machine step; the committed bundle is the artifact under test + thereafter. +- **`htmlDependency` dedup into `site_libs` (4H):** conditional on Julia + emitting an HTML dependency, which the default Plots/gr backend does not do + deterministically. Left as an if-observed manual check; the dedup mechanism + itself is upstream `store_html_dependencies` behavior with its own coverage. +- **Daemon-mode behavior (4E):** see V-1 — deliberately not frozen until the + Julia-side semantics are observed. +- **`_site/site_libs/` shared assets and sidebar/navbar transforms (4H):** + generic website-epic behaviors with their own coverage upstream; the Julia + render observes them (recorded), but no Julia-gated row duplicates that + binding. ## Design Notes ### Debugging approach -The subprocess architecture helps debugging — you can run the Deno engine-host independently: +The subprocess architecture helps debugging — you can run the Deno engine-host independently. (Corrected 2026-07-02; the original snippet had the wrong entry point and message shape.) The entry point is `src/main.ts` (guarded by `import.meta.main` — `src/host.ts` is the platform-neutral dispatch loop and does nothing when run directly), or the production esbuild bundle `dist/engine-host-deno.js`. Every frame is a newline-delimited `{id, msg}` envelope; `init` carries `global` (a `HostGlobalConfig`), and the engine path goes in a separate `loadEngine` frame: ```bash # Run engine-host manually for debugging -echo '{"type":"init","enginePath":"./julia-engine.ts","context":{...}}' | \ - deno run --allow-all ts-packages/quarto-engine-host-deno/src/host.ts +printf '%s\n%s\n' \ + '{"id":1,"msg":{"type":"init","global":{"resourceDir":"...","runtimeDir":"...","dataDir":"...","isInteractiveSession":false,"runningInCi":false,"quartoVersion":"0.0.0"}}}' \ + '{"id":2,"msg":{"type":"loadEngine","enginePath":"./_extensions/julia-engine/julia-engine.js"}}' | \ + deno run --allow-all ts-packages/quarto-engine-host-deno/src/main.ts ``` -You can also add `console.error()` statements in the engine or harness and see them on stderr. +You can also add `console.error()` statements in the engine or harness and see them on stderr (the Rust host forwards child stderr to `tracing` target `engine_host`). ### Standard library imports -The Julia engine imports `"path"`, `"fs/exists"`, `"encoding/base64"` from Deno's standard library. Following Quarto 1's approach, these are resolved at **build time** via the import map (`"path"` → `jsr:@std/path`, etc.) and inlined into the bundled `.js` file. At runtime, no import resolution is needed. +The Julia engine imports `"path"`, `"fs/exists"`, `"encoding/base64"` from Deno's standard library. Following Quarto 1's approach, these are resolved at **build time** via the import map and inlined into the bundled `.js` file. At runtime, no import resolution is needed (the engine-host runs bundles with `deno run` and no `--config`). + +Q1 ships these aliases in `src/resources/extension-build/import-map.json` (`path` → `jsr:@std/path@1.0.8`, `fs/` → `jsr:/@std/fs@1.0.16/`, `encoding/` → `jsr:/@std/encoding@1.0.9/`, plus `path/posix` and `log`). q2's `resources/extension-build/deno.json` currently maps only `@quarto/*` and the `@std/` prefix — restoring the Q1 aliases is a Phase 4A work item, so the engine source bundles unchanged. -The build step for the Julia engine fixture: +The build step for the Julia engine fixture is `q2 build-ts-extension` (which shells out to `deno bundle` with the 4-tier config precedence: `--config` > extension-local `deno.json` > `deno.workspace.json` > shipped `deno.json`): ```bash -deno bundle --config=resources/extension-build/deno.json julia-engine.ts > julia-engine.js +cargo run --bin q2 -- build-ts-extension src/julia-engine.ts ``` ### CI gating -Julia engine tests should be: -- Gated behind a feature flag or test tag (Julia may not be installed in CI) -- Run manually during development -- Optionally run in CI if Julia is available +Julia engine tests use the same mechanism as the echo E2E suite (decided with +the seam spec — no feature flag or nextest tag): a **runtime probe with an +early-return skip**, `deno_available() && julia_available()`, printing +`eprintln!("SKIP: …")` so the skip is visible in test output. On a machine +with both installed they run; a skip there is a signal, not a pass. They run +manually during development and automatically anywhere Julia+Deno are +present. + +Note the real environmental prerequisite is more than `julia` in PATH: on +first run `ensure_environment.jl` instantiates the QuartoNotebookRunner +project (network + package downloads), and Phase 4C's `using Plots` is a +heavyweight install. Budget for a slow, network-dependent first render, and +don't let the 10s server-ready timeout / 15-try transport-file poll in +`julia-engine.ts` masquerade as a q2 bug when it's a cold Julia environment. ## Success Criteria -- [ ] Julia engine extension discovered and loaded by q2 -- [ ] Simple Julia code cell executes and produces correct output -- [ ] Figure generation works -- [ ] Multiple cells with shared state work -- [ ] Error handling produces useful messages -- [ ] All modifications to julia-engine.ts documented -- [ ] No regressions in existing tests -- [ ] `cargo xtask verify` passes +- [x] Julia engine extension discovered and loaded by q2 +- [x] Simple Julia code cell executes and produces correct output +- [x] Figure generation works +- [x] Multiple cells with shared state work +- [x] Error handling produces useful messages +- [x] All modifications to julia-engine.ts documented +- [x] Website-project integration: a multi-page project with both + markdown and Julia pages renders to `_site/`, with Julia figures in + per-page directories and any Julia HTML dependencies deduped under + `site_libs/` +- [x] Zero-load resolution: Deno subprocess is not spawned during Pass 1 **or + during Pass-2 resolution** — spawn orders after resolution-complete, at the + first execute (J9; reworded round 3 — the old "during Pass 1" phrasing was + the vacuous surface the seam check retired) +- [x] Julia's `launch()` receives a populated per-render `EngineProjectContext` + (`project_dir`/`output_dir` non-empty), not `default()` — the outcome of Plan 1c.2 P1.1, + validated here +- [x] Frontmatter `execute:`/`julia:` options demonstrably reach the engine — the + outcome of Plan 1c.2 P1.1b, validated by 4E (J2 `echo: false`, J3 `exeflags` + observable in cell output; daemon-mode *behavior* is V-1 evidence, recorded + not asserted — its Julia-side semantics are the open observation) +- [x] `Arc` is shared across all files in a project + render (one Deno PID across N pages) +- [x] No regressions in existing tests +- [x] `cargo xtask verify` passes diff --git a/claude-notes/plans/2026-04-16-plan1a-engine.md b/claude-notes/plans/2026-04-16-plan1a-engine.md new file mode 100644 index 000000000..ba62a4120 --- /dev/null +++ b/claude-notes/plans/2026-04-16-plan1a-engine.md @@ -0,0 +1,1408 @@ +# Plan 1a (engine): TsEngine and ExecutionEngine trait extensions + +**Grand plan:** [2026-04-16-ts-engine-extensions-subprocess.md](2026-04-16-ts-engine-extensions-subprocess.md) +**Companion plans:** [plan1a-protocol](2026-04-16-plan1a-protocol.md) (data types), [plan1a-host](2026-04-16-plan1a-host.md) (subprocess + transport) +**Depends on:** plan1a-protocol (uses Ts* types), plan1a-host (uses `TsEngineHost` API) +**Soft-depends on:** Plan 1b (Deno harness) — a **runtime-only** contract: the +`discovery` `OnceLock` benign-race correctness relies on the harness handling +repeat `LoadEngine` idempotently (see "Race-free init"). Plan 1a is implemented +and unit-tested against `MockTransport` **without** Plan 1b; end-to-end coverage +of the composition lands in Plan 1c's echo-engine test. +**Blocks:** Plan 1c (constructs `TsEngine`, calls trait methods) +**Estimated sessions:** 1 + +## Overview + +Extend the `ExecutionEngine` trait with discovery and file-conversion +methods, add the `LanguageClaim` enum and the `resolve_engines` resolver +(including the AST scan that enumerates the document's computational +languages), relocate `HtmlDependency` for q2-native consumers, and create the +`TsEngine` struct that bridges the (synchronous) trait to the protocol + +subprocess. Includes the two-step lazy lifecycle, hint-based pre-filter, alias +map, race-free init via harness idempotency, and the `MockTransport`-driven +test suite. + +**This plan is the engine-side of the multi-engine resolution model.** Since +the April draft, sequential multi-engine execution, capture/replay, and the +discovery cache landed on `main` (bd-5yff4 / bd-45yw / bd-c5u2g). The trait's +claim surface and the resolver here feed that machinery; the cross-cutting +model — kinds/tiers, per-language ownership, `handled_languages` enforcement, +replay-from-captures — is specified once in +`claude-notes/designs/engine-resolution.md` and referenced throughout this +plan rather than re-derived. + +## Drift notes (verified 2026-06-24, before execution start) + +The plan text predates a few changes that landed with plan1a-host. None are +blockers; adapt as you go: + +- **`ExecutionError::Timeout { engine, operation }` already exists** (with a + `timeout(..)` constructor) — plan1a-host added it. The Phase 3 "add `Timeout`" + item is **already done**; only `NotSupported(&'static str)` and + `NoHandlerForLanguage { engine, language }` remain to add. (`ProcessCrashed` + also already exists.) +- **`stage::cancellation` is already `pub mod`** (not private as the Phase 4 + prerequisite assumed). `Cancellation` is reachable as + `crate::stage::cancellation::Cancellation`; a `pub use` re-export at the + `stage` level is now only ergonomic, not required to compile. +- **`MockTransport` / `with_transport` shipped a richer split-half API** than + this plan's prose describes. Reality (in `ts_process.rs`, `#[cfg(test)]`): + `TsEngineHost::with_transport(write: Arc, read: Box, ctx)` fed by `MockTransport::pair()` / + `MockTransport::pair_with_handle() -> (write, read, Arc)`. The + write handle exposes `enable_auto_echo()`, `script_response(id, resp)`, + `script_response_delayed(..)`, `signal_eof()`, and `sent_messages() -> + Vec`. All Phase-4 test capabilities the plan needs exist; use these + names. +- **`HtmlDependency` relocation → keep-in-place + add derives** (see the amended + Phase 3 dep item below). + +## Work Items + +### Phase 3: ExecutionEngine trait — discovery + file conversion + +Extend the `ExecutionEngine` trait with discovery and `markdown_for_file`. +**All trait surface uses q2-native types only.** + +Q1's other lifecycle hooks (`filterFormat`, `executeTargetSkipped`, +`postprocess`, `canKeepSource`, `postRender`, `dependencies`, +`partitionedMarkdown`) are intentionally **not** added to the trait. +For most of them, no q2 caller exists, and adding q2-native equivalents +without a real second implementer would calcify the design prematurely. +For `partitionedMarkdown` specifically, q2's pipeline shape replaces the +need: `DocumentProfile` (post-merge, pre-mutation checkpoint) carries +the title/heading/draft data project-scoped features read, and +filter-aware notebook conversion folds into `markdown_for_file`. See +`claude-notes/plans/2026-04-23-ipynb-filters-and-engine-partitioning.md`. + +**Quarto 1 references:** +- `ExecutionEngineDiscovery` in `src/execute/types.ts` — discovery interface +- `ExecutionEngineInstance` in `src/execute/types.ts` — full lifecycle interface + +- [x] **Add `ExecutionError::NotSupported(&'static str)` variant** to + `crates/quarto-core/src/engine/error.rs`. Used by trait method defaults to + signal "this engine doesn't implement X." The constructor `not_supported` + follows the existing pattern. + +- [x] **Add `ExecutionError::Timeout { engine, operation }` variant** to + `crates/quarto-core/src/engine/error.rs` (constructor `timeout`, existing + pattern). plan1a-host's `request` returns a **distinguishable forcible-abort + error** so `TsEngine::execute` can decide whether to poison the instance: a + user-cancel → the existing `ExecutionError::Cancelled`, a per-request + timeout → this new `Timeout`. A normal engine failure stays + `ExecutionFailed`. `execute` poisons **only** on `Cancelled | Timeout` + (the forcible aborts that can leave the daemon ambiguous), never on + `ExecutionFailed` — see the `execute` bullet. + +- [x] **Add `ExecutionError::NoHandlerForLanguage { engine, language }` + variant** to `crates/quarto-core/src/engine/error.rs` (constructor + `no_handler_for_language`, existing pattern). The §10-case-4 loud failure: a + resolved owner is handed a language it owns but cannot run (e.g. jupyter + + `{sql}`). It is a **clean refusal, not a forcible abort** — so it is **not** + in the poison match (`execute` poisons only on `Cancelled | Timeout`; this + falls through to no-poison by exclusion, like `ExecutionFailed`). See the + "Loud failure" item in Phase 3.5 and design doc §10. + +- [x] Add the **`LanguageClaim` enum** in `engine/mod.rs` (co-located with the + `HANDLED_LANGUAGES` constant below — both are shared, WASM-clean types + consumed by the trait in `traits.rs`, the resolver in `resolution.rs`, and + `ts_engine.rs`; the module root keeps them free of a `traits.rs`↔`resolution.rs` + dependency cycle). This replaces the April `Option` design: the + multi-engine semantics need three distinct *kinds* that don't fit a sign + convention. + See `claude-notes/designs/engine-resolution.md` §3.1 for the full contract. + ```rust + pub enum LanguageClaim { + Primary(i32), // I execute this. (default priority 1) + Interop(i32), // extend my ownership to this iff I'm already present. (default 0) + Fallback(i32), // universal kernel (jupyter's role; declarable by any engine). (default 0) + None, + } + ``` + **Semantics (the resolver in §4 of the design doc consumes these):** `kind` + sets the resolution tier; `priority` orders *only within* a kind (kind + dominates priority — `Primary(-100)` beats `Fallback(100)`); `Interop` is + presence-gated (fires only for an engine already in the sequence via a + positive claim — "extend if I'm already here," not "claim anywhere"); + `Fallback` is the universal-kernel role, no longer hardcoded to jupyter. + +- [x] Add **discovery methods** to `ExecutionEngine` trait with defaults: + ```rust + fn valid_extensions(&self) -> Vec { Vec::new() } + fn claims_language(&self, _language: &str, _first_class: Option<&str>) -> LanguageClaim { LanguageClaim::None } + fn claims_file(&self, _file: &str, _ext: &str) -> bool { false } + // FORWARD-NOTE (Plan 1c / D2): also add + // fn quarto_required(&self) -> Option<&str> { None } + // here — the engine's `quartoRequired` version constraint (Q1 + // `types.ts:65`), enforced at registry-build / first load against the + // spoofed compat version. See plan1c "Enforce engine version requirements". + ``` + **All new trait methods ship with a default body, so no existing + `ExecutionEngine` impl is forced to change and there is no compile + cascade.** There is no pre-existing `claims_language` to "preserve" — this + is new surface. Built-ins override only what they need: knitr/jupyter + override `claims_language`; markdown keeps the `None` default; + `claims_file` / `valid_extensions` / `markdown_for_file` stay on defaults + for all three built-ins (non-QMD support is future work, Plan 1c). + `TsEngine` overrides all four. + +- [x] Add **file conversion method** with q2-native return type: + ```rust + /// Convert a non-QMD file to QMD text. Called only for files this + /// engine claimed via `claims_file`. For QMD files, q2 handles + /// parsing directly and this method is never called. + /// + /// Convert a non-QMD file to QMD text. Returns the converted text; the + /// `SourceInfo` slot is reserved for faithful original-file provenance + /// (deferred — see "Provenance" below) and is `SourceInfo::default()` in + /// v1. + fn markdown_for_file( + &self, + _file: &Path, + _runtime: &Arc, + ) -> Result<(String, SourceInfo), ExecutionError> { + Err(ExecutionError::not_supported("markdown_for_file")) + } + ``` + The `runtime` parameter is the q2-canonical FS abstraction; engines that + need to read the file use it. `TsEngine` ignores `runtime` (the subprocess + reads files via Deno) and returns the harness's + `TsMappedStringWithMap.value` as the converted text. The signature stays + `(file, runtime)` — no `SourceContext` is threaded in (see "Provenance"). + + **Provenance — v1 registers the converted text as an ephemeral intermediate + file (decided 2026-06-24; scope = C′).** Faithful byte-mapping back to the + *original* non-QMD file (so a diagnostic in a converted `.ipynb` cell points + at the source cell) is **deferred** — it has no q2 consumer yet, and the two + faithful mechanisms (see "Future work") are each a real investment. v1 does + the honest, cheap thing instead, mirroring how engine intermediates are + already handled (`engine_execution.rs:423/701` `add_file` ephemeral content + → a real `FileId`): + - The converted text is registered as an **ephemeral intermediate file** + via `SourceContext::add_file(synthetic_name, Some(text))` on the + document's existing context — **the qmd parser already does exactly this** + (`qmd.rs:106`), so for the normal convert-then-parse path the `FileId` is + invented for free and every node gets honest + `Original { file_id, start, end }` provenance **into the converted + buffer**. No `&mut SourceContext` on the trait, no `parent_source_info`, + no transform pass. + - **Synthetic identity reflects the engine that produced it.** Register + under a name that names the converting engine, not the bare original path + — e.g. `"<{original} (converted by {engine})>"` (matching the codebase's + ``/`` synthetic-name idiom). This is deliberate + honesty: the offsets are positions in the *converted* buffer, not the + original bytes, so the identity must not masquerade as the original file + (which would point a reader at wrong line/cols in e.g. the `.ipynb` JSON). + Naming the engine signals "this is a derived buffer." + - **`source_map` stays on the wire, unconsumed.** `TsMappedStringWithMap` + keeps its `source_map`/`file_name` fields (the protocol does **not** + change), but v1 ignores them; they are the input the future A′/B′ + back-mapping will consume. The returned `SourceInfo` is `default()` in v1 + — real provenance comes from the parser's `add_file`, not from this slot. + + **Future work (commendable, not in these plans): faithful original-file + mapping.** When a consumer needs converted-cell → source-cell positions, + prefer **A′ — a generalized remap pass**: parse the converted text (its own + `FileId`), register the original file, then walk the AST rewriting each + `Original { converted_fid, s, e }` into the original-file `SourceInfo` via + `source_map`. This *extends the proven include/engine FileId-remap idiom* + (`include_expansion.rs:199` swaps a `FileId`; A′ generalizes "swap" to + "apply an offset map"). Avoid **B′ — `parent_source_info` / `SourceInfo::Concat`** + (pass the `Concat` as the parse's `parent_source_info` so nodes become + `Substring(Concat, …)`): it rides the **dormant** `parent_source_info` path + (no production caller passes it non-`None` today, `location.rs:215`) *and* + `Concat` resolution requires byte-contiguous pieces (`source_info.rs:418-456`), + which the gappy mappings real conversions produce will violate. A′ is more + code but proven; B′ is less code but unproven + constrained. + + **Not on Rust trait** (harness-internal for TS engines): + - `target()` — q2 constructs execution target data from its AST. TS + engines may implement it for Quarto 1 API compat (transient notebooks, + kernelspec). The harness builds the `ExecutionTarget` from + `TsExecuteOptions` fields when the engine doesn't implement it. + - `dependencies()` — Q1's deferred-deps resolution flow. The harness + folds this into `execute` (see plan1a-protocol Phase 1 protocol notes); q2 receives a + resolved q2-shaped `Vec` on `ExecuteResult`, not the + deferred map. + + > **⚠ Correction — RTQ §FC-2:** the harness no longer folds `dependencies()` into `execute`. `dependencies` is a first-class wire verb driven by q2's render orchestrator at the merged output, with `engineDependencies` carried on the execute result (deferred) when `dependencies:false`. (Text above is the as-built fold RTQ removes.) + +- [x] **Fix the `intermediate_files` doc-comment on the trait.** The + existing `intermediate_files` doc-comment in + `crates/quarto-core/src/engine/traits.rs` currently says the returned + files "may need to be cleaned up after rendering completes" — wrong + framing. `intermediate_files` is a *pure prediction of intermediate + file paths derived from the input path* (NOT post-execution + introspection, NOT a cleanup list): the argument is the original + source path; the return lists paths the engine will produce alongside + the primary output (e.g. a generated `.ipynb`, `.html.md` backups); + the result is used to **exclude those paths from the project's + input-file set** so they are not treated as separate render targets. + Rewrite the doc-comment to match these semantics when this plan is + implemented. + +- [x] Implement on built-in engines (claim tables per + `claude-notes/designs/engine-resolution.md` — jupyter's `Fallback(0)` and + the T4 gate are §4.3; the knitr/markdown rows are the §4.4 worked cases and + the §3 model): + - **JupyterEngine**: `claims_language(..) → LanguageClaim::Fallback(0)` for + every language it is asked about — jupyter is the default universal + fallback (asked only about the doc's actual executable, non-handler + languages; it never enumerates). **Deliberate q2 design choice (new + surface, not a Q1 port):** jupyter does not claim "julia" at priority 1 + the way Quarto 1 did (`claimsLanguage` jupyter.ts:113–117). Under the enum this + falls out for free — jupyter's `Fallback(0)` *loses* to the Julia + extension's `Primary(1)` (kind dominates priority), so the Julia + extension wins cleanly when installed, and `{julia}` without it still + reaches jupyter via the `Fallback` tier (T4). **`claims_file`, + `valid_extensions`, and `markdown_for_file` use the trait defaults for + now** — jupyter does not claim `.ipynb` or percent scripts in the scope + of these plans. Built-in non-QMD support is documented as future work in + Plan 1c (its "Future Work: Built-in engine percent/spin script support" + section); doing it well requires a Rust port of Q1's + `markdownFromJupyterPercentScript` plus an `.ipynb` parser, neither of + which has a current q2 consumer. The trait machinery is shipped here so + the future implementation is a drop-in. ipynb-filter handling, when + implemented, lives inside jupyter's `markdown_for_file` override (see + ipynb-filters research plan). + - **KnitrEngine**: `claims_language("r", _) → Primary(1)`; **`Interop` for + `["python", "sql", "bash", "sh"]`** so knitr *keeps* them when it's + already running R but *cedes* them to a dedicated engine when one is + present (its `Primary` out-ranks knitr's `Interop`). This set is the + knitr `knit_engines` capability — the languages knitr actually executes + in-session — not a guess: `python` via `eng_python`/reticulate, `sql` via + `eng_sql`/DBI, `bash`/`sh` via the shell engines. `sql` is **pinned, not + deferred**: Q1 ships dedicated support for knitr-executed SQL — + `knitr-fixup.lua:4-12` repairs the `knitsql-table` div `eng_sql` emits and + `_quarto-rules.scss:385` styles `.knitsql-table` — so `{sql}`-in-knitr is a + supported path, not a maybe. (Optional future extension if `Interop` means + raw `knit_engines` capability rather than verified-output handling: + `awk`/`ruby`/`perl`/`stan`. `python` remains the load-bearing case.) + **Deliberate q2 design choice (not a Q1 port):** Q1 knitr's + `claimsLanguage` claims *only* `"r"` + (`external-sources/quarto-cli/src/execute/rmd.ts:77-79`) — the other + languages are a knitr-package *execution-time* capability (`knit_engines`), + never a claim-layer claim. q2 does **not** call reticulate itself (zero + references in `quarto-cli/src/`); it lifts knitr's implicit in-session + capability into an explicit `Interop` claim so the multi-engine resolver + (§4) can reason about it and hand e.g. `{python}` to a dedicated engine + when one is present. (Same shape as the jupyter/julia change above — q2 + makes an implicit Q1 behavior explicit at the claim layer.) Note the + distinct axis: knitr's `handled_languages` (`ojs`/`mermaid`/`dot`) are + *pass-through cell handlers* knitr re-emits, **not** languages it + executes — the opposite of `Interop`. **No `claims_file` / + `valid_extensions` overrides** (same scope decision: spin-script support + is future work). Trait default for `markdown_for_file` for now. + - **MarkdownEngine**: returns `LanguageClaim::None` (claims nothing). + +- [x] **Promote knitr's hardcoded `["ojs", "mermaid", "dot"]` to a shared + constant.** Add `pub const HANDLED_LANGUAGES: &[&str] = &["ojs", "mermaid", + "dot"]` in `crates/quarto-core/src/engine/mod.rs`. The literal currently + appears at **three sites** that must all read from the constant: + `crates/quarto-core/src/engine/knitr/mod.rs:187`, + `crates/quarto-core/src/engine/knitr/types.rs:250`, and the test at + `crates/quarto-core/src/engine/knitr/subprocess.rs:903`. `TsEngine::execute` + reads from the same constant when populating + `TsExecuteOptions.handled_languages`. + + **Semantics: instruction, not documentation.** This list tells the engine + which language blocks to **leave alone** in its output — q2 will handle + them downstream via cell handlers (today: ojs, mermaid, dot — none of + these are real cell handlers in q2 yet, but the protocol contract is + established now so it doesn't change later). Engines take the whole + document and return the whole document, so they need to know which + blocks not to execute. Knitr's R subprocess already follows this + contract; TS engines must follow the same. When q2 grows real cell + handlers, this constant migrates to a registry — single source of + truth in the meantime. + +- [x] **Add `Serialize`/`Deserialize` to `HtmlDependency` and friends *in + place* in `pampa`.** *(Amended 2026-06-24: the original "relocate to + `quarto-core::dependency`" instruction was **impossible** — it would create a + dependency cycle. `quarto-core` depends on `pampa`, never the reverse, and + `pampa` itself **constructs and uses** these types: `quarto_doc.rs` builds + them in `extract_html_dependencies`/`extract_text_includes`, and + `unified_filter.rs`, `lua/shortcode.rs`, `lua/filter.rs` all hold + `Vec`/`Vec`. Moving the definitions out of + `pampa` would force `pampa → quarto-core`. The relocation's stated motivation + — "don't force `quarto-core` to depend on `pampa::lua`" — is already moot: + `quarto-core/src/dependency.rs:16` already imports `pampa::lua::{HtmlDependency, + IncludeLocation, TextInclude}`, and **no crate outside `pampa`/`quarto-core` + names these types**. So we keep them where they are and reference them via the + existing re-export.)* + The types live in `crates/pampa/src/lua/quarto_doc.rs` and are re-exported + (un-gated, WASM included) from `pampa/src/lua/mod.rs:36-38`. **Leave them + there.** `ExecuteResult.html_dependencies: Vec` references the + type through the existing `pampa::lua` re-export (the import + `quarto-core/src/dependency.rs` already uses; `engine/context.rs` imports the + same). + **`HtmlDependency`, `TextInclude`, *and* `IncludeLocation` must gain + `Serialize` / `Deserialize` derives (added in `quarto_doc.rs`).** (`TextInclude` + contains `IncludeLocation`, so the enum needs the derives too or + `TextInclude`'s won't compile; today all three derive only `Debug, Clone` — + `IncludeLocation` also `PartialEq, Eq`.) `pampa` already has `serde` with the + `derive` feature (`Cargo.toml:58`) and `serde_json` (`:59`), so the derives are + trivially available. `ExecuteResult` is already `Serialize`/`Deserialize` on + `main` (captured as a `serde_json::Value` inside `EngineCapture` for the + trace/replay path, bd-45yw); a new `html_dependencies` field that doesn't + round-trip would break capture serialization. Add the derives in the same + commit that adds the `ExecuteResult` field. + +- [x] **Add `html_dependencies: Vec` to `ExecuteResult`** in + `crates/quarto-core/src/engine/context.rs`. `EngineExecutionStage` calls + `crate::dependency::store_html_dependencies` on this field after each + execute, in addition to extending `ctx.includes` from `result.includes`. + **Note on the current `ExecuteResult` shape (`main`):** the struct already + derives `Serialize`/`Deserialize`/`Default`, the supporting-files field is + named `supporting_files` (not Q1's `supporting`) and carries + project-resource semantics (bd-o8pr: drained from `StageContext.resource_report` + and copied into the output dir), and there is no `metadata` field. The new + `html_dependencies` accumulation must run **inside the multi-engine loop** + (one `store_html_dependencies` call per engine), alongside the existing + per-engine `includes` / `supporting_files` accumulation. + + **The two channels are disjoint** (see plan1a-protocol's "Two disjoint dep + channels" note). `ExecuteResult.includes` (`PandocIncludes`) carries + pre-rendered HTML/text fragments from Q1-shaped engines (the harness + routes `engine.dependencies(...)` results here); `ExecuteResult.html_dependencies` + carries structured `{ name, stylesheets, scripts }` manifests from + engines that opt into a Q2-native registration API (Plan 1b's + `quarto.htmlDependency()` helper). Engines populate one or both; + q2 routes each to its own sink without dedup logic at the boundary. + + > **⚠ Correction — RTQ §PROTO-2/ENG-3:** `quarto.htmlDependency()` is a per-`Execute` closure-local **value-constructor** whose output is **returned** on the execute result's `html_dependencies` field — not a shared/cross-render "registration API." (Wording correction; the channel is as-built.) + +- [x] **Dedup `HtmlDependency` by `name` in `store_html_dependencies`.** + + > **⚠ Correction — RTQ §ENG-2:** the as-built guard is name-keyed first-wins **plus a content-equality check** — identical re-registration is skipped **silently** (no warning); only **differing** content under the same `name` drops + warns. The "always warns on duplicate" framing below is imprecise (the as-built behavior is correct; ENG-2 fixes the docs + adds the silent-arm test). + + This is a **different** dedup from the one q2 already does, and the two + must not be conflated. `store_html_dependencies` stores under + `ArtifactScope::Project`, which dedupes the **same** artifact shared across + pages (cross-page sharing). It does **not** guard against two engines + registering **different** content under the **same** `name` — those both + write to `libs/{name}/…` and the second clobbers the first. Add a + name-collision guard: key on `name` only, **first-wins**, drop the later + registration entirely (matching Q1's unit-of-dedup at + `external-sources/quarto-cli/src/command/render/pandoc-dependencies-html.ts:228-237`, + which `continue`s past a later dependency whose `name` already appears). + **Improve on Q1:** Q1's drop is *silent*; q2 pushes a + `DiagnosticMessage::warning` naming both registrants. The dedup happens at + storage time (q2's artifact-store-as-canonical-sink), unlike Q1 which + dedupes at injection time. Document the two dedups (project-scope vs. + name-collision) in the function's doc-comment, and cover the name collision + with a regression test (two engines emit `{ name: "jquery" }` with different + content → first wins, one warning). + + **Deferred q2-native fields:** `preserve` (HTML preservation / + postprocess) and `pandoc` (format-affecting options) are NOT added to + `ExecuteResult` in this plan. They have no q2 consumer (no postprocess + stage; format mutation is upstream of execute). When q2 grows the + consumers, the harness will translate from Q1's deferred shape into + q2-native fields. See `claude-notes/plans/2026-04-18-html-js-deps-design.md` + for the broader JS-deps story. + +- [x] Write tests for built-in engine claiming: knitr's + `claims_language("r", _)` returns `Primary(1)` and `claims_language(L, _)` + returns `Interop(_)` for each of `L ∈ {"python", "sql", "bash", "sh"}` + (and `None` for an unclaimed language like `"julia"`); jupyter's + `claims_language` returns `Fallback(0)` for + all inputs (including "julia" — verifying it loses to a `Primary(1)` julia + claim but wins over `None`); markdown returns `None` for everything. No + `claims_file` tests for built-ins — they use the trait default (returns + `false`), which is checked once via the trait-default test below. +- [x] Write tests for default `markdown_for_file` returning `NotSupported`. + +### Phase 3.5: Engine resolution (`resolve_engines`) + ownership enforcement + +The pure resolver that turns claims into an ordered sequence + a per-language +ownership map, and the execute-time enforcement that makes engines cede cells +they don't own. Full model: `claude-notes/designs/engine-resolution.md` +(§4 tiers, §5 enforcement, §9 artifact). + +- [x] **Create `crates/quarto-core/src/engine/resolution.rs`** with the pure + resolver and its artifact: + ```rust + pub struct EngineResolution { + pub sequence: Vec, // ordered, distinct owners + pub ownership: HashMap, // language -> owning engine name + } + impl EngineResolution { + /// HANDLED_LANGUAGES ∪ { lang : ownership[lang] != engine } + pub fn handled_languages_for(&self, engine: &str) -> Vec; + } + pub fn resolve_engines( + meta: &ConfigValue, ast: &Pandoc, registry: &EngineRegistry, claimed: Option<&str>, + ) -> EngineResolution; + ``` + The four tiers (Primary → explicit-Fallback → Interop → implicit-Fallback), + presence-gating, kind-dominates-priority, the implicit-only gate on T4, and + the per-language ownership rule all live here. `claimed` is the file-claim + engine. (**Superseded by plan1c §8 revert, 2026-06-28:** the original + `Primary`-seed semantics — seed the claimer as a synthetic `Primary` and + re-run the tiers — were reverted to Q1-faithful **single-engine**: when + `claimed = Some(engine)`, `resolve_engines` **short-circuits the tiers** and + returns that one engine. plan1c deletes the `explicit_with_seed` logic this + landed code carries; see engine-resolution.md §8.) The result is a pure + function of `(meta, ast, registry, claimed)` — no I/O — which is what makes + a Pass-1 lift possible at all. **Landed shape (Plan 6):** the lift is + **per-doc and load-free-only**, not a blanket zero-cost move for every + doc — a doc's resolution is stamped on `DocumentProfile` only when it + provably needs no engine load to compute (the P1–P4 predicate); a doc + falls through to Pass-2 as soon as one registered engine is both + claims-less and untabled and could contend for one of the doc's languages, + even if every *other* engine in the registry is fully static. See + [Plan 6](2026-06-29-plan6-pass1-engine-resolution.md). + + **`DetectedEngine.config` provenance (the resolver does more than name + engines).** `sequence` is `Vec` and `DetectedEngine` is + `{ name, config: Option }` (`detection.rs:38-47`); the stage + threads `detected.config` into each engine's `ExecutionContext` via + `with_engine_config`. The tiers resolve *names*, so `resolve_engines` must + also attach config: it reads the explicit `engine:` block out of `meta` and + **attaches each listed engine's config to the matching resolved owner**; + **claim-derived owners** (e.g. jupyter reached via T4, or an `Interop` + extension) get `config: None`. State this so an implementer doesn't ship a + resolver that returns name-only entries and silently drops user + `engine:`-supplied config. + +- [x] **Enumerate the document's computational languages from the AST** — + the `languages` input the tiers consume (design doc §4.1/§4.2). **This scan + does not exist today:** `detect_engine_sequence` is metadata-only, and + `engine/detection.rs` explicitly lists "Code block languages (`{python}` → + jupyter)" as a *Future Enhancement*. Add it now, in `resolution.rs`, as a + pure helper feeding `resolve_engines`: + ```rust + /// Ordered, de-duplicated computational languages of the document, each + /// paired with the cell's first non-language class (`first_class`, §4.2). + /// Mirrors Q1's `languagesWithClasses(markdown)` (engine.ts:174) — the + /// first occurrence of a language wins its `first_class`. + fn computational_languages(ast: &Pandoc) -> Vec<(String, Option)>; + ``` + Rules (design doc §4.1, "What counts as a computational language"): + - **Executable cells only** — a braced `{lang}` fence. Reuse the existing + per-block primitive + `engine::capture_splice::engine_cell_lang(&Block) -> Option<&str>` + (it matches the brace-wrapped class pampa preserves; plain ` ```r ` + highlight fences have no braces and are skipped). + - **Recurse** into container blocks (Divs, `BlockQuote`, list items, etc.) — + cells can be nested, so the per-block primitive must be driven by a full + block walk. **No shared block-walker exists to reuse** — the tree has only + ad-hoc local walkers (e.g. `engine_execution.rs:1003`'s + `fn walk_block(b, out)` collecting `FileId`s, the closest structural + precedent). Hand-roll a small private recursion in `resolution.rs` + mirroring that idiom; do not build a general visitor. + - **Exclude `HANDLED_LANGUAGES`** (`ojs`/`mermaid`/`dot` — cell handlers, + not engines). Raw-attribute fences (`` ```{=html} ``) need no handling + here: pampa parses them as `RawBlock` (`fenced_code_block.rs:74-87` routes + a raw format to `Block::RawBlock`), and `engine_cell_lang` matches only + `CodeBlock`, so it never returns them. `HANDLED_LANGUAGES` is the only + thing the scan filters. (Do **not** add a "tokens starting with `=`" + filter — there are no such tokens at this point; an earlier draft assumed + `engine_cell_lang` returns `=fmt`, which is false.) + - **`first_class`** is the cell's first class *after* the language token + (e.g. `{python .marimo}` → language `python`, first_class `marimo`), + read from the `CodeBlock` attr class list. It sharpens *selection* but + not ownership (§4.2), and is passed straight to `claims_language`. + - **Empty set → no engine → markdown passthrough** (§4.1). + `resolve_engines` calls this internally; `EngineExecutionStage` passes the + AST it already holds. Update `detection.rs`'s "Future Enhancements" comment + (the future has arrived) — though the explicit `engine:`-key path in + `detection.rs` stays metadata-only; this is the *language* axis, not the + declared-engine axis. +- [x] **`EngineExecutionStage` calls `resolve_engines` once** at the top of + `run`, stashes `EngineResolution` on `StageContext` (mirroring + `project_index` in `run_pipeline`), reads `ownership` to build each + engine's `handled_languages` via `handled_languages_for`, and the trace + records `sequence`. This is a function + `StageContext` artifact, **not** a + new pipeline stage (it transforms no `PipelineData`). +- [x] **jupyter execute-time `handled_languages` enforcement.** jupyter's + *claiming* is already correct (`Fallback(0)`), but it has **no** + `handled_languages` consumption today and runs every cell it's handed. Add + an execute-time gate: jupyter skips / re-emits verbatim any cell whose + language is in its leave-alone set. Required when jupyter is **non-terminal** + in a sequence (e.g. explicit `[jupyter, knitr]`); as the terminal/fallback + engine it owns the remainder and never cedes. knitr already enforces via + `knit_engines` (the population just changes from the static + `[ojs,mermaid,dot]` constant to the ownership projection); TS engines honor + the contract via `TsExecuteOptions.handled_languages`. See design doc §5. +- [x] **Loud failure when an owner can't execute a language it owns** (design + doc §10 case 4; scope expansion blessed 2026-06-24 — "adapting the existing + engines to the TsEngine/Quarto-API contract is part of the work"). The + four-tier model can hand an engine a language it has no handler for: e.g. + `engine: [knitr, jupyter]` with `{sql}` routes `sql` to jupyter via + explicit-`Fallback` (T2 > knitr's `Interop`, §4.4), but jupyter has no SQL + kernel — whereas knitr's `eng_sql` does. The owning engine MUST fail with a + clear `ExecutionError` naming **engine + language** ("engine `jupyter` has no + kernel for `sql`"), **not** silently skip the cell or emit it unexecuted. + - **GATE (plan1c, 2026-06-28): case 4 fires only for `|sequence| > 1`.** As + landed, `partition_cells` raises `NoHandlerForLanguage` for *any* + owned-but-unrunnable cell. plan1c gates the loud branch on **multi-engine**: + a **single-engine** sequence (a claimed file, §8, or a `.qmd` resolving to + one engine) is handed the whole document and **passes through** what it + can't run — full Q1 parity (Q1 is always single-engine and never errors on a + non-kernel cell; `quartoMdToJupyter` makes it display-only). The loud failure + here is the deliberate q2 *multi-engine* divergence. See engine-resolution.md + §8/§10 and plan1c's failure-model item. + - **This is an execute-time failure by design, NOT a pre-execute capability + probe (decided 2026-06-24).** Resolution stays capability-blind so engine + *selection* is a deterministic, environment-independent pure function — + which is what lets it lift to Pass-1 / `DocumentProfile`. An eager "can + jupyter run sql?" check would make which-engine-is-chosen depend on the + installed kernels. So we accept that `[knitr, jupyter]`+`{sql}` runs knitr's + `{r}` cells first and *then* halts at the `{sql}` cell (partial work before + the loud halt) — the trade for deterministic selection. Design doc §10. + - For **jupyter**: when it owns a language (not in its leave-alone set) for + which its (single, per-doc) kernel can't run, error rather than no-op. + Reuse/extend the existing kernel-not-found path (§10 case 3); a distinct + message that names the *language* (not just the kernel) is the improvement. + Add a new `ExecutionError::NoHandlerForLanguage { engine, language }` + variant (constructor `no_handler_for_language`, existing pattern). + - **`NoHandlerForLanguage` does NOT poison the instance.** It is a clean + refusal — the engine never started computing — so it behaves like + `ExecutionFailed`, not a forcible abort. `execute` poisons **only** on + `Cancelled | Timeout` (the existing match), so this new variant correctly + falls through to no-poison by exclusion; do **not** add it to the poison + match. + - This closes the only silent-failure hazard the broadened knitr `Interop` + set (`[python, sql, bash, sh]`) introduces; the common knitr-only + `{r}`+`{sql}` case still resolves `sql → knitr` (Interop wins with no + explicit fallback present). +- [x] **Unit-test the resolver in isolation** with a `MockEngine` registry of + hand-written claim tables (no subprocess, no AST execution): the worked + cases from design doc §4.4 — implicit `{r}`+`{python}` → `[knitr]` + (reticulate); implicit `{r}`+`{sql}` → `[knitr]` (`sql → knitr` via Interop); + explicit `[knitr, jupyter]` → r→knitr, python→jupyter; **explicit + `[knitr, jupyter]` with `{sql}` → `sql → jupyter`** (T2 explicit-`Fallback` + preempts knitr's `Interop` — the routing that triggers the §10-case-4 loud + failure at execute); pure `{python}` → `[jupyter]`; `{julia}` ± extension; + `Fallback` priority ordering beating registration order; `Primary(-100)` + beating jupyter's `Fallback(0)` (kind dominates — the §4.4 table row); T4 + implicit-only (explicit `[knitr]` + `{julia}` does **not** add jupyter — + stated in §4.3 prose). These tests pin the tier logic without any of the + TsEngine subprocess machinery. + +### Phase 4: TsEngine struct + +The Rust struct that implements `ExecutionEngine` by delegating to the shared subprocess. + +> **⚠ Correction — RTQ §ENG-1:** the discovery/instance tier split below is being corrected. `generates_figures` moves off the instance `LaunchEngineResult` onto the discovery `LoadEngineResult`; `can_freeze` is added to `LoadEngineResult` too (kept on the instance — Q1 has both); and `quarto_required: Option` joins the discovery tier (load-time semver gate deferred to grand-plan Phase 12). Target shape: discovery `LoadEngineResult { name, valid_extensions, generates_figures, can_freeze, quarto_required }`; instance `LaunchEngineResult { can_freeze }`. The as-built `{ can_freeze, generates_figures }` instance pair shown below is what RTQ corrects. + +- [x] Create `crates/quarto-core/src/engine/ts_engine.rs`: + ```rust + pub struct TsEngine { + /// The registry key under which this TsEngine was inserted — + /// either the `name` declared in `_extension.yml` (declared + /// path) or the extension id (lazy-alias path). Used for log + /// messages and as the value returned by `name()` until/unless + /// the lazy-alias resolution updates it. + name: String, + /// Whether `name` was declared up-front in `_extension.yml`. + /// When `true`, the first `LoadEngine` validates that + /// `LoadEngineResult.name == self.name` and errors on mismatch. + /// When `false`, the first `LoadEngine` records the runtime + /// name in the registry's alias map. + name_declared: bool, + host: Arc, // Shared subprocess (from EngineRegistry). + // Bundle is embedded in the q2 binary + // via include_str! (plan1a-host's + // "Bundle embedding" design note); + // TsEngine doesn't carry a bundle path. + // Two-step init state machine. + // None: not yet loaded. Some: module loaded, discovery available. + discovery: OnceLock, + // None: not yet launched. Some: instance running, execute/etc. available. + // `Mutex>`, NOT `OnceLock` (which is set-once and can't be + // cleared): an `Execute` cancel/timeout *poisons* the instance + // (plan1a-host's Execute-scoped poison policy). `poison_instance` clears + // this to `None` so the next instance request re-runs `LaunchEngine` + // (~0) and reconnects/restarts the detached daemon. + instance: Mutex>, + // Cache of claims_language(language, first_class) results. + // Sound iff the engine's claimsLanguage is a pure function of its + // inputs — see "Cache determinism contract" in design notes. + claims_language_cache: Mutex), LanguageClaim>>, + // Cache of claims_file(path, ext) results, scoped to one + // project render (the `Arc` lifetime owned by + // `ProjectContext`; see Plan 1c Phase 2). Engines may inspect + // file content (Julia checks for `# %%` percent-script markers); + // without this cache, project scans re-read the same files for + // every (file, engine) pair. + // Cache key is the canonical path; the ext argument is derived + // from the path so it isn't part of the key. Lifetime is one + // project render (q2's pipeline is stateless across renders). + claims_file_cache: Mutex>, + // Static hints from _extension.yml (see Plan 1c). + // None: not declared by the extension author. + // Some(empty): explicit "claims none" — silent, no dynamic call. + // Some(non-empty): pre-filter; only consult subprocess if input matches. + // These are the *pre-filter* form of the static-claim story (design + // doc §3.3): a hint is a conservative superset ("might I claim this?") + // that avoids a load when the language clearly doesn't match, but still + // loads to get the precise claim when it does. The *complete* static + // form is a full `claims:` declaration in _extension.yml (kind + + // priority / fallback) that resolution reads without loading at all — + // dynamic `claims_language` is then the back-compat escape hatch. A + // fully-static engine (declared `name` + `file-extensions` + `claims`) + // is loaded only to *execute*, never to resolve. + language_hints: Option>, + file_extension_hints: Option>, + // FORWARD-NOTE (Plan 1c / D1): the "complete static form" above is now + // in scope. Plan 1c replaces `language_hints` with an authoritative + // `claims: Option>` (kind/priority) and + // adds `claims_files: Option>` (unconditional claims_file), + // keeping `file_extension_hints` as `valid_extensions`. `claims_language` + // / `claims_file` then answer from the static declaration *without* + // loading when it is authoritative (language-only / extension-only), + // use the keys as a pre-filter otherwise, and validate against the + // dynamic method on the first execute-time load. See plan1c Phase 1. + } + ``` + `TsEngine` does NOT own the subprocess — it shares `TsEngineHost` with other + TS engines via `Arc`. The transport `Mutex` is inside `TsEngineHost`, not + `TsEngine`. + + **`Send + Sync` is satisfied at the type level** because `Arc`, + `OnceLock` (discovery), and `Mutex<…>` (the instance slot + and the caches) are all `Send + Sync`. Required by the existing + `Arc` registry contract (`engine/registry.rs`). + + **Concurrent correctness** under the now-live rayon-per-worker parallelism of + Pass-2 is achieved two ways, slot by slot: the `discovery` `OnceLock` leans on + Plan 1b's idempotent harness lifecycle (a benign double-`LoadEngine` race + resolves to one cached result), and the `instance` `Mutex>` + serializes its own init under a short-held lock (cheap — `LaunchEngine` starts + no daemon). See "Race-free init" below. Neither uses Rust-side double-checked + locking. + +- [x] **Two-step lazy lifecycle.** Four internal helpers (not on the trait — + called from inside trait method impls): + - `ensure_loaded(&self, c: &Cancellation) -> Result<&LoadEngineResult>` — + ensures the shared subprocess is running (`host.ensure_started()`), then + calls `host.load_engine(path, c)` if `discovery` is empty (plan1a-host's + higher-level helper over the demux `request` — not a raw `send`/`recv` + pair). Cheap (~10–50ms total). Required before any discovery method. + - `ensure_launched(&self, c: &Cancellation) -> Result` — + calls `ensure_loaded` first, then locks `instance`; if `None`, calls + `host.launch_engine(name, c)` **under the lock** and stores `Some(...)`. + Returns the result **by value** (a `Copy`-cheap `{ can_freeze, + generates_figures }` pair) — a `Mutex>` can't lend a `&` past its + guard. Holding the lock across `launch_engine` is fine because it is + **~0** (`LaunchEngine` only constructs the `ExecutionEngineInstance` object + on the Deno side; it starts no daemon — the expensive Julia/Jupyter startup, + 5+s, happens lazily inside `execute()` on the first call). The short lock + makes init *exclusive* (no double-launch) while keeping the slot clearable + for `poison_instance`. Required before any instance method. + - `poison_instance(&self)` — locks `instance` and `.take()`s it back to + `None`. Called by `execute` when an `Execute` `request` resolves with a + **forcible-abort error** (`Cancelled | Timeout` — *not* a plain + `ExecutionFailed`; plan1a-host's Execute-scoped poison policy): the detached + daemon may be mid-computation, so the next instance request must + re-`LaunchEngine` and reconnect/restart it. `discovery` is never poisoned — + `LoadEngine` engages no daemon, nothing to invalidate. + - `name()` and `is_available()` are local-only — never touch the + subprocess. + Shutdown is on `TsEngineHost`, not per-engine, and is **explicit** — + matching q2's existing convention (e.g., `JupyterDaemon::shutdown_all` + at `crates/quarto-core/src/engine/jupyter/daemon.rs:272-279` is an + explicit method, not a `Drop`). The orchestrator calls + `registry.shutdown_all()` at end-of-render (Plan 1c owns this site) + before `ProjectContext` drops. `registry.shutdown_all()` iterates the + unique `Arc` clones held by `TsEngine` instances and + calls `host.shutdown()?` on each; errors are surfaced through the + caller's `Result`. As a backstop against panic/unexpected drop, the + child process spawned by `StdioTransport` is reaped by an explicit + `Drop` impl (`std::process::Child` has no `kill_on_drop` — that is a + `tokio::process::Command` method; see plan1a-host) so a forgotten explicit + shutdown still kills the subprocess. + +- [x] **Race-free init.** The engine→subprocess path is **fully synchronous**: + the `ExecutionEngine` trait is a sync trait (`fn execute(&self, …) -> Result<…>`), + `TsEngine` calls plan1a-host's **synchronous** `EngineTransport` (blocking + stdio I/O through the host demux — `StdioTransport` over the child's + stdin/stdout in v1; loopback TCP is the deferred Phase 1.6), and there is + **no `async`/`await` + or `block_on`** between the trait method and the wire. (The `PipelineStage` + layer *above* is `?Send` async, but by the time control reaches + `engine.execute` it is a plain blocking call.) **Pass-2 is now parallel** + (rayon-per-worker), so concurrent callers on the same `TsEngine` are live, not + hypothetical. The two init slots handle that differently: + - **`discovery` (`OnceLock`)** — naive (`get()` → `host.load_engine` → + `set()`), no `Mutex<()>` double-checked locking. Two racers both pass + `is_none()`, both issue concurrent `LoadEngine` `request`s (distinct ids, + both in flight), both land at `set()`; the late `set()` fails silently and + both read the same value via `discovery.get().unwrap()`. **Benign because + Plan 1b's harness is idempotent** for repeat `LoadEngine` (cache hit, no + re-`import()`). Cost: one extra round-trip per *racer* — and under parallel + Pass-2 the cold-start racers are the N rayon workers all first needing the + same engine, so a cold `discovery` can fan out to **up to N concurrent + `LoadEngine`s** (not merely 1–2), each a cheap harness cache-hit after the + first. `LoadEngine` engages no daemon, so the fan-out leaks nothing; it + settles to a single cached `LoadEngineResult`. + - **`instance` (`Mutex>`)** — init is **exclusive**: lock, check + `None`, `host.launch_engine` *under the lock*, store `Some`. The second + racer blocks on the lock and finds `Some`, so `LaunchEngine` is issued + **exactly once**. Holding the lock across `launch_engine` is acceptable + precisely because it is ~0 (no daemon start) — and the slot must be a + clearable `Mutex>` anyway for `poison_instance`, so exclusivity is + free. (Harness `LaunchEngine` idempotency still holds as a backstop, but the + lock means we don't lean on it here.) + + Why not `Mutex<()> + OnceLock` double-checked locking for `discovery`? The + closest analog in the q2 tree (`JupyterDaemon` — + `crates/quarto-core/src/engine/jupyter/daemon.rs`) uses a naive + `OnceLock` for the process-global daemon handle plus a per-key check in + the daemon's session map (itself a `tokio::sync::RwLock`) without + init-mutex serialization across the check-then-insert gap. Adopting + double-checked locking here would + introduce a pattern that doesn't exist anywhere else in the codebase. + Idempotent lifecycle on the harness side is the right place to put + the obligation: the harness already maintains a + `Map`, so the work is a few-line + contract addition rather than a new Rust idiom. + + **Test (Plan 1a, Rust-side invariant only).** Race two + `ensure_launched` calls on the same `TsEngine` (two threads, + `Barrier` synchronizing the start) against a `MockTransport` (see + the testing items below). Assert: the `instance` slot ends up `Some` + with a single `LaunchEngineResult`, no panic, and — because instance + init is exclusive under the `Mutex` — the `LaunchEngine` message count + observed by the mock is **exactly 1**. A companion test races two + `ensure_loaded` calls and asserts the `LoadEngine` count is **1 or 2** + (the `discovery` `OnceLock`'s benign double-issue window — never 0, + never > thread count). The end-to-end "engine.launch() invoked exactly + once across the real harness" assertion is **Plan 1b's contract**, tested + against the real harness; Rust+harness composition lives in Plan 1c's + echo-engine integration test. + +- [x] **Add `cancellation: Cancellation` AND `execute_timeout: Option` + to `ExecutionContext`** (the cross-plan dependency plan1a-host's "Cancellation + wiring" and "Per-request timeouts" are blocked on). `cancellation` is the only + way the token reaches the engine: `request`'s timeout/cancel loop polls + `is_cancelled()`, but `execute` receives only `&ExecutionContext`, which today + carries no token. `execute_timeout` is the resolved `Execute` window — + **`EngineExecutionStage` reads `execute.timeout` from `doc_ast.ast.meta`** + (via `get_path(&["execute","timeout"])`, tri-state per plan1a-host) since + `TsEngine` cannot reach the top-level `execute:` block itself; add a + `with_execute_timeout(Option)` builder next to `with_cancellation`. + `TsEngine::execute` passes `ctx.execute_timeout` as the `window`. Other engines + (markdown/knitr/jupyter) ignore both fields — though wiring `execute_timeout` + here means a future jupyter could honor it instead of its hardcoded + `DEFAULT_EXECUTE_TIMEOUT` (`engine/jupyter/execute.rs:24`), out of scope now. + - Populate it in `EngineExecutionStage` from `ctx.cancellation` where the + `ExecutionContext` is built (`crates/quarto-core/src/stage/stages/engine_execution.rs:~310`). + - `TsEngine::execute` passes `&ctx.cancellation` to + `host.request(msg, window, cancellation)`. Discovery methods + (`claims_language`/`claims_file`), which run without a full + `ExecutionContext`, pass a default (non-cancellable) `Cancellation` — the + 10s discovery `window` still bounds them; threading a real token into + discovery is out of scope. + - **Constructor churn:** `ExecutionContext` is built in several places + (`engine_execution`, `fixture`, `replay`, `preview_record`, tests). Add both + fields with defaults (`cancellation: Cancellation::new()`, + `execute_timeout: Some(DEFAULT_EXECUTE_TIMEOUT)`) — exposed via + `with_cancellation` / `with_execute_timeout` builders — so this is **not** a + breaking cascade. `Cancellation` is already cfg-portable (native + WASM — + `stage/cancellation.rs`), so the WASM build is unaffected. + - **Prerequisite: re-export `Cancellation` from `stage`.** Today + `stage/mod.rs:84` declares `mod cancellation;` (**private**), and the type + is reached only via `super::cancellation::Cancellation` *inside* `stage` + (`stage/context.rs:27`). `engine::context` is a sibling module under + `crate`, so naming the type there needs a `pub use cancellation::Cancellation;` + (or `pub mod cancellation;`) added to `stage/mod.rs` — without it, adding + the field is an unresolved-import compile error. One line, but a real + prerequisite of this work item. + - **Prerequisite: promote `DEFAULT_EXECUTE_TIMEOUT` to a shared home** (the + `execute_timeout` default surfaced this when the host plan was rebased in). + It is currently a **private** `const DEFAULT_EXECUTE_TIMEOUT` in + `engine/jupyter/execute.rs:24` (300s), so the `ExecutionContext` default + `Some(DEFAULT_EXECUTE_TIMEOUT)` can't reach it. Move the const to a shared + location (e.g. `engine/mod.rs`, alongside `HANDLED_LANGUAGES`) and have + jupyter re-reference it — making it the single source of truth the host + plan already assumes ("a future jupyter could honor `execute_timeout` + instead of its hardcoded `DEFAULT_EXECUTE_TIMEOUT`"). Same class of + one-line-visibility prerequisite as the `Cancellation` re-export above. + - **Bound test (cross-plan obligation from plan1a-host's Test Seam Spec).** + `EngineExecutionStage` resolves `execute.timeout` from `doc_ast.ast.meta` via + `get_path(&["execute","timeout"])` (tri-state). Add a test on the resolver: + `{execute:{timeout:5}}` → `Some(5s)`; `{execute:{timeout:false}}` → `None`; + absent → `Some(300s)`. **Named revert:** delete the `get_path`/`as_bool`/`as_int` + branch (hardcode `Some(300s)`) → the `5s` and `None` cases go RED. (Vacuity: + the three cases must map to three distinct windows, not all to the default.) + +- [x] Implement `ExecutionEngine` trait — all methods that touch the + subprocess go through `ensure_loaded` or `ensure_launched`: + + **Existing trait methods:** + - `name()` → `self.name` (no subprocess call) + - `is_available()` → check Deno in PATH (no subprocess call). The + bundle is embedded via `include_str!` and always present at runtime + — no file-existence check needed. + - `can_freeze()` → if launched (`self.instance.lock()` holds `Some`), read its + `can_freeze`; if `None` (never launched, or poisoned back to `None`), + conservative `false` (no subprocess call to find out) + - `execute(input, ctx)` → `ensure_launched`, build `TsExecuteOptions` from + `ctx`. Translation: `doc.ast.meta` (`ConfigValue`) → flat + `HashMap` per "ConfigValue → TsMetadataValue" + in plan1a-protocol's appendix; q2's `Format` → `TsFormatIdentifier` (the four identifier + fields the protocol forwards; q2's other `Format` fields stay on + the q2 side); both packed into a single `TsFormatInfo`. `SourceInfo` → + `Vec` per the source-map flattening rules in plan1a-protocol's appendix. + `HANDLED_LANGUAGES` constant → `handled_languages`. Issue the `Execute` + `request` (the long-window call), translate the response back to q2-native + `ExecuteResult` (`html_dependencies` from `TsHtmlDependency[]`, + `includes` from `TsPandocIncludes`, etc.). **Match the `request` result: on + a forcible-abort error (`ExecutionError::Cancelled | ExecutionError::Timeout`) + call `poison_instance()` before returning it; on a plain `ExecutionFailed` + (a normal engine error) do NOT poison** — the instance is still healthy. + plan1a-host's `request` returns these as *distinguishable* errors precisely + so `execute` can make this call. `Execute` is the only request that engages + the daemon, so it is the only one that poisons; `intermediate_files` starts + no daemon and never poisons. + - `intermediate_files(input_path)` → `ensure_launched`, send + `IntermediateFiles`, recv result, translate `Vec` → `Vec`. + **Semantics:** this is a *pure prediction of intermediate file paths + derived from the input path* — NOT post-execution introspection of + what `execute()` produced. The argument is the original source path; + the return lists paths the engine will produce alongside the primary + output (e.g. a generated `.ipynb`, `.html.md` backups). The result is + used to **exclude those paths from the project's input-file set** so + they are not treated as separate render targets. It stays on the + instance tier (needs `LaunchEngine`), faithful to Q1's + `ExecutionEngineInstance.intermediateFiles`; because `LaunchEngine` + is cheap (it starts no daemon), this costs nothing during a project + crawl. + + **Discovery methods (defined in Phase 3):** + - `valid_extensions()` → **hints are the source of truth pre-load.** + Q1's `validExtensions()` is consulted in two dispatch sites that + Plan 1c's dispatcher needs to match: a per-file pre-gate inside + `fileExecutionEngine` (`external-sources/quarto-cli/src/execute/engine.ts:312-318`) + that rejects files whose extension no engine declares, and a + project-wide aggregate (`engine.ts:140-144`) used for project file + discovery. In Q1 these calls are sync, in-process, free; in q2 + they would force a `LoadEngine` round-trip per TS engine just to + answer "do you handle `.qmd`?" The rule is therefore: + - `file_extension_hints == Some(...)` → return the hints directly, + no load. Hints are the authoritative answer pre-load. + - `file_extension_hints == None` → fall back to + `ensure_loaded` and return `discovery.valid_extensions`. Engines + that want fast project-level discovery should declare hints. + Hint-validation at load time (below) catches mismatches between + declared hints and runtime `valid_extensions`. + - `claims_language(language, first_class) -> LanguageClaim` → **static + `claims:` declarations (if present) answer with no load; otherwise hints + pre-filter the dynamic path**: + - A full static `claims:` entry for the language → return it directly + (no load). This is the zero-load path (design doc §3.3). + - `language_hints == Some(empty)` → return `None` (no load — explicit + "claims none"). + - `language_hints == Some(non-empty)` and language not in list → + return `None` (no load — pre-filter rejection). + - `language_hints == Some(non-empty)` and language IS in list → + check `claims_language_cache`; on miss, `ensure_loaded`, send + `ClaimsLanguage`, recv `ClaimsLanguageResult`, cache, return. + - `language_hints == None` (no hints declared) → check + `claims_language_cache`; on miss, `ensure_loaded`, send + `ClaimsLanguage`, recv `ClaimsLanguageResult`, cache, return. + Engines that want to avoid loading on first dispatch should declare + hints; engines that want to avoid loading *entirely* during resolution + should declare full static `claims:`. + The harness normalizes the engine's JS return into the wire claim — **no + sign games**: `false`/`null` → `None`, `true` → `Primary(1)`, `number n` + → `Primary(n)` (negative = low-priority primary, never interop), and the + object form maps to `Primary`/`Interop`/`Fallback` directly (design doc + §3.2). `Interop` and `Fallback` are reachable only via the object. + + **Wire → resolution conversion (mind the shape gap).** The protocol + `TsLanguageClaim` (`ts_protocol.rs:138-144`) has only **three** variants + in struct form — `Primary{priority} | Interop{priority} | Fallback{priority}` + — and **no `None`**: "no claim" is modeled by the *absence* of a claim. So + `ClaimsLanguageResult` carries an **`Option`** (`None` ⇒ + no claim), and `TsEngine` maps it to the resolution-layer `LanguageClaim` + (tuple form, with a `None` variant) via a `From> + for LanguageClaim` conversion living in `ts_engine.rs` at the + protocol→native boundary (next to the `Format`/`SourceInfo` translations): + `None ⇒ LanguageClaim::None`, `Some(Primary{p}) ⇒ Primary(p)`, etc. This + is the only seam where the two near-identically-named types meet; keep the + protocol DTO inside `ts_engine.rs` and hand resolution the native enum. + - `claims_file(file, ext)` → **hints are the source of truth pre-load**: + - `file_extension_hints == Some(empty)` → return `false` (no load — + explicit "claims none"). + - `file_extension_hints == Some(non-empty)` and ext not in list → + return `false` (no load — pre-filter rejection). + - `file_extension_hints == Some(non-empty)` and ext IS in list → + check `claims_file_cache` keyed on the canonical path; on hit, + return cached. On miss: `ensure_loaded`, send `ClaimsFile`, + recv `ClaimsFileResult`, cache, return. + - `file_extension_hints == None` (no hints declared) → check + `claims_file_cache`; on miss: `ensure_loaded`, send `ClaimsFile`, + recv `ClaimsFileResult`, cache, return. Engines that want to + avoid loading on first dispatch should declare hints. + + The cache is scoped to one project render — same lifetime as + the `Arc` owned by `ProjectContext` (Plan 1c + Phase 2) and the `Arc` clones it owns. A project + scan that consults N engines for M files would otherwise pay + N×M file reads (engines like Julia inspect content for + percent-script markers); the cache reduces that to N×M results + but ≤M file reads per engine. The cache assumes the file's + content does not change *during* the render — a reasonable + invariant since q2 reads inputs once at pipeline entry. q2's + render pipeline is currently stateless across renders (each + render builds a fresh `ProjectContext`), so cross-render cache + staleness is not a concern; if a future architecture reuses + `ProjectContext` across renders (e.g., a long-running preview + server), that plan revisits cache invalidation. Engine-content-aware + caching across renders is otherwise out of scope. + + **Concurrent claims caching (parallel Pass-2).** Resolution runs per-document + in Pass 2, and Pass 2 is parallel, so multiple rayon workers call + `claims_language` / `claims_file` on the *same* shared `TsEngine` + concurrently. The `Mutex` caches keep this memory-safe, and the + failure mode mirrors the `discovery` race: two workers can both miss the same + key and both issue the dynamic `ClaimsLanguage` / `ClaimsFile` query (the + cache does **not** dedup *in-flight* misses) — benign, because the determinism + contract below makes both answers identical and the second `set` is an + idempotent overwrite. The lock is held only for the map read/write, never + across the round-trip. Worst case on a cold key under N-way contention: up to + N redundant (idempotent) queries, settling to one cached value. + + **Hint validation at load time:** when `LoadEngine` returns + `discovery.valid_extensions`, validate that any non-empty + `file_extension_hints` superset-contains it. If the engine claims + extensions outside the declared hints, emit a `DiagnosticMessage::warning` + naming the extensions and the engine — the static-hint pre-filter + would silently miss those claims. Same channel for the missing-hints + cost note (TS engines without `file_extension_hints` trigger + `LoadEngine` on every `.qmd` render). All Plan-1a-time engine + warnings/errors flow through `DiagnosticMessage` (q2's standard + user-facing diagnostic channel — `DiagnosticMessage::warning` at + `crates/quarto-error-reporting/src/diagnostic.rs:247`), + not `tracing::warn!`. The registry holds + `pub diagnostics: Mutex>` (see registry-state + block below); `ProjectContext` drains it at end of init and forwards + to the pipeline observer. `tracing::warn!` is reserved for operator + logging. **Push diagnostics idempotently:** hint-validation runs inside + `LoadEngine`, so the benign `discovery` double-issue (parallel Pass-2) can run + it twice for one engine and push the *same* warning twice. Either guard the + push so it fires only on the `OnceLock`-winning load, or de-dup the drained + vec by `(engine, message)` before forwarding — otherwise users see duplicated + warnings under load. + + **`EngineRegistry` struct definition.** plan1a-engine owns the + registry's struct shape because plan1a-engine is what mutates the + fields (alias insertion under `LoadEngine`, diagnostic pushes during + hint validation). Plan 1c constructs an instance with + `EngineRegistry::new()` and populates it; this is the canonical + definition. + ```rust + pub struct EngineRegistry { + engines: HashMap>, // immutable post-construction + aliases: Mutex>, // runtime_name → extension_id, lazily populated + diagnostics: Mutex>, // hint-validation, missing-hints, name-collision warnings + } + ``` + Both mutexes are independent of `TsEngineHost`'s transport mutex. + Following the `JupyterDaemon` pattern (`crates/quarto-core/src/engine/jupyter/daemon.rs`) + — separate locks for separate concerns; no cross-locking. + + **Migration from `main`'s registry (`#[derive(Clone)]` blocker — the + Clone-drop is NOT self-contained; decided 2026-06-24).** On `main`, + `EngineRegistry` is `{ engines: HashMap> }` + and derives `Clone`. `Mutex` is **not** `Clone`, so adding the `aliases` / + `diagnostics` fields **requires dropping `#[derive(Clone)]`** — and dropping + the derive **breaks the build at 8 real sites across 4 files** that clone an + `Option`: + - **6 explicit `Option::clone()` calls:** + `quarto-core/src/pipeline.rs:847`, `quarto-preview/src/lib.rs:200`, `:206`, + `:244`, `quarto-preview/src/capture_driver.rs:109`, and + `quarto-core/src/render_to_file.rs:328`. + - **2 transitive `#[derive(Clone)]` structs holding `Option` + by value:** `PreviewConfig` (`quarto-preview/src/lib.rs:65`) and + `RenderToFileOptions` (`quarto-core/src/render_to_file.rs:83`). + + (The previously-named sites — `EngineExecutionStage::with_registry` + (`engine_execution.rs:113` — a *stage* method, not on the registry), the + per-document construction in `EngineExecutionStage::new`, and + `with_replay_many` — do **not** break *from the `Clone`-drop*: they *move* the + registry in or *build it fresh*, so they never relied on `Clone`. Note + `with_registry`'s *signature* still changes as part of the `Arc`-type + propagation below — "doesn't break from `Clone`" and "signature changes for + `Arc`" are both true and not in tension.) + + **Decision: pull the minimal `Arc`-wrap into Plan 1a** so the registry + change stays independently compilable (the project's green-build-per-plan + discipline). Plan 1a reroutes those clone sites to `Option>` + (a cheap `Arc` clone replaces the deep clone) as part of this work item — + this is mandatory-to-compile, not optional cleanup. Plan 1c then does the + deeper `ProjectContext`-owned ownership (built once, shared across Pass 1 + + Pass 2), building on the `Arc` Plan 1a introduces. + + **Scope of the type change — ~25–30 mechanical sites, zero semantic change + (verified 2026-06-24).** The 8 clone sites are where the build *first* + breaks, but the `Option` → `Option>` + type then propagates transitively through a closed, mechanical set: + - **`EngineExecutionStage`** — field `registry: EngineRegistry` → + `Arc`, plus `with_registry(...)` signature and the + `new()` body (`Arc::new(EngineRegistry::new())`). All reads are through + `&self` methods (`get`/`default_engine`), and `Arc` derefs transparently, + so **no consume site changes behavior**. + - **A third config struct the clone-site list omitted: `HtmlRenderConfig`** + (`pipeline.rs:118`) and its `with_engine_registry` builder + (`pipeline.rs:131`). The `render_to_file.rs:328` clone feeds + `config.engine_registry`, so the `Arc` reaches into `quarto-core`'s + `HtmlRenderConfig` — **name it explicitly so the count isn't a surprise.** + - **The `quarto-preview` pass-through chain** (~9 fn signatures in + `re_execute.rs` / `capture_driver.rs` / `cache.rs` that carry the + registry as an opaque `Option<…>` param) — pure signature type swaps, no + body logic. + - **One real construction site**: `render_to_file.rs:331` + (`Arc::new(EngineRegistry::with_replay_many(...))`); plus test fixtures + that build a registry and pass it in (`Arc::new(...)` at the test + boundary). + + **Nothing mutates the registry post-construction** (no `&mut`/`.register()` + after it enters a config or stage — confirmed), so `Arc`-wrapping is sound + and matches the new "engines immutable post-construction" design. Note too + that **in production the override is `None` everywhere** — the real default + registry is built inside `EngineExecutionStage::new()`; the only `Some` that + flows through config in production is the replay path. So the wrap is mostly + test-fixture + replay churn, all trivial type substitution. + + **`with_replay_many` stays in Plan 1a, untouched.** It builds a fresh + registry and does not depend on `Clone`, so the Clone-drop does not affect + it. Its removal is purely the §6.2 capture-driven-replay rework — replay + drives from recorded `engine_captures` instead of injecting `ReplayEngine`s + into the (now immutable) registry — which is **Plan 1c's** architectural + change, not a consequence of Plan 1a's registry edit. See + `claude-notes/designs/engine-resolution.md` §6.2. + + **Name validation at load time:** when `name_declared` is true, + assert `LoadEngineResult.name == self.name`. Mismatch is a hard + error pointing at the YAML: `Engine extension declares 'name: {self.name}' + in _extension.yml but the loaded module reports 'name: {actual}'. + Update _extension.yml or the engine module's name property.` + When `name_declared` is false, the registry's `aliases` map is + updated with `LoadEngineResult.name → self.name` so subsequent + lookups by runtime name resolve to this engine. **Insertion is + transactional**: a single `aliases.lock()` covers both the + collision check and the insert, so two concurrent `LoadEngine` + round-trips for *different* extensions returning the same runtime + name produce a deterministic hard error on whichever ran second. + **The collision check must be identity-aware**, because the benign + `discovery` double-issue (parallel Pass-2, above) can run *this same + engine's* alias insert twice concurrently: check `aliases.get(name)` and + treat `Some(existing)` as a hard collision **only if `existing` is a + *different* extension id** — a re-insert of the same `runtime_name → same + ExtensionId` is an idempotent no-op, not a self-collision. Keying on the + stable `ExtensionId` (`Eq + Hash`) rather than mere name-presence is what + makes the same-engine race a no-op while a genuine two-extension clash is + still a hard error. + + **Name-collision policy: hard error.** Any of the following is a + hard error that fails the render with a clear message naming both + conflicting engines: + + 1. Two extensions declare the same `name` in their `_extension.yml`. + The error fires at registry construction time (Plan 1c Phase 2), + before Pass 1 begins. + 2. Two lazy-loaded engines (no declared name) self-report the same + runtime `name` from `LoadEngine`. The error fires under the + `aliases.lock()` when the second engine's `LoadEngineResult.name` + would overwrite the first entry. + 3. A lazy-loaded engine self-reports a name that collides with a + built-in (`markdown`, `knitr`, `jupyter`) or another already-known + declared engine. Same error. + + The relaxed-collision case (e.g., last-writer-wins, or namespacing by + extension id) is deferred — we can revisit if a real use case + surfaces. The hard-error stance keeps the registry deterministic and + the YAML-vs-runtime contract simple. + + **Cache determinism contract:** the `claims_language` cache assumes the + engine's `claimsLanguage` is a pure function of `(language, first_class)`. + q2 doesn't enforce this; engine authors who introduce non-determinism + (reading mutable state, side effects) will see stale cache hits. The + contract is documented in + [Plan 1c](2026-04-16-plan1c-extension-integration.md) — the + extension-author-facing surface lives there alongside the + `_extension.yml` schema, hint declarations, and engine-API docs. + Same rule applies to `claims_file`'s content-inspection (cache key is + the canonical path; if the engine reads mutable file metadata the + cache will go stale within a single render, which is expected to be + rare in practice). + + **Cache writes only on success.** When the engine throws during + `claimsLanguage` (subprocess sends `FromEngine::Error`), the Rust side + propagates `ExecutionError` to the caller without touching the cache. + Per plan1a-host's "Error categories" item 4, discovery errors are + terminal — the render fails and the host is torn down — so no second + query ever happens against the same cache slot. The cache value type + stays `LanguageClaim` (success states only: `None` for "no claim", or + `Primary`/`Interop`/`Fallback` for a real claim); there is no need for a + `Result<_, _>` slot to encode "engine errored." Same rule applies + trivially to `claims_file_cache`: errors propagate, render fails, no + cache write. + + **File conversion (defined in Phase 3):** + - `markdown_for_file(file, runtime)` → `ensure_launched`, send + `MarkdownForFile`, recv `MarkdownForFileResult`. Return + `(result.value, SourceInfo::default())` — the converted text plus the + reserved (v1-default) provenance slot. (`runtime` is unused — the + subprocess reads files via Deno; `result.source_map` is carried on the + wire but **not consumed** in v1.) Provenance for the converted text is + invented downstream when the convert-then-parse path registers it as an + ephemeral intermediate file under an engine-reflecting synthetic name — + see the **Provenance** note in Phase 3 (scope C′; A′/B′ deferred). Called + only for non-QMD files claimed via `claims_file`. + + Note: `run()` is excluded from the protocol — it's fundamentally different + (long-running interactive mode, not request/response). Deferred to a future + plan. `partitioned_markdown` is excluded too (see Phase 3 rationale and + the ipynb-filters research plan). + +- [x] **Drop `#[derive(Clone)]` on `EngineRegistry` and reroute + `Option` → `Option>`** (mandatory-to-compile + once the `aliases` / `diagnostics` `Mutex` fields land — see the migration + note for the full rationale and verified site list). The build first breaks + at the 8 clone sites; the type then propagates through **~25–30 mechanical + sites total, all trivial type substitutions with zero semantic change**: + the `EngineExecutionStage` field + `with_registry` + `new()`; the three + config structs `PreviewConfig` / `RenderToFileOptions` / **`HtmlRenderConfig`** + (`pipeline.rs:118`) and `with_engine_registry` (`pipeline.rs:131`); the + `quarto-preview` pass-through signature chain; `Arc::new(...)` at the one + real construction site (`render_to_file.rs:331`) + test fixtures. No + post-construction mutation exists, so the wrap is sound. This keeps Plan + 1a independently compilable; Plan 1c does the deeper `ProjectContext` + ownership. `with_replay_many` is untouched. + +- [x] Wire into engine module (`engine/mod.rs`): add the `ts_engine` module + (native-gated, same gate as knitr/jupyter) and the **un-gated** + `resolution` module (see Phase 3.5 — it must compile for WASM). Re-export + `TsEngine` from `engine/mod.rs`. `ts_protocol` is already wired + (plan1a-protocol, done); `ts_process` and the `TsEngineHost` re-export are + added by **plan1a-host**. + **The new *shared* types stay un-gated and must be WASM-clean:** + `LanguageClaim`, `EngineResolution` / `resolve_engines`, the new + `ExecutionContext` leave-alone field, and `ExecuteResult.html_dependencies` + all live in `quarto-core` and feed `wasm-quarto-hub-client`. They are pure + data + a pure function, so they compile for WASM; in a WASM build the + registry is markdown-only (knitr/jupyter native-gated) and execution is + bypassed by `CaptureSpliceStage`, so resolution is inert there but must + still compile and degrade gracefully (no engine → markdown passthrough, no + panic). Gate the rebase commits with full `cargo xtask verify` (not + `--skip-hub-build`) — see design doc §13. +- [x] Transport access is **multiplexed by the `TsEngineHost` demux**, not + serialized by a single transport `Mutex`. Under parallel Pass-2 many + blocking rayon workers call the host concurrently; each `host.request` + allocates an `id`, registers a pending slot, and blocks on *its own* slot + while one reader thread routes `Response`s by `id` (plan1a-host). The + transport's write half is briefly mutexed (one framed write); **no lock is + held across the round-trip**, so cross-engine requests run concurrently (the + Deno event loop interleaves them) and same-engine requests are serialized on + the *harness* side (per-instance queue). The `claims_language_cache` / + `claims_file_cache` mutexes are separate from the transport; under parallel + Pass-2 they *can* be briefly contended (workers resolve concurrently — see + "Concurrent claims caching"), but the contention is short and benign. **The + Rust transport is synchronous** + (plan1a-host's `EngineTransport` is a blocking duplex — `StdioTransport` + over the child's stdin/stdout in v1, newline-framed JSON; **stdout is the + protocol channel**, **stderr** is diagnostic-only, drained on a separate + host-side reader thread. The deferred Phase 1.6 `TcpTransport` (loopback TCP) + moves the protocol off stdout to delete the `console.log` footgun). + `TsEngine` calls it from the sync `ExecutionEngine` methods — no runtime, no + `block_on`, no async bridge. The concurrency is real but lives on the Deno + event loop, surfaced to blocking workers via the demux — + `claude-notes/designs/engine-host-concurrency.md`. (The earlier "single + transport `Mutex` / lockstep request-response / async buys no concurrency" + framing predated parallel Pass-2 and is **retired**.) +- [x] **Transport seam ownership (host vs. engine).** The transport + abstraction is **plan1a-host's**: it defines the `EngineTransport` trait, + `StdioTransport` (the v1 impl; loopback `TcpTransport` is the deferred + Phase 1.6), the `TsEngineHost` demux (all in `ts_process.rs`), and owns + the `StdioTransport` `deno`-gated smoke test. **`MockTransport` is also + a plan1a-host deliverable** (reassigned during the 2026-06-24 review — host's + own timeout/cancel/crash tests need it, and it lives in host's + `ts_process.rs`). plan1a-engine *consumes* both. The test seam: + - **plan1a-host** adds the test-only constructor + `TsEngineHost::with_transport(Box, EngineHostContext)` + — which **starts the real reader thread** so tests run the production demux + path (not a synchronous shortcut). + + > **⚠ Correction — RTQ §Item A:** `EngineHostContext` is being split into a process-stable `Init { global: HostGlobalConfig }` (sent once at `ensure_started`) and a per-render `LaunchEngine { project: EngineProjectContext }`; the shared `HostState.context` slot + launch-gating are removed. This constructor's signature will take the `global` config (project supplied at launch). + - **plan1a-host** owns `MockTransport` — a test-only impl of the + `EngineTransport` trait (under `#[cfg(test)]` in `ts_process.rs`). It is + **id-keyed, delay-capable, and BLOCKS in `recv()`** until a paired/scripted + response is available (a passive `VecDeque` would read empty-as-EOF and + false-trigger the crash path); `shutdown()` signals EOF. It captures sent + messages (`sent_messages() -> &[ToEngine]`) and echoes each `Request`'s + `id`. See plan1a-host's Design Note "MockTransport & the test demux" for + the full shape and rationale. + All Phase 4 unit tests construct a + `TsEngineHost::with_transport(Box::new(MockTransport::…), …)`; no Deno is + required to run plan1a-engine's test suite. + + End-to-end coverage of the real Plan 1b bundle lives in + Plan 1c's echo-engine integration test (Plan 1c Phase 3); the + harness-side idempotency contract is tested in Plan 1b directly + against Plan 1b's harness. Plan 1a does not own subprocess-level + fidelity tests. +- [x] Write `MockTransport` round-trip test: build a `TsEngineHost` + via `with_transport`; send a `LoadEngine` followed by a + `ClaimsLanguage`; assert the response round-trips and that the + captured `sent_messages()` contains the expected sequence. +- [x] Write state-machine test: a `TsEngine` backed by a + `MockTransport` answers discovery without triggering + `LaunchEngine`; calling `execute` triggers `LaunchEngine` exactly + once on the wire (one `ToEngine::LaunchEngine` in the captured + log). +- [x] Write race-free-init tests (two, one per slot): + - **instance (exclusive):** two threads concurrently call + `ensure_launched()` on the same `TsEngine` (synchronized via + `std::sync::Barrier`) against a `MockTransport` pre-seeded with two + `Launched` results. Assert: the `instance` slot converges to a single + value, no panic, captured `LaunchEngine` count is **exactly 1** (the + `Mutex>` serializes init). + - **discovery (benign double-issue):** two threads concurrently call + `ensure_loaded()` against a `MockTransport` seeded with **two distinct** + `LoadEngineResult`s; the **binding** assertion is that **both threads + observe the same cached value** (the `OnceLock` converges) — see Test + Seam Spec row 7. The `LoadEngine` count `1 ≤ n ≤ 2` is kept only as a + shape bound, **not** the discriminator (an over-locked impl also passes + "1 or 2", so count alone is vacuous). + These test the Rust-side invariants Plan 1a owns; the "engine.launch() + invoked exactly once across the real harness" assertion lives in Plan 1b. +- [x] Write poison test: a `TsEngine` whose `MockTransport` returns a + cancel/timeout for an `Execute` request; assert `execute` calls + `poison_instance` (the `instance` slot is `None` afterward) and that a + subsequent `execute` re-issues `LaunchEngine` on the wire. + +## Test Seam Spec (frozen — prevalidated 2026-06-24) + +**Freeze this before writing any test.** Each row names the **one production +hunk** whose revert turns the **named assertion** RED; once a test goes green +its assertions + harness are frozen (never edited to go green). All rows are +`cargo nextest`, native, **no Deno** (subprocess fidelity is Plan 1b; the real +harness composition is Plan 1c's echo E2E). The bulleted test items above are +the prose; this table binds them. Tiers: **claim** (pure trait method), +**resolver** (pure `resolve_engines`), **engine** (`TsEngine` over a +`MockTransport`-backed `TsEngineHost`), **registry**, **dep**. + +| # | Test | Tier | Real unit (not mocked) | Mock boundary | Named revert → RED assertion (+ vacuity guard) | +|---|------|------|------------------------|---------------|------------------------------------------------| +| 1 | Built-in claim tables | claim | `{Knitr,Jupyter,Markdown}Engine::claims_language` | — (pure) | Revert jupyter's blanket `Fallback(0)` to Q1's `"julia"→Primary(1)` → assert `jupyter.claims_language("julia",_) == Fallback(0)` RED. Revert knitr's `Interop` arm for `sql` → assert `knitr.claims_language("sql",_) == Interop(_)` RED. **Vacuity:** assert the exact *kind+payload* (`Fallback(0)`/`Interop(_)`/`Primary(1)`/`None`), never just "non-`None`" — kind dominates, so "some claim" hides the regression. | +| 2 | Trait default `markdown_for_file` | claim | the trait default body | — | Revert the default from `Err(not_supported("markdown_for_file"))` to `Ok((String::new(), default))` → assert `matches!(MarkdownEngine.markdown_for_file(…), Err(ExecutionError::NotSupported("markdown_for_file")))` RED. **Vacuity:** match the `NotSupported` variant **and** its `&'static str` payload, not "is `Err`" (an `Io` err also matches `Err`). | +| 3 | Resolver tiers (§4.4) | resolver | `resolve_engines` + the four tiers | `MockEngine` claim tables (no AST exec) | One revert per rule, each reddening a distinct case: (a) revert **kind-dominates** (sort by priority ignoring kind) → `Primary(-100)` vs `Fallback(0)` case: assert owner==weak-engine RED. (b) revert **Interop presence-gating** (fire Interop unconditionally) → pure `{python}`: assert `sequence==[jupyter]` (not `[knitr]`) RED. (c) revert **T2>T3** (Interop above explicit-Fallback) → `[knitr,jupyter]`+`{sql}`: assert `ownership["sql"]=="jupyter"` RED. (d) revert **T4 implicit-only gate** → explicit `[knitr]`+`{julia}`: assert jupyter **not** added RED. **Vacuity:** assert the `ownership`/`sequence`, and keep the `{r}+{sql}→knitr` vs `[knitr,jupyter]+{sql}→jupyter` pair — they must resolve to *different* owners or presence-gating is untested. | +| 4 | §10 case-4 loud failure | engine | the owner's "owns language, no handler" guard | kernelspec lookup → none | Revert the guard (let the owned-but-unrunnable cell run/skip) → assert jupyter handed `{sql}` it owns returns `Err(NoHandlerForLanguage{engine:"jupyter",language:"sql"})` RED. **Vacuity + path-exercised:** resolution must actually give `sql` to jupyter (assert `ownership["sql"]=="jupyter"` in setup) else it passes vacuously; assert the error **names the language**, and assert **no** unexecuted-cell output (silent no-op would otherwise pass). | +| 5 | Two-step lifecycle | engine | `ensure_loaded` vs `ensure_launched` split | `MockTransport` `sent_messages()` | Revert `claims_language` to call `ensure_launched` (not `ensure_loaded`) → after a discovery-only sequence assert `sent_messages()` has **zero** `LaunchEngine` RED; after one `execute`, **≥1** `LaunchEngine`. **Vacuity:** the discovery call must be one that *could* have launched (a real `ClaimsLanguage` that hits the wire), and assert the **count**, not presence of `LoadEngine`. | +| 6 | Race-free **instance** (exclusive) | engine | `ensure_launched` `Mutex>` init | `MockTransport` + `std::sync::Barrier`, 2 threads | Revert the `Mutex