Skip to content

test(e2e): add ExTester end-to-end test for the Deepnote notebook flow#430

Draft
tkislan wants to merge 2 commits into
mainfrom
tk/extester-e2e
Draft

test(e2e): add ExTester end-to-end test for the Deepnote notebook flow#430
tkislan wants to merge 2 commits into
mainfrom
tk/extester-e2e

Conversation

@tkislan

@tkislan tkislan commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

@coderabbitai ignore

Summary

Adds a black-box ExTester (vscode-extension-tester) E2E test that drives the full Deepnote happy path through the real VS Code UI: open a workspace folder + a one-notebook .deepnote file → create a Deepnote environment → select it for the notebook (kernel connects, venv + deepnote-toolkit provisioned) → Run All → assert the rendered output contains hello world.

What's included

  • test/e2e/ — the suite (helloWorld.e2e.test.ts), config (tsconfig.json, .mocharc.js, settings.json), and the hello-world fixture
  • package.jsonvscode-extension-tester devDep + compile-e2e / setup:e2e* / test:e2e scripts
  • .gitignore / .vscodeignore — ignore ExTester artifacts and keep them out of the packaged VSIX
  • specs/e2e-extester-testing-plan.md — self-contained plan/reference, with a "§0 Implementation reality" section documenting the fixes below

Key fixes needed to pass headlessly

  • Open the containing workspace folder (the serializer otherwise blocks on a "no workspace folder" snapshot warning) and open the notebook via Quick Open — ExTester's code -r reuse-window silently no-ops in the sandbox
  • Accept the simple folder dialog via its OK button (Enter navigates into directories)
  • Run via the toolbar "Run All" button, re-issued until output renders (deepnote.runallcells is gated behind context keys that are unset under automation)
  • Idempotent environment creation with a stable name (reuses the venv)

No proposed-API grant needed

The extension declares Jupyter's enabledApiProposals, but the core flow (activation, the notebook serializer, kernel execution, output rendering) runs on stable VS Code APIs, so the suite runs on a plain stable VS Code with no product.json allow-listing — the way the published Marketplace / Open VSX extension runs for end users. (An earlier revision patched product.json; that was verified unnecessary and removed.)

How to run

npm run compile
npm run setup:e2e   # download test VS Code + ChromeDriver, install ms-python.python
xvfb-run --auto-servernum --server-args='-screen 0 1920x1080x24' npm run test:e2e

Verification

Verified passing locally on headless Linux (Xvfb, VS Code 1.111.0), with no proposed-API grant in product.json, including a clean-state run that provisions the venv from scratch:

✔ creates an environment, connects the kernel, runs the cell and renders output
1 passing

🤖 Generated with Claude Code

https://claude.ai/code/session_01LQk7n13UfmeDo2ujy8X4LX

Adds a black-box ExTester (vscode-extension-tester) E2E suite that drives the
full Deepnote happy path through the real VS Code UI: open a workspace folder
and a one-notebook `.deepnote` file, create a Deepnote environment, select it
for the notebook (kernel connects, venv + deepnote-toolkit provisioned), run
the cell, and assert the rendered output contains "hello world".

Beyond the test itself, this wires up reproducible setup and the fixes required
to make it pass in a headless/sandboxed VS Code instance:

- enable-proposed-api.js allow-lists the extension's proposed APIs in the test
  VS Code's product.json (the extension does not activate otherwise)
- open the containing folder as a workspace (the serializer otherwise blocks on
  a "no workspace folder" snapshot warning) and open the notebook via Quick
  Open, since ExTester's `code -r` reuse-window silently no-ops in the sandbox
- accept the simple folder dialog via its OK button (Enter navigates into dirs)
- run via the toolbar "Run All" button and re-issue until output renders
  (deepnote.runallcells is gated behind context keys unset under automation)
- idempotent environment creation with a stable name (reuses the venv)
- exclude e2e artifacts from the VSIX in .vscodeignore

Verified passing locally on headless Linux (Xvfb, VS Code 1.111.0), including a
clean-state run that provisions the venv from scratch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LQk7n13UfmeDo2ujy8X4LX
@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0%. Comparing base (aef728e) to head (8bc972c).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@     Coverage Diff     @@
##   main   #430   +/-   ##
===========================
===========================
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Verified the extension does not need a proposed-API grant: although it declares
Jupyter's `enabledApiProposals`, the core flow — activation, the notebook
serializer, kernel execution, and output rendering — runs on stable VS Code APIs
(output goes through `controller.createNotebookCellExecution` with a
`replaceCells` fallback; the one genuinely-proposed call is guarded). On a plain
stable VS Code the proposals are simply ungranted (a non-fatal log) and nothing
breaks — which is how the published Marketplace / Open VSX extension runs for end
users.

So the E2E suite no longer patches the test VS Code's `product.json`:
- remove `test/e2e/enable-proposed-api.js`
- remove the `setup:e2e:proposed-api` script and drop it from `setup:e2e`
- update the plan doc accordingly (§0, §6.x, §8, §9)

Verified: `npm run test:e2e` passes with no Deepnote entry in the test VS Code's
`product.json` allow-list, including a clean-state run that builds the venv from
scratch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LQk7n13UfmeDo2ujy8X4LX
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant