Skip to content

Run the assembled-desktop E2E suite in CI #186

Description

@Azganoth

Summary

The assembled-desktop E2E suite sits outside pnpm check and outside CI, so regressions at the assembled boundary — IPC, the real folder watcher, persisted state, the window close handshake — are caught only when a maintainer on Windows remembers to run it. Add the suite as a third job in ci.yml on the same triggers as the existing checks, closing that enforcement gap.

The spike in #184 measured the job on a GitHub-hosted runner: it passes, it costs about 5 minutes warm, and it runs in parallel with the existing checks for roughly 30 seconds on the critical path.

Affected areas

  • .github/workflows/ci.yml — one new job.
  • CONTRIBUTING.md, "Local Development" — the suite stops being enforced only by memory.
  • docs/architecture.md, "Verification Strategy" — the suite stops being Windows-local.

Intended outcome

A Desktop E2E job runs on every pull request and every push to main, alongside Frontend Checks and Backend Checks, and fails the run when the assembled suite fails. The suite remains outside pnpm check; CI enforcement does not change what a contributor runs locally.

The job's measured shape from #184:

  • runs-on: windows-2025-vs2026 — the same pinned label as the existing jobs, so image drift cannot silently change the E2E environment alone.
  • timeout-minutes: 20 — above the 6m20s cold run, low enough to catch a hang.
  • CARGO_PROFILE_DEV_DEBUG: 0, matching check-backend.
  • Its own Swatinem/rust-cache entry keyed to ./src-tauri -> target/desktop-e2e. The existing backend cache covers ./src-tauri -> target and shares no artifacts with this build's target directory and feature set.
  • A teardown assertion for orphaned processes and port 4445, allowing about 10 seconds of grace for WebView2 host processes to exit.
  • Failure-only upload of e2e/desktop/artifacts/ at retention-days: 7.

Related context

Done when

  • The job runs on pull_request and push to main and fails the run when the suite fails.
  • Cold and warm runs both complete inside the 20-minute ceiling.
  • Failure evidence uploads and is retrievable, and a passing run uploads nothing.
  • The teardown check passes with no orphaned leafdown-e2e or msedgedriver process and no listener on port 4445.
  • CONTRIBUTING.md and docs/architecture.md are updated in the same change.

Notes

Measured on windows-2025-vs2026 across five runs. Cold: 6m20s total, 2m25s build, 1m27s suite. Warm: 4m27s–5m01s total, 44–48s build, 88–98s suite. The suite's CI runtime matches the ~92s measured locally.

WebView2 is preinstalled per-machine at 150.0.4078.105 despite not appearing in the runner image manifest, so no install step is needed. Uploaded artifacts contained only runner paths, so the sensitivity caveat in CONTRIBUTING.md applies to local runs rather than CI ones.

Monetary cost is zero: the repository is public and this is a standard GitHub-hosted runner.

Implementation direction

The prototype workflow from #184 is a measurement harness, not production-standard, and its branch has been deleted. Write the job against the existing ci.yml conventions rather than porting it.

pnpm test:e2e:desktop runs the build and the suite together. Splitting them into separate steps makes the build and suite costs visible in the run timing and is worth keeping.

The standing response to a flaky failure is to re-run once to confirm, then fix the assertion to target durable state.

Out of scope

  • WebdriverIO retries or any other retry mechanism.
  • Path filters on the trigger. Nearly every pull request touches src/ or src-tauri/, so a filter would add configuration and save close to nothing.
  • Parallel workers, a platform matrix, visual regression, and native dialog automation.
  • Adding the suite to pnpm check.

Metadata

Metadata

Assignees

Labels

MaintenanceChore, refactor, dependency, or test work without user-facing change

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions