test(visual): add visual regression testing across all routes and themes

[afonsojramos]

Summary

Adds pixel-level visual regression testing. The existing 54 .snap files serialize DOM structure, which is structurally blind to the class of change the recent theming work has been making: colour tokens, spacing, and CSS variables all move without altering markup.

This screenshots every route in a real browser and commits the baselines.

36 baselines, ~1.5 MB:

• 11 routes x {light, dark} Classic = 22
• Notifications + Settings x {light/dark colorblind, light/dark tritanopia, dark dimmed} = 10
• Notifications + Settings x Glass {light, dark} = 4

Built on Vitest 4 browser mode rather than a standalone Playwright suite, so it reuses the existing helpers, mocks, and config. Playwright is present only as the browser driver; there is no second test runner, no playwright.config.ts, and no separate spec format.

How it works

Screenshots only match when the browser build and font stack are identical, so both commands run the suite inside a pinned Playwright container, and CI verifies against that same container. Contributors do not need Playwright's browsers installed locally.

pnpm test:visual          # verify against committed baselines
pnpm test:visual:update   # regenerate after an intentional UI change

.gitignore permits only *-chromium-linux.png, so a screenshot taken on a contributor's own OS can never become a baseline. pnpm test is unchanged in scope and still runs the DOM and node projects only.

scripts/visual.sh reads the image tag from the playwright devDependency, so there is nothing to keep in sync there. The CI job's container: is a literal tag because zizmor's unpinned-images rule rejects an expression it cannot evaluate, and suppressing a security rule to save one string is a poor trade; a Renovate group keeps it moving in lockstep with the package instead.

Determinism

Baselines are worthless if they drift, so each source of nondeterminism is pinned explicitly:

Source	Handling
Relative timestamps	Clock frozen at a fixed instant (Date only, so React's scheduler keeps real timers)
Remote avatars	Rewritten to an inline SVG; nothing is fetched during a run
Twemoji SVGs	Stubbed; img.emoji is sized purely by CSS so layout is unchanged
Entrance animations	Settled to their end state before render
Emoji selection	randomElement stubbed
Timezone / locale	Forced to UTC / en-US at the browser context

Two of these were found the hard way. Avatars were leaking to githubusercontent because GitifyUser spells it avatar while notifications use avatarUrl. And the login route flaked 4 runs in 5: its tab indicator measures positions with getBoundingClientRect in a layout effect, and Playwright's built-in animation freezing happens at screenshot time, far too late for that measurement.

Guard against false greens

Vitest namespaces baselines per platform and creates missing ones automatically. Run bare on macOS, the first invocation therefore writes a fresh -darwin set from the current working tree and the second passes against it — reporting green on a branch carrying a real regression, without ever consulting the committed Linux baselines. This was hit in practice during review.

A globalSetup guard now fails the run outside Linux with an actionable message, and the npm scripts route through the container, so the trap is unreachable. Verified: with an 8px sidebar regression present, both the first and second run now fail.

Verification

• Catches large regressions. A 4px sidebar width change failed all 36 with a legible diff image.
• Catches small ones. A swapped 18px icon (~150 differing pixels) fails 9 tests.
• Stable. Six consecutive container runs, 36/36 each.
• CI matches local. The job passes against baselines generated on a maintainer's laptop, byte for byte at zero pixel tolerance.

allowedMismatchedPixels: 0 is deliberate. Rendering here is fully deterministic, and a ratio-based tolerance of even 0.01 would permit ~2000 pixels on a 500x400 viewport, which is larger than an entire icon and enough to hide a completely swapped glyph.

Included fix

fix(mocks) corrects two notification fixtures whose hardcoded display.icon values production can never emit:

Fixture	Subject	Was	Now
Bump Version	PullRequest	FeedPullRequestOpenIcon	GitPullRequestIcon
Release 0.0.1	Release	IssueOpenedIcon	TagIcon

FeedPullRequestOpenIcon appears nowhere in src outside that mock. This caused zero snapshot churn, which is precisely why it went unnoticed: no existing test ever rendered those two icons.

Tradeoffs worth reviewing

The job runs on ubuntu-24.04-arm, not ubuntu-latest. Chromium segfaults under amd64 emulation on Apple Silicon, which makes local runs impossible on the machines maintainers actually use. Pinning both sides to arm64 keeps generation and verification on identical hardware. The cost is that running the suite locally needs an arm64 host with Docker; everyone else relies on CI and can read the diff from the visual-regression-diffs artifact.

Glass coverage is partial. These capture the backdrop-filter fallback that Linux and Windows receive. The macOS native vibrancy material is a window property Chromium cannot render, so it still needs a real-desktop capture.

Baselines are binary blobs. Git stores them without useful deltas, so every intentional theme change rewrites them and grows history permanently. The matrix is deliberately scoped to keep that bounded rather than screenshotting every route against all 8 themes.

Follow-up, not included

Size.LARGE is 18, and octicons round down to their 16px variant, so NotificationRow.tsx:102, RepositoryNotifications.tsx:90, and Accounts.tsx:159 all upscale a 16px-hinted glyph by 1.125-1.25x and render slightly soft. Left alone here because changing icon sizes is a design call rather than a bug fix.

Test plan

• pnpm test — 163 files, 1276 tests passing
• pnpm test:visual — 36/36, verified six consecutive runs
• pnpm tsc --noEmit clean
• pnpm check clean
• Regression detection confirmed for a layout shift, an icon swap, and the false-green path
• CI green, including the Visual Regression job running all 36 on ubuntu-24.04-arm

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[setchy]

Awesome addition, @afonsojramos 👏