feat(visual-regression): prototipo testes visuais com Playwright#2789
feat(visual-regression): prototipo testes visuais com Playwright#2789alinelariguet wants to merge 9 commits into
Conversation
- Configura Playwright com toHaveScreenshot para visual regression - Cria visual-app separado do projects/app (isolamento completo) - Estrutura co-localizada em e2e/visual/ por componente - Testa 11 componentes basicos + 13 estados do po-input - Adiciona job CI com deteccao de branch po-style - Se branch de mesmo nome existe no po-style, builda e usa tgz - Corrige bindings p-show-required e p-required-field-error-message - Timeout de 60s para compatibilidade com Windows
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
- Upload do relatorio HTML como artefato (visual-regression-report) - Upload dos resultados de teste como artefato (visual-regression-test-results) - Ambos disponiveis mesmo quando testes falham (if: !cancelled()) - Retencao de 30 dias para revisao pelo desenvolvedor
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| - name: Generate baselines for CI environment | ||
| if: steps.baselines-cache.outputs.cache-hit != 'true' | ||
| run: npx playwright test --update-snapshots --reporter=list |
There was a problem hiding this comment.
🔴 CI baseline regeneration overwrites committed baselines, making visual regression tests always pass on cache miss
When cache-hit != 'true' (which happens whenever spec/HTML files change, when the cache is evicted, or on first runs for new branches), the CI runs npx playwright test --update-snapshots which regenerates all 186 committed baseline screenshots from the current code. The subsequent npx playwright test then compares against those just-generated baselines, which trivially always passes.
This completely defeats the purpose of visual regression testing in the exact scenario where it matters most: when a developer modifies UI components AND updates test files in the same PR. The --update-snapshots flag updates ALL snapshots (not just missing ones), so even existing tests that should catch regressions in unchanged components get their baselines silently overwritten.
Additionally, the restore-keys fallback (visual-baselines-${{ runner.os }}-) provides no benefit—actions/cache@v4 sets cache-hit to false for restore-key matches, so the restored baselines are immediately overwritten by the regeneration step.
The baselines are committed to the repo but ignored by CI
There are 186 baseline PNGs committed in e2e/visual/__snapshots__/ which represent the approved visual state. The checkout step (actions/checkout@v3) makes them available, but the regeneration step at line 255 overwrites them. The fix should check if committed baselines already exist before regenerating, e.g.:
- name: Generate baselines for CI environment
if: steps.baselines-cache.outputs.cache-hit != 'true'
run: |
if find e2e/visual/__snapshots__ -name '*.png' | grep -q .; then
echo "Baselines found from repo/cache, skipping regeneration"
else
npx playwright test --update-snapshots --reporter=list
fi| - name: Generate baselines for CI environment | |
| if: steps.baselines-cache.outputs.cache-hit != 'true' | |
| run: npx playwright test --update-snapshots --reporter=list | |
| - name: Generate baselines for CI environment | |
| if: steps.baselines-cache.outputs.cache-hit != 'true' | |
| run: | | |
| if find e2e/visual/__snapshots__ -name '*.png' | grep -q .; then | |
| echo "Baselines found from repo or restore-keys, skipping regeneration" | |
| else | |
| echo "No baselines found, generating new ones" | |
| npx playwright test --update-snapshots --reporter=list | |
| fi |
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
O ponto é válido, mas intencional neste caso. As baselines commitadas no repo são geradas em ambiente local (para desenvolvimento) e possuem dimensões ligeiramente diferentes do ambiente CI (GitHub Actions Ubuntu 24.04) — por exemplo, o po-textarea renderiza com 7px de diferença na altura entre os dois ambientes. Se usássemos as baselines commitadas diretamente no CI, os testes sempre falhariam por mismatch de dimensão.
O fluxo desenhado é:
- Primeiro run (cache miss): gera baselines no ambiente CI → passa
- Runs subsequentes (cache hit): compara contra baselines do cache do CI → detecta regressões entre execuções no CI
Quando os arquivos de spec/HTML mudam, o cache key muda e baselines são regeneradas — isso é esperado, pois os testes mudaram.
Para detecção de regressão cross-versão mais robusta (comparando contra baselines de uma branch base), seria necessário um approach mais sofisticado como Chromatic ou cache baseado na branch target. Isso pode ser evoluído em iterações futuras.
1 participant
feat(visual-regression): prototipo testes visuais com Playwright
Summary
Adds a Playwright-based visual regression testing prototype to
po-angular. The infrastructure captures pixel-by-pixel screenshots of PO UI components and compares them against committed baselines, failing on any visual difference.Key architectural decisions:
visual-appregistered inangular.json— completely isolated fromprojects/app/(used for behavior validation). Lives undere2e/visual/app/.e2e/visual/containing spec, component template, and nearby__snapshots__/baselines.test-visualjob inci.ymlchecks if a branch with the same name exists inpo-ui/po-style. If found, it clones, builds, and packs that branch into a.tgz, installing it over the npm version before running visual tests. This prevents false positives when CSS and components change together across repos.Coverage: 24 tests total — 11 basic component samples (po-button, po-table, po-accordion, etc.) + 13
po-inputstate combinations (disabled, readonly, required, loading, label/helper permutations, etc.).Commands:
npm run test:visual— run tests against baselinesnpm run test:visual:update— regenerate baselines (needed on first run or after intentional changes)npm run test:visual:report— open HTML report with visual diffsUpdates since last revision
actions/upload-artifact@v4steps to thetest-visualCI job:visual-regression-report— Playwright HTML report (e2e/visual/playwright-report/)visual-regression-test-results— test result diffs and screenshots (e2e/visual/test-results/)if: ${{ !cancelled() }}so artifacts are available even when tests failretention-days: 30Review & Testing Checklist for Human
Test visual regressionjob run, and confirm bothvisual-regression-reportandvisual-regression-test-resultsartifacts are listed and downloadable. Download the report artifact and openindex.htmlto confirm it renders a valid Playwright test report.e2e/visual/__snapshots__/po-input/— particularlypo-input-state-required-*.pngandpo-input-state-required-error-*.png. Verify the required asterisk and error message are actually visible in the screenshots (these use[p-show-required]="true"and[p-required-field-error-message]="true"property bindings).curltoapi.github.comwithout token). Rate limit is 60 req/hour per IP on shared runners — if hit, the step silently falls back to the npm version. Consider whether${{ secrets.GITHUB_TOKEN }}should be added for authentication..pngfiles were generated on Linux (chromium-linux). Local dev on Windows/macOS will get mismatches until they runtest:visual:update. Confirm this is acceptable UX for contributors.test-results/artifact may be empty on success — Playwright only writes diffs totest-results/when tests fail. On a green run the artifact will contain no meaningful files. This is expected behavior but worth confirming.Suggested test plan:
npx playwright install chromiumnpm i && npm run build:ui:litenpm run test:visual— all 24 tests should pass (may needtest:visual:updateon non-Linux platforms)e2e/visual/po-button/po-button.visual.spec.tsto change the button label), re-runtest:visual, and verify it fails with a clear diff in the HTML reportNotes