Skip to content

ci: warn when changesets don't cover modified packages [SIMULATION]#508

Closed
turnipdabeets wants to merge 8 commits intomainfrom
ci/changeset-coverage-check
Closed

ci: warn when changesets don't cover modified packages [SIMULATION]#508
turnipdabeets wants to merge 8 commits intomainfrom
ci/changeset-coverage-check

Conversation

@turnipdabeets
Copy link
Copy Markdown
Contributor

💡 Motivation and Context

Draft / simulation — do not merge as-is.

Adds a non-blocking CI workflow that compares packages modified in a PR against packages declared in any changeset added/modified in that PR, and posts a sticky comment if they don't line up. Designed to catch the class of mistake that caused #491#507: changeset frontmatter pointing at the wrong package.

This PR also intentionally simulates that mistake so the workflow's comment can be observed end-to-end:

  • Code change (the marker file) is in posthog-android-gradle-plugin/
  • Changeset declares "posthog-android": patch
  • The workflow should post a comment flagging both directions of the mismatch

When validation is done, the simulation files (.changeset/coverage-check-simulation.md and posthog-android-gradle-plugin/SIMULATION_DELETE_BEFORE_MERGE.md) get deleted, leaving just the workflow + script.

What the workflow does

  • Triggers on pull_request against main.
  • Parses pnpm-workspace.yaml and each package's package.json to map directories → package names.
  • Diffs against the merge base; collects packages where any non-CHANGELOG.md/package.json source file changed.
  • Reads frontmatter of any .changeset/*.md added or modified in the PR.
  • Renders a markdown report and posts (or updates) a sticky PR comment via marocchino/sticky-pull-request-comment.
  • Never fails the build — exits 0 in every case. Skipping the check = don't add a changeset.

💚 How did you test it?

  • Dry-ran scripts/check-changeset-coverage.mjs locally on this branch — it correctly identified posthog-android-gradle-plugin as modified-but-undeclared and posthog-android as declared-but-unmodified.
  • The simulation in this PR is the integration test — once CI runs, the sticky comment should appear with the same content.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran pnpm changeset to generate a changeset file
  • Added the "release" label to the PR to indicate we're publishing new versions for the affected packages

(The changeset present in this PR is part of the simulation, not a real release. No release label.)

Follow-ups before this is mergeable

  • Pin marocchino/sticky-pull-request-comment@v2 to a SHA (the rest of the codebase pins third-party actions to SHA).
  • Delete the two simulation files.
  • Optional: add the workflow to required-checks-but-warn-only docs if PostHog has any.

Comment thread .github/workflows/changeset-coverage.yml Fixed
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 4, 2026

⚠️ Possible changeset mismatch — modified and declared packages differ

This is informational — the PR is not blocked. Click the triangle above to collapse, or push a fix and this comment will auto-delete.

Modified in this PR but not in any changeset:

  • posthog-android-gradle-plugin

If this package should ship the change, add it to the changeset frontmatter:

---
"posthog-android-gradle-plugin": patch
---

Declared in a changeset but no source files modified:

  • posthog-android

Double-check this is intentional — for example, releasing a previously-merged change.

Changesets in this PR:

  • posthog-android — patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants