Skip to content

CI: fix milestone-tag-assistant race when labels change post-merge#67337

Open
potiuk wants to merge 1 commit into
apache:mainfrom
potiuk:fix-milestone-tag-assistant-race
Open

CI: fix milestone-tag-assistant race when labels change post-merge#67337
potiuk wants to merge 1 commit into
apache:mainfrom
potiuk:fix-milestone-tag-assistant-race

Conversation

@potiuk
Copy link
Copy Markdown
Member

@potiuk potiuk commented May 22, 2026

The `milestone-tag-assistant.yml` workflow snapshots PR labels at the `get-pr-info` job (via `listPullRequestsAssociatedWithCommit`) and then spends ~1.5 minutes installing Breeze and running `breeze ci set-milestone`. If a maintainer adds and removes a backport label inside that window, the action commits to the stale-snapshot decision and sets the wrong milestone.

This happened on #67301: a `backport-to-v3-2-test` label that lived for 49 seconds caused an `Airflow 3.2.3` milestone to be set on a `main`-only documentation PR. The workflow's comment justifying the decision was posted ~1.5 minutes after the label had already been removed.

Timeline (PR #67301)

Time Actor Event
20:31:18 shahar1 Merges PR (no milestone, no backport label)
20:31:28 shahar1 Adds `backport-to-v3-2-test` (+10 s)
20:32:17 shahar1 Removes `backport-to-v3-2-test` (+49 s after adding)
20:33:49 github-actions[bot] Sets milestone `Airflow 3.2.3` ❌
20:33:51 github-actions[bot] Posts justifying comment
next morning eladkal Removes the incorrect milestone

Fix

Re-read `issue.labels` from the freshly-fetched issue (which is already in scope after the existing `milestone is not None` check) before computing the milestone. If the labels changed since the snapshot:

  • Honour any skip label that appeared after the snapshot.
  • Re-run `_determine_milestone_version` with the current labels and use the fresh decision; if the decision flips to "no milestone", bail out before posting the comment.

Adds three regression tests covering the three race-window cases (backport label removed, replaced, skip label added) and updates two existing happy-path tests to populate `mock_issue.labels` so the re-read sees the same labels as the snapshot.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

The `milestone-tag-assistant.yml` workflow snapshots PR labels at the
`get-pr-info` job (via `listPullRequestsAssociatedWithCommit`) and then
spends ~1.5 minutes installing Breeze and running `breeze ci
set-milestone`. If a maintainer adds and removes a backport label
inside that window, the action commits to the stale-snapshot decision
and sets the wrong milestone — see the incident on PR apache#67301 where a
backport label that lived for 49 seconds caused an Airflow-3.2.3
milestone to be set on a `main`-only documentation PR.

Re-read `issue.labels` from the freshly-fetched issue before computing
the milestone. If the labels changed since the snapshot:

- Honour any skip label that appeared after the snapshot.
- Re-run `_determine_milestone_version` with the current labels and
  use the fresh decision; if the decision flips to "no milestone",
  bail out before posting the comment.

Adds three regression tests covering the three race-window cases
(backport label removed, replaced, skip label added) and updates two
existing happy-path tests to populate `mock_issue.labels` so the
re-read sees the same labels as the snapshot.
@potiuk potiuk requested a review from ashb as a code owner May 22, 2026 13:58
@potiuk potiuk added this to the Airflow 3.2.2 milestone May 22, 2026
@boring-cyborg boring-cyborg Bot added area:dev-tools backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch labels May 22, 2026
Copy link
Copy Markdown
Member

@jason810496 jason810496 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching the race condition! (I didn't foresee this situation when developing)

console_print(f"[info]Snapshot labels: {sorted(labels)}[/]")
console_print(f"[info]Current labels: {sorted(current_labels)}[/]")

if _should_skip_milestone_tagging(current_labels):
Copy link
Copy Markdown
Member

@jason810496 jason810496 May 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand the event correctly, we should add additional check in _should_skip_milestone_tagging.
If there's any event that maintainer remove any backport label themself. We need to skip the auto tagging.

Even with the new if set(current_labels) != set(labels): check, I don't think the existing implementation of _should_skip_milestone_tagging would be enough.

# Labels that indicate the PR should be skipped from milestone auto-tagging
MILESTONE_SKIP_LABELS: frozenset[str] = frozenset({"area:dev-tools", "area:dev-env", "area:CI"})

def _should_skip_milestone_tagging(labels: list[str]) -> bool:
"""Check if the PR should be skipped from milestone auto-tagging."""
return bool(set(labels) & MILESTONE_SKIP_LABELS)

Btw, I can help follow-up for this issue if you don't have bandwidth at this moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants