Cherry pick from release-3.15 to develop#7415
Open
hanwen-cluster wants to merge 9 commits into
Open
Conversation
The tags are not propagated to root volumes in us-iso regions
…utils The SSM automation test pipeline deletes tests/performance_tests before running tests, breaking nccl_common.py's top-level import from that package. Moreover, The function is a generic DynamoDB reporting utility with no performance-test-specific logic, so tests/common/utils is a more appropriate home regardless.
In us-iso*, us-isob* regions, CloudFormation has a behavior where an UpdateStack call that includes both a Tags change and a resource whose change is only in Metadata does not update that resource. This breaks the head node update flow: cfn-hup on the head node polls DescribeStackResource for Metadata changes on HeadNodeLaunchTemplate, never sees them when tags are updated in the same call, and the HeadNodeWaitCondition times out after 30 minutes. Until the CloudFormation behavior is addressed, block tag updates at the validation layer in ADC regions so the failure mode surfaces immediately. Commercial, GovCloud, and China regions are unaffected. Policy changes: - Add UpdatePolicy.SUPPORTED_UNLESS_ADC, equivalent to SUPPORTED outside ADC and UNSUPPORTED inside ADC. - Cover the new policy with parametrized unit tests across commercial, GovCloud, China, us-iso, us-isob, and empty region values.
Previously, OS rotation was seeded by day-of-year, meaning multiple builds on the same day would always test the same OS. This limited coverage when running tests more than once per day. Now, when --global-build-number is provided and non-zero, it is used as the rotation seed instead. Each build picks a different OS, enabling full coverage across consecutive pipeline runs. Falls back to day-based rotation when global-build-number is None or 0.
himani2411
reviewed
May 28, 2026
| {"Name": "HeadNode", "parallelcluster:node-type": "HeadNode"}, | ||
| ), | ||
| "tag_getter_kwargs": {"cluster": cluster, "os": os}, | ||
| "skip": "us-iso" in cluster.region, |
Contributor
There was a problem hiding this comment.
Can you add a comment / TODO for this commit so that we remember to revert this commit once Tag is supported in us-iso-region
himani2411
reviewed
May 28, 2026
| return any(path.startswith("SlurmQueues[") for path in change.path) | ||
|
|
||
|
|
||
| def _is_adc_region(): |
Contributor
There was a problem hiding this comment.
Can you add a comment / TODO for this commit so that we remember to revert this commit once Tag is supported in us-iso-region
Contributor
There was a problem hiding this comment.
Also lets add a TBR or Some kind of Tag in commit message so that we can track which commit needs to be reverted in future
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist
developadd the branch name as prefix in the PR title (e.g.[release-3.6]).Please review the guidelines for contributing and Pull Request Instructions.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.