fix(rbac): retry plugin permission metadata reads during conditional policy reconcile#9770
Open
MaciekStrzelbicki wants to merge 1 commit into
Open
Conversation
…policy reconcile On fresh boot the permission metadata route of a target plugin (for example catalog) can mount later than RBAC startup, which made staging of conditional policies from the conditional policies file fail and abort the reconcile until a later reload or restart. Metadata reads in this path are now retried with bounded exponential backoff and jitter (default budget of roughly 4 minutes), configurable via permission.rbac.conditionalMetadataRetry. Reads triggered through the REST API are not retried and keep failing fast. Signed-off-by: MaciekStrzelbicki <maciek@strzelbicki.pl>
Contributor
Changed Packages
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves RBAC conditional policy reconciliation from permission.rbac.conditionalPoliciesFile by adding a bounded, configurable retry (exponential backoff + jitter) around plugin permission-metadata reads, addressing startup races where target plugins mount their metadata routes after the RBAC backend starts.
Changes:
- Adds retry option parsing/validation and backoff delay computation for conditional-metadata reads, with safe defaults and config overrides.
- Wires retry options into the YAML conditional policies file watcher (file-based reconcile path only), while keeping REST-driven condition changes fail-fast.
- Adds unit/regression coverage plus documentation and config schema updates for the new
permission.rbac.conditionalMetadataRetrysettings.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| workspaces/rbac/plugins/rbac-backend/src/policies/permission-policy.ts | Reads retry config and passes it into the YAML conditional file watcher. |
| workspaces/rbac/plugins/rbac-backend/src/helper.ts | Implements retry options (defaults + config read + validation) and retrying metadata fetch used by conditional policy mapping. |
| workspaces/rbac/plugins/rbac-backend/src/helper.test.ts | Adds tests for retry option resolution, config reading, backoff delay, and processConditionMapping retry behavior. |
| workspaces/rbac/plugins/rbac-backend/src/file-permissions/yaml-conditional-file-watcher.ts | Resolves retry options and applies them when staging conditional policy additions from file. |
| workspaces/rbac/plugins/rbac-backend/src/file-permissions/yaml-conditional-file-watcher.test.ts | Adds coverage for metadata retry behavior during file-based reconcile and validation errors. |
| workspaces/rbac/plugins/rbac-backend/README.md | Documents the new retry behavior and configuration knobs. |
| workspaces/rbac/plugins/rbac-backend/config.d.ts | Extends config schema with permission.rbac.conditionalMetadataRetry. |
| workspaces/rbac/.changeset/mighty-wolves-retry.md | Declares a minor release with user-facing behavior change and configuration details. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey, I just made a Pull Request!
What
Conditional policy reconciliation from
permission.rbac.conditionalPoliciesFilereads the permission metadata of every target plugin (e.g.catalog) to map permission actions to permission names. On a fresh boot, the permission metadata route of a target plugin can mount later than the RBAC backend starts. The metadata collector reports these transient failures (network errors, 5xx) as a missing permission list, so staging failed, and since #9731 the reconcile aborts safely — but the conditions from the file are still not applied until a later file reload or restart.This PR adds a bounded retry with exponential backoff and jitter around the permission metadata reads in the conditional-policies-file reconcile path:
permission.rbac.conditionalMetadataRetry(maxAttempts,baseDelayMs,maxDelayMs);maxAttempts: 1restores the previous fail-fast behavior.POST/PUT /roles/conditions) are not retried and keep failing fast — the retry only applies to the file watcher path, where startup blocks while other plugins continue mounting.conditional_reconcile_aborted) and stored conditions are preserved, as introduced in fix(rbac): make conditional policy reconcile non-destructive and support sibling merges #9731.We have been running this behavior in production as a local patch on top of
@backstage-community/plugin-rbac-backend7.16.0 to eliminate startup races where thecatalogmetadata fetch failed during staging.How
helper.ts:processConditionMappingaccepts optional{ metadataRetry, logger }; newresolveConditionalMetadataRetryOptions/readConditionalMetadataRetryOptionsFromConfig/computeConditionalMetadataBackoffDelayMsfollowing the existingConditionValidationLimits/ConditionalPoliciesFileLimitspatterns (defaults + validated overrides,InputErroron non-positive-integer values).yaml-conditional-file-watcher.ts: accepts optional retry overrides in the constructor and passes them (plus its logger) toprocessConditionMapping.permission-policy.ts: reads the config and wires it into the watcher.config.d.ts: schema forpermission.rbac.conditionalMetadataRetry.✔️ Checklist
Signed-off-byline in the message. (more info)