Skip to content

fix(rbac): retry plugin permission metadata reads during conditional policy reconcile#9770

Open
MaciekStrzelbicki wants to merge 1 commit into
backstage:mainfrom
MaciekStrzelbicki:fix/rbac-conditional-metadata-retry
Open

fix(rbac): retry plugin permission metadata reads during conditional policy reconcile#9770
MaciekStrzelbicki wants to merge 1 commit into
backstage:mainfrom
MaciekStrzelbicki:fix/rbac-conditional-metadata-retry

Conversation

@MaciekStrzelbicki

Copy link
Copy Markdown

Hey, I just made a Pull Request!

What

Conditional policy reconciliation from permission.rbac.conditionalPoliciesFile reads the permission metadata of every target plugin (e.g. catalog) to map permission actions to permission names. On a fresh boot, the permission metadata route of a target plugin can mount later than the RBAC backend starts. The metadata collector reports these transient failures (network errors, 5xx) as a missing permission list, so staging failed, and since #9731 the reconcile aborts safely — but the conditions from the file are still not applied until a later file reload or restart.

This PR adds a bounded retry with exponential backoff and jitter around the permission metadata reads in the conditional-policies-file reconcile path:

  • Defaults: 12 attempts, 2s base delay, 30s max delay — roughly a 4-minute retry budget that covers slow cold-start mounts of permission metadata routes without retrying forever.
  • Configurable via permission.rbac.conditionalMetadataRetry (maxAttempts, baseDelayMs, maxDelayMs); maxAttempts: 1 restores the previous fail-fast behavior.
  • Each retry logs a warning with the attempt number and delay.
  • Reads triggered through the REST API (POST/PUT /roles/conditions) are not retried and keep failing fast — the retry only applies to the file watcher path, where startup blocks while other plugins continue mounting.
  • If the metadata endpoint stays unavailable beyond the retry budget, the reconcile still aborts safely (conditional_reconcile_aborted) and stored conditions are preserved, as introduced in fix(rbac): make conditional policy reconcile non-destructive and support sibling merges #9731.

We have been running this behavior in production as a local patch on top of @backstage-community/plugin-rbac-backend 7.16.0 to eliminate startup races where the catalog metadata fetch failed during staging.

How

  • helper.ts: processConditionMapping accepts optional { metadataRetry, logger }; new resolveConditionalMetadataRetryOptions / readConditionalMetadataRetryOptionsFromConfig / computeConditionalMetadataBackoffDelayMs following the existing ConditionValidationLimits / ConditionalPoliciesFileLimits patterns (defaults + validated overrides, InputError on non-positive-integer values).
  • yaml-conditional-file-watcher.ts: accepts optional retry overrides in the constructor and passes them (plus its logger) to processConditionMapping.
  • permission-policy.ts: reads the config and wires it into the watcher.
  • config.d.ts: schema for permission.rbac.conditionalMetadataRetry.

✔️ Checklist

  • A changeset describing the change and affected packages. (more info)
  • Added or updated documentation
  • Tests for new functionality and regression tests for bug fixes
  • Screenshots attached (for UI changes) — n/a, backend only
  • All your commits have a Signed-off-by line in the message. (more info)

…policy reconcile

On fresh boot the permission metadata route of a target plugin (for
example catalog) can mount later than RBAC startup, which made staging
of conditional policies from the conditional policies file fail and
abort the reconcile until a later reload or restart.

Metadata reads in this path are now retried with bounded exponential
backoff and jitter (default budget of roughly 4 minutes), configurable
via permission.rbac.conditionalMetadataRetry. Reads triggered through
the REST API are not retried and keep failing fast.

Signed-off-by: MaciekStrzelbicki <maciek@strzelbicki.pl>
@backstage-goalie

Copy link
Copy Markdown
Contributor

Changed Packages

Package Name Package Path Changeset Bump Current Version
@backstage-community/plugin-rbac-backend workspaces/rbac/plugins/rbac-backend minor v7.16.0

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves RBAC conditional policy reconciliation from permission.rbac.conditionalPoliciesFile by adding a bounded, configurable retry (exponential backoff + jitter) around plugin permission-metadata reads, addressing startup races where target plugins mount their metadata routes after the RBAC backend starts.

Changes:

  • Adds retry option parsing/validation and backoff delay computation for conditional-metadata reads, with safe defaults and config overrides.
  • Wires retry options into the YAML conditional policies file watcher (file-based reconcile path only), while keeping REST-driven condition changes fail-fast.
  • Adds unit/regression coverage plus documentation and config schema updates for the new permission.rbac.conditionalMetadataRetry settings.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated no comments.

Show a summary per file
File Description
workspaces/rbac/plugins/rbac-backend/src/policies/permission-policy.ts Reads retry config and passes it into the YAML conditional file watcher.
workspaces/rbac/plugins/rbac-backend/src/helper.ts Implements retry options (defaults + config read + validation) and retrying metadata fetch used by conditional policy mapping.
workspaces/rbac/plugins/rbac-backend/src/helper.test.ts Adds tests for retry option resolution, config reading, backoff delay, and processConditionMapping retry behavior.
workspaces/rbac/plugins/rbac-backend/src/file-permissions/yaml-conditional-file-watcher.ts Resolves retry options and applies them when staging conditional policy additions from file.
workspaces/rbac/plugins/rbac-backend/src/file-permissions/yaml-conditional-file-watcher.test.ts Adds coverage for metadata retry behavior during file-based reconcile and validation errors.
workspaces/rbac/plugins/rbac-backend/README.md Documents the new retry behavior and configuration knobs.
workspaces/rbac/plugins/rbac-backend/config.d.ts Extends config schema with permission.rbac.conditionalMetadataRetry.
workspaces/rbac/.changeset/mighty-wolves-retry.md Declares a minor release with user-facing behavior change and configuration details.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants