diff --git a/skills/bedrock-adoption-readiness/CHANGELOG.md b/skills/bedrock-adoption-readiness/CHANGELOG.md new file mode 100644 index 0000000..0553f34 --- /dev/null +++ b/skills/bedrock-adoption-readiness/CHANGELOG.md @@ -0,0 +1,15 @@ +# Changelog + +## [1.0.0] - 2026-08-21 +### Added +- Initial release (4 dimensions) +- D1: IAM governance with full policy document inspection (managed + inline) +- D2: Data retention / ZDR with Covered Model detection fallback and three-regime model +- D3: Quota headroom with per-model comparison, cache awareness, and 5:1 burndown rate +- D6: Operational observability aligned with CWR checklist thresholds +- Dual-surface support: Standard Bedrock (AWS/Bedrock) and Mantle (AWS/BedrockMantle) +- Multi-region discovery and assessment +- Three-state dimension model: ASSESSED / NOT_ASSESSED / INSUFFICIENT_DATA +- Graceful degradation for member-account SCP access and retention API availability +- Severity-rated findings (CRITICAL/HIGH/MEDIUM/LOW/INFO) +- Structured report output with priority matrix and verdict diff --git a/skills/bedrock-adoption-readiness/README.md b/skills/bedrock-adoption-readiness/README.md new file mode 100644 index 0000000..cede3eb --- /dev/null +++ b/skills/bedrock-adoption-readiness/README.md @@ -0,0 +1,70 @@ +# Bedrock Adoption Readiness Assessor + +Assess an AWS account's readiness to run Amazon Bedrock at production scale. Evaluates four dimensions: IAM governance, data retention (ZDR), quota and capacity headroom, and operational observability. Operates across both Standard Bedrock and Mantle (OpenAI models) surfaces. + +## What This Skill Does + +Runs a readiness assessment against your Bedrock configuration across both control planes and returns prioritized findings with specific remediation steps. Designed for teams preparing to scale Bedrock usage from experimentation to production. Covers four dimensions: IAM governance, data retention (ZDR), quota and capacity headroom, and operational observability - all API/metrics-driven with no dependency on customer configuration state. + +**Example prompts:** +- "Review my Bedrock readiness for production" +- "Check if my Bedrock quotas can handle our projected growth" +- "Assess my Bedrock IAM permissions and ZDR configuration" +- "Am I ready to scale Bedrock to production?" +- "Bedrock operational review" +- "Audit my Bedrock security posture" + +## Prerequisites + +### IAM Permissions Required + +The DevOps Agent role needs read-only access. Most required actions are covered by the [AIDevOpsAgentAccessPolicy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AIDevOpsAgentAccessPolicy.html) managed policy. + +Additional permissions that may be needed beyond the managed policy: +- `bedrock:GetAccountDataRetention` (for full D2 ZDR assessment - fallback available if not present) +- `iam:ListRolePolicies`, `iam:GetRolePolicy`, `iam:GetPolicyVersion` (for deep IAM policy inspection) + +**Note:** All operations are read-only. This skill does not modify any resources. + +### AWS Resources + +- At least one Bedrock model must have been invoked in the last 7 days for metrics-based analysis +- Organization-level SCP assessment requires the account to have Organizations API access (management account or delegated admin) + +## How to Use with DevOps Agent + +1. Upload this skill to your DevOps Agent Space +2. Select the "Chat tasks" subagent +3. Ensure the DevOps Agent role has the IAM permissions listed above +4. Ask natural language questions about Bedrock readiness + +**Subagents:** Chat tasks, Evaluation + +## Output + +The skill generates a structured assessment report with: +- Overall readiness verdict (READY / READY WITH ACTIONS / NOT READY) +- Per-dimension status (ASSESSED / NOT_ASSESSED / INSUFFICIENT_DATA) +- Findings by severity with specific remediation steps +- Model inventory with per-model utilization metrics +- Priority actions sorted by urgency + +## Key Features + +- **Dual-surface assessment**: Checks both Standard Bedrock (AWS/Bedrock) and Mantle (AWS/BedrockMantle) namespaces +- **Per-model quota comparison**: Compares each model's peak against its specific quota (not account aggregate) +- **Cache-aware arithmetic**: Accounts for prompt caching in quota calculations (cache reads don't consume quota) +- **Covered Model detection**: Identifies compliance-relevant retention implications from metrics alone +- **Graceful degradation**: Never silently skips a dimension - reports NOT_ASSESSED with reason when data is unreachable +- **Multi-region**: Discovers and assesses all regions with Bedrock activity + +## Limitations + +- Does not currently assess model selection fitness or cost optimization (requires model invocation logging to be enabled) +- Account-level data retention mode may require `bedrock:GetAccountDataRetention` permission not yet in standard DA policy (fallback available) +- Organization SCP assessment may be limited from member accounts without delegated admin access +- Mantle retention endpoint (`/v1/data_retention`) is a REST endpoint without SDK client - not directly assessable, uses indirect evidence + +## Disclaimer + +> This skill is sample code, not intended for production use without additional review and testing. Users should validate findings in a non-production environment first. Recommendations are based on AWS best practices as of the skill version date and may not reflect the latest service changes. diff --git a/skills/bedrock-adoption-readiness/SKILL.md b/skills/bedrock-adoption-readiness/SKILL.md new file mode 100644 index 0000000..8fda327 --- /dev/null +++ b/skills/bedrock-adoption-readiness/SKILL.md @@ -0,0 +1,312 @@ +--- +name: bedrock-adoption-readiness +description: > + Amazon Bedrock production readiness assessment covering IAM governance, data retention + (ZDR), quota and capacity headroom, and operational observability across Standard Bedrock + and Mantle surfaces. Use this skill when a user asks to review Bedrock readiness, assess + Bedrock security posture, evaluate quota headroom, check ZDR configuration, validate + Bedrock operational setup, or prepare for Bedrock production deployment. Triggers on + "Bedrock readiness review", "am I ready for Bedrock production", "Bedrock security + assessment", "check my Bedrock quotas", "Bedrock adoption audit", "Bedrock operational + review", or "assess my Bedrock environment". +metadata: + version: "1.0.0" + author: sruved + aws-devops-agent-skills.agent-types: "Chat tasks, Evaluation" + aws-devops-agent-skills.aws-services: "Amazon Bedrock" + aws-devops-agent-skills.technical-domains: "AI/ML" +--- + +# Bedrock Adoption Readiness Assessment + +Assess an AWS account's readiness to run Amazon Bedrock at production scale. Covers four dimensions: IAM governance, data retention (ZDR), quota and capacity headroom, and operational observability. Operates across both Standard Bedrock and Mantle (OpenAI models) surfaces. + +## Important: Two Surfaces + +Bedrock operates across two control planes. Both must be assessed. + +| | Standard Bedrock | Mantle (OpenAI models) | +|---|---|---| +| CloudWatch namespace | `AWS/Bedrock` | `AWS/BedrockMantle` | +| Metric names | `Invocations`, `InputTokenCount`, `OutputTokenCount` | `Inferences`, `TotalInputTokens`, `TotalOutputTokens` | +| CW dimensions | `ModelId` | `Model`, `Project` | +| IAM prefix | `bedrock:`, `bedrock-runtime:` | `bedrock-mantle:` | +| Cost discriminator | No marker in USAGE_TYPE | `-mantle-` substring in USAGE_TYPE | +| Cross-region inference | Yes - Geographic (`us.`, `eu.`, `apac.` prefixes) and Global (`global.` prefix) | No (in-region only) | + +## When to Use + +Activate this skill when the user asks to: +- Review or assess Bedrock production readiness +- Audit Bedrock IAM permissions or access governance +- Check Bedrock quota utilization or capacity planning +- Evaluate Zero Data Retention (ZDR) configuration +- Validate Bedrock operational monitoring setup +- Prepare for scaling Bedrock usage in production + +## Dimension States + +Every dimension reports one of three states: +- **ASSESSED**: Data collected, rules applied, findings produced +- **NOT_ASSESSED**: Collection failed or data unreachable (state the reason) +- **INSUFFICIENT_DATA**: Data returned but volume too low for meaningful analysis + +A dimension producing zero findings after successful collection = GOOD. A dimension producing zero findings because collection failed = NOT_ASSESSED. These must render differently. + +## Step 1: Identify Scope + +Ask the user: +- Which AWS account to assess +- Which regions to review (if unknown, discover by listing metrics in us-east-1, us-east-2, us-west-2) +- Whether they have specific concerns or want a full assessment + +## Step 2: Discover Active Regions and Models + +For each candidate region, check both namespaces: +- List metrics in `AWS/Bedrock` namespace +- List metrics in `AWS/BedrockMantle` namespace + +Any region returning metrics on either namespace is in scope. Record which ModelId/Model values appear - these identify active models for D2, D3, and later dimensions. + +## Step 3: Collect Data + +### 3.1 Standard Bedrock Metrics (7 days, per active region) + +Query without dimensions for aggregates: +- `Invocations` (Sum) +- `InvocationThrottles` (Sum) +- `InvocationLatency` (Average) +- `InputTokenCount` (Sum) +- `OutputTokenCount` (Sum) +- `CacheReadInputTokenCount` (Sum) - NOTE: correct metric name, NOT `CacheReadInputTokens` +- `CacheWriteInputTokenCount` (Sum) +- `InvocationServerErrors` (Sum) +- `InvocationClientErrors` (Sum) + +Then query WITH `ModelId` dimension for per-model quota utilization: +- `EstimatedTPMQuotaUsage` (Maximum) per ModelId discovered in Step 2 + +### 3.2 Mantle Metrics (7 days, per active region) + +Query at zero-dimension for aggregates: +- `Inferences` (Sum) +- `TotalInputTokens` (Sum) +- `TotalOutputTokens` (Sum) +- `InferenceClientErrors` (Sum) + +Then per-Model for attribution (TotalInputTokens supports Model dimension): +- `TotalInputTokens` with `Model` dimension per model discovered in Step 2 +- `TotalOutputTokens` with `Model` dimension + +For non-GPT-5.x models that emit `BurnDownConsumed`, query with `(Model, Project)` pair. + +### 3.3 Service Quotas + +List all quotas under `serviceCode: bedrock`. Paginate fully (can be 1,001+). + +Separate: +- Standard Bedrock quotas: names NOT prefixed with `[bedrock-mantle endpoint]` +- Mantle quotas: names prefixed with `[bedrock-mantle endpoint]` + +### 3.4 IAM (full depth) + +**Step A**: List all roles in the account. + +**Step B**: For each role, get attached managed policies AND inline policies. + +**Step C**: For inline policies, get the policy document directly. + +**Step D**: For managed policies, get the policy version document using the DefaultVersionId. + +Search all policy documents for: `bedrock:`, `bedrock-runtime:`, `bedrock-mantle:`, `bedrock-agentcore:`, and bare `*` in Action fields. + +Note: CDK/CloudFormation execution roles with `*` are expected. Flag as INFO, not CRITICAL. + +If policy documents cannot be retrieved (only metadata returned), mark D1 as NOT_ASSESSED: "IAM policy documents not retrieved." + +### 3.5 Alarms and Observability + +List all CloudWatch alarms in each active region. Identify which reference Bedrock metrics. + +Check model invocation logging configuration. If the API returns empty response, logging is DISABLED. + +Check CloudTrail event selectors for `bedrock-runtime.amazonaws.com` data events (management events alone do not capture model invocations). + +### 3.6 Guardrails + +List Bedrock guardrails. Zero guardrails on a production deployment using Standard Bedrock (`bedrock-runtime`) is a finding. Note: Guardrails are NOT available on the Mantle endpoint (`bedrock-mantle`). Do not flag missing guardrails for Mantle-only workloads. + +### 3.7 VPC Endpoints + +Check for VPC endpoints for all Bedrock services: +- `com.amazonaws..bedrock` (Control Plane) +- `com.amazonaws..bedrock-runtime` (Runtime) +- `com.amazonaws..bedrock-mantle` (Mantle/OpenAI) +- `com.amazonaws..bedrock-agent` (Agents Build-time) +- `com.amazonaws..bedrock-agent-runtime` (Agents Runtime) + +Ref: https://docs.aws.amazon.com/bedrock/latest/userguide/vpc-interface-endpoints.html + +### 3.8 SCPs (if accessible) + +List service control policies at the organization level. If access denied (member account), mark SCP check as NOT_ASSESSED: "organization-level access required." + +### 3.9 Data Retention (classic plane) + +Attempt to read account-level data retention configuration via `GetAccountDataRetention`. + +If the call succeeds, record the mode per region. If it fails (permission not in DA policy), use fallback: check if any Covered Model (Fable 5, Mythos 5) appears in the invoked ModelId list from Step 3.1. Covered Models cannot be invoked without provider data sharing being active - their presence in metrics IS retention evidence. + +## Step 4: Analyze - Four Dimensions + +### Dimension 1: IAM & Access Governance + +Analyze policy documents from Step 3.4: + +| Finding | Severity | +|---|---| +| Bare `*` Action on non-deployment role | CRITICAL | +| Any bedrock prefix with `*` resource on non-deployment role | HIGH | +| No SCP referencing Bedrock (regulated customer) | HIGH (or NOT_ASSESSED if org access unavailable) | +| Zero guardrails configured (Standard Bedrock workloads) | HIGH | +| No VPC endpoints for bedrock-runtime | MEDIUM | +| Broad permissions on CDK/deployment roles | INFO | + +### Dimension 2: Retention & Zero Data Retention (ZDR) + +Three retention regimes exist: + +| | Claude (standard) | Claude Covered Models (Fable 5, Mythos 5) | OpenAI GPT-5.x | +|---|---|---|---| +| ZDR obtainable? | Yes, self-service | NO - mandatory 30-day retention | Yes, must be granted | +| Scope | Set the mode | Cannot be changed | Per account, per model, per region | + +Assessment: + +1. If `GetAccountDataRetention` succeeded, report the mode per region +2. If Covered Model detected in invocation metrics, flag: provider data sharing is active +3. If GPT-5.x models present in Mantle metrics, note: ZDR must be explicitly granted per account/model/region +4. Check SCP results for retention enforcement policies + +| Finding | Severity | +|---|---| +| Covered Model invoked without documented awareness | CRITICAL (regulated) / HIGH (general) | +| GPT-5.x in use, ZDR status unknown | HIGH (regulated) / MEDIUM (general) | +| No SCP enforcing retention policy (regulated customer) | HIGH | +| Model invocation logging disabled | HIGH | +| Retention state unreadable | UNRESOLVED - flag for customer confirmation | + +### Dimension 3: Quota & Capacity Headroom + +Compare PER-MODEL utilization against PER-MODEL quotas. Never compare an account aggregate against a per-model limit. + +For Standard Bedrock: +- Use per-ModelId `EstimatedTPMQuotaUsage` from Step 3.1 +- Compare each model's 7-day peak against its specific quota from Step 3.3 +- Cache awareness: cache reads do NOT consume quota, cache writes DO + +For Mantle GPT-5.x (where `BurnDownConsumed` does not emit): +- Compute utilization from per-Model `TotalInputTokens + TotalOutputTokens` +- Compare against `[bedrock-mantle endpoint]` quota for that model +- Note: output tokens burn at 5:1 rate for Claude Opus/Sonnet 4.5+ (real consumption = InputTokenCount + CacheWriteInputTokenCount + OutputTokenCount x 5) + +| Finding | Severity | +|---|---| +| Any model's peak utilization >90% of its quota | CRITICAL | +| Any `InvocationThrottles` > 0 in 7 days | HIGH | +| Any model's peak >70% with growth trend | HIGH | +| No CRIS enabled + >50% utilization (Standard only, N/A Mantle) | MEDIUM | + +Note: CRIS is detectable via inference profile prefixes in ModelId - `us.`, `eu.`, `apac.` (geographic) or `global.` (global). Absence of any prefix means single-region only. + +If per-model quota data cannot be joined to metrics (naming mismatch), mark as INSUFFICIENT_DATA with available numbers shown. + +### Dimension 6: Operational Observability + +Check which monitoring is in place. Reference: CWR checklist `bedrock` v2.0 defines alarm checks with thresholds. Key checks: + +- Throttle alarm (InvocationThrottles > 0) +- Server error alarm (InvocationServerErrors as % of Invocations) +- Client error alarm (InvocationClientErrors as % of Invocations) +- Quota utilization alarm (EstimatedTPMQuotaUsage approaching limit) +- Latency alarm (InvocationLatency p90 by model family) +- Cost alarm + +Also check measured values against thresholds: +- If `InvocationServerErrors / Invocations > 1%` = active breach, HIGH +- If `InvocationClientErrors / Invocations > 5%` = active breach, HIGH + +| Finding | Severity | +|---|---| +| Measured metric actively breaching a threshold | HIGH | +| Zero Bedrock-related alarms configured | HIGH | +| No throttle monitoring | HIGH | +| Model invocation logging disabled | HIGH | +| CloudTrail data events not enabled for bedrock-runtime | MEDIUM | +| Missing cost alarm | MEDIUM | + +Note on metric names: The correct CloudWatch names are `InvocationThrottles` (not `ThrottledEvents`) and `EstimatedTPMQuotaUsage` (not `QuotaUtilization`). Some documentation uses alternate names that return zero datapoints. + +## Step 5: Generate Report + +Output format: + +``` +# Bedrock Adoption Readiness Assessment +Account: | Regions: | Date: +Surfaces: Standard Bedrock [Y/N] | Mantle [Y/N] +7-Day Volume: invocations | 30-Day Spend: $ +Assessment: READY / READY WITH ACTIONS / NOT READY + +## Dimension Status +- D1 IAM: [ASSESSED / NOT_ASSESSED: reason] +- D2 Retention: [ASSESSED / UNRESOLVED: needs confirmation] +- D3 Quota: [ASSESSED / INSUFFICIENT_DATA] +- D6 Observability: [ASSESSED] + +## Findings +[Sorted by severity, then dimension] + +| # | Finding | Severity | Dimension | Recommendation | +|---|---------|----------|-----------|----------------| + +## Model Inventory +| Model | Surface | Region | 7-Day Invocations | TPM Peak | Quota | % Used | +|-------|---------|--------|-------------------|----------|-------|--------| + +## Priority Actions +1. [Highest severity + remediation] +2. [Next] +3. [Next] + +## Next Steps +- Immediate (CRITICAL) +- This week (HIGH) +- This month (MEDIUM) +``` + +Verdict thresholds: +- **READY**: 0 critical, <=2 high, 0 unresolved, 0 NOT_ASSESSED +- **READY WITH ACTIONS**: 0 critical, (>2 high OR any unresolved OR any NOT_ASSESSED) +- **NOT READY**: Any critical finding + +## Severity Definitions + +| Severity | Definition | SLA | +|----------|-----------|-----| +| CRITICAL | Immediate risk to security, data exposure, or unbounded cost | Fix before production use | +| HIGH | Significant gap that will cause issues at scale | Fix within 1 week | +| MEDIUM | Notable improvement opportunity | Plan within 30 days | +| LOW | Minor optimization | Address when convenient | +| INFO | Observation, no action required | N/A | + +## References + +- Bedrock Security: https://docs.aws.amazon.com/bedrock/latest/userguide/security.html +- Bedrock Quotas: https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html +- Cross-Region Inference: https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html +- Model Invocation Logging: https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html +- Inference Profiles: https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles.html +- Data Retention: https://docs.aws.amazon.com/bedrock/latest/userguide/data-retention.html +- Enforce ZDR with SCPs: https://aws.amazon.com/blogs/security/enforce-zero-data-retention-on-amazon-bedrock-with-bedrock-projects-and-service-control-policies/ +- CloudWatch Metrics TTFT & EstimatedTPMQuotaUsage: https://aws.amazon.com/blogs/machine-learning/improve-operational-visibility-for-inference-workloads-on-amazon-bedrock-with-new-cloudwatch-metrics-for-ttft-and-estimated-quota-consumption/ diff --git a/skills/bedrock-adoption-readiness/evals/eval_queries.json b/skills/bedrock-adoption-readiness/evals/eval_queries.json new file mode 100644 index 0000000..479c692 --- /dev/null +++ b/skills/bedrock-adoption-readiness/evals/eval_queries.json @@ -0,0 +1,62 @@ +{ + "skill_name": "bedrock-adoption-readiness", + "version": "1.0.0", + "trigger_queries": [ + { + "query": "Review my Bedrock readiness for production", + "should_trigger": true + }, + { + "query": "Assess my Bedrock IAM permissions and ZDR setup", + "should_trigger": true + }, + { + "query": "Check if my Bedrock quotas can handle growth", + "should_trigger": true + }, + { + "query": "Am I ready to scale Bedrock?", + "should_trigger": true + }, + { + "query": "Bedrock operational review", + "should_trigger": true + }, + { + "query": "Audit my Bedrock security posture", + "should_trigger": true + }, + { + "query": "Help me deploy a Lambda function", + "should_trigger": false + }, + { + "query": "Review my EKS cluster configuration", + "should_trigger": false + }, + { + "query": "What is Amazon Bedrock?", + "should_trigger": false + }, + { + "query": "Help me write a prompt for Claude", + "should_trigger": false + }, + { + "query": "Check my S3 bucket permissions", + "should_trigger": false + }, + { + "query": "Troubleshoot my RDS connection timeout", + "should_trigger": false + }, + { + "query": "How do I enable Bedrock in a new region?", + "should_trigger": false + }, + { + "query": "Review my EC2 cost optimization opportunities", + "should_trigger": false + } + ] +} diff --git a/skills/bedrock-adoption-readiness/evals/evals.json b/skills/bedrock-adoption-readiness/evals/evals.json new file mode 100644 index 0000000..f87eb91 --- /dev/null +++ b/skills/bedrock-adoption-readiness/evals/evals.json @@ -0,0 +1,71 @@ +{ + "skill_name": "bedrock-adoption-readiness", + "version": "1.0.0", + "evals": [ + { + "id": "full-assessment", + "description": "User requests a complete Bedrock readiness review", + "prompt": "I'm preparing to move our Bedrock usage to production. Can you review our readiness?", + "assertions": [ + "Asks which account and region to assess", + "Checks IAM permissions for Bedrock", + "Evaluates data retention / ZDR configuration", + "Checks quota utilization against limits", + "Reviews CloudWatch alarms for Bedrock", + "Produces findings with severity ratings", + "Generates a structured report" + ] + }, + { + "id": "quota-focused", + "description": "User specifically asks about quota headroom", + "prompt": "We're getting throttled on Bedrock. Can you check our quota situation and tell me what to do?", + "assertions": [ + "Checks current TPM and RPM quotas", + "Pulls 7-day CloudWatch throttling metrics", + "Checks max_tokens configuration pattern", + "Evaluates CRIS enablement", + "Recommends right-sizing max_tokens if overconsumption detected", + "Recommends quota increase if utilization >70%" + ] + }, + { + "id": "security-focused", + "description": "User asks about Bedrock security and data handling", + "prompt": "We're in healthcare and need to make sure our Bedrock setup meets compliance requirements. Can you check our ZDR and IAM configuration?", + "assertions": [ + "Checks inference profile ZDR mode", + "Flags if provider_data_share is enabled without awareness", + "Reviews IAM policies for wildcarded Bedrock permissions", + "Checks for SCP enforcement at org level", + "Flags regulated industry context in severity assessment", + "Recommends ZDR mode 'none' for regulated workloads" + ] + }, + { + "id": "cost-optimization", + "description": "User asks about Bedrock cost optimization", + "prompt": "Our Bedrock bill is growing fast. What can we optimize?", + "assertions": [ + "Pulls 30-day Cost Explorer data for Bedrock", + "Identifies daily spend trend", + "Checks model selection fitness (expensive models for simple tasks)", + "Evaluates prompt caching eligibility", + "Checks batch inference opportunity", + "Recommends Savings Plan if spend threshold met" + ] + }, + { + "id": "observability-check", + "description": "User asks about monitoring setup", + "prompt": "How do I know if something goes wrong with our Bedrock usage? Do we have proper monitoring?", + "assertions": [ + "Checks existing CloudWatch alarms for Bedrock metrics", + "Verifies model invocation logging status", + "Checks CloudTrail Bedrock event configuration", + "Recommends specific alarm set if gaps found", + "Flags if no throttle monitoring exists" + ] + } + ] +}