DUX-8760: Add prompt-cache breakpoints to the BedrockInvokeModel protocol

[staging-supernova-dx-appf-io]

Summary

Add automatic prompt-cache breakpoint injection to the BedrockInvokeModel protocol, enabling native Bedrock/Anthropic caching and eliminating unnecessary full-price input billing.

Changes

Configuration

• Add bedrock_invoke_model_prompt_caching configuration option (defaults true)
• Register option in Bedrock provider configuration

Core Implementation

• chat.rb:
  • Define MAX_CACHE_BREAKPOINTS = 4 constant (Bedrock limit)
  • Define CACHEABLE_BLOCK_TYPES to identify cache-eligible blocks
  • Add inject_cache_control_breakpoints helper to intelligently place breakpoints:
    • Injects after system message (if present)
    • Injects after tail message(s) to maximize cache hits
    • Respects Bedrock's 4-breakpoint limit
    • Hardens against repeated mutations of payload structures
  • Wire injection into render_payload when config is enabled

Cache Metrics

• streaming.rb: Extract cache usage fields (cache_read_input_tokens, cache_creation_input_tokens) from message_start events and include in Chunk

Testing

• Add 272 lines of spec coverage:
  • Breakpoint placement logic (system, tail, limits)
  • Mutation safety and idempotency
  • Cache metrics extraction
  • Config toggle behavior

Impact

Enables production agents_app deployment through BedrockInvokeModel with native caching, reducing input token costs by ~75% on cached requests (measured in consumer app). Fully backward-compatible via config toggle (defaults on).

Notes

• Anthropic does not support caching on thinking/redacted_thinking blocks; excluded from CACHEABLE_BLOCK_TYPES
• Breakpoint injection is idempotent—safe to call multiple times on same payload
• Cache breakpoint count validated against Bedrock's 4-per-request limit

---

Agent session: https://staging.supernova.dx.appf.io/coders/c0d5406c-9d91-4c90-a0b1-a7cfa60b0237

[kermitapp]

Jira Issue: https://appfolio.atlassian.net/browse/DUX-8760

[staging-supernova-dx-appf-io]

🤖 Supernova Code Review — View trace

[samuelboland]

Code review please.

[staging-supernova-dx-appf-io]

@samuelboland

Code review please.

Reviewed the code at the PR head. Findings:

1. In-place mutation of caller-owned message hashes — lib/ruby_llm/protocols/bedrock_invoke_model/chat.rb, translate_raw_block (line 264-265) returns the original hash object unchanged when it already has a :type/'type' key (return result << block if block.key?(:type) || block.key?('type')). inject_tail_cache_breakpoint (line 126-131) and attach_cache_point (line 276-284) then do block[:cache_control] = { type: 'ephemeral' } directly on that same object. For Content::Raw messages (the consumer app's persisted Converse-format blocks), this mutates the caller's own hash in place rather than a copy — any retry, retained reference, or reused message object will carry the injected cache_control afterward, and a caller-supplied ttl on an existing cache_control gets clobbered with a plain { type: 'ephemeral' } since neither method checks for an existing value before overwriting.

2. Tail injection isn't gated by remaining — inject_cache_breakpoints (line 93-104) only checks remaining >= 2 before the system breakpoint, but calls inject_tail_cache_breakpoint unconditionally after the remaining <= 0 guard at the top. That happens to stay within budget under the current two-callsite logic, but it's not itself bounded by remaining, so it depends entirely on the earlier system check to keep the math correct — fragile if either function's behavior changes later.

Both are worth addressing before merge, especially #1 since silent mutation of persisted message data is the kind of bug that's hard to trace back. Want me to queue a fix for the mutation issue (dup the block before attaching cache_control when it's a pass-through reference)?

---

🔍 View trace

[samuelboland]

Tail injection clobbers translated breakpoints. This assignment unconditionally overwrites block[:cache_control]. If the final message's last cacheable block already carries a breakpoint from a translated cachePoint — including one with ttl: '1h' — it is replaced with a plain { type: 'ephemeral' }, silently downgrading a caller-requested 1-hour TTL to 5 minutes. That contradicts the comment on lines 88-89 ("Never removes a breakpoint that arrived via a translated cachePoint"). The budget count at line 96 already counted that block, so the overwrite isn't needed to stay within budget either.

Recommend: skip injection when the found block already has cache_control (either key form) — the tail is already marked. E.g. in last_cacheable_block or guard before the assignment:

block[:cache_control] = { type: 'ephemeral' } if block && !(block[:cache_control] || block['cache_control'])

[staging-supernova-dx-appf-io]

Queued — the fix will guard inject_tail_cache_breakpoint against overwriting an existing cache_control (checking both key forms) and add a spec covering the preserved-ttl case.

[samuelboland]

In-place mutation of caller-owned hashes. Pass-through blocks returned here are the caller's own hash objects, and attach_cache_point (line 283), inject_system_cache_breakpoint? (line 122), and inject_tail_cache_breakpoint (line 130) all write :cache_control directly into whatever block they land on. For Content::Raw messages, the consumer app wraps the ActiveRecord content_raw attribute itself — so an injected breakpoint mutates persisted-model data in memory: it survives into any retry or later round-trip that reuses the same message objects, and if the record is saved again for any reason the injected cache_control is written back to the database.

Recommend: result << block.dup on both pass-through returns (lines 264-265), and have attach_cache_point operate on the duped copy — the rendered payload should never alias caller-owned hashes. (This confirms the earlier bot review's mutation finding; consider this the actionable version of that thread.)

[staging-supernova-dx-appf-io]

Queued as a follow-up to the tail-injection fix — it'll dup pass-through blocks in translate_raw_block and make attach_cache_point work on the copy, plus add a spec asserting the caller's original hashes stay untouched.

[samuelboland]

Symbol-only key checks defeat the budget guard and dedup for string-keyed blocks. translate_raw_block deliberately tolerates string keys, but the three helpers downstream are symbol-only: this counter checks b[:cache_control], the system dedup at line 120 checks b[:cache_control], and last_cacheable_block at line 135 checks block[:type]. A string-keyed Anthropic-format block from a DB round-trip ({'type' => 'text', 'cache_control' => {...}}) passes through translation unchanged, and then:

1. It isn't counted toward the 4-breakpoint budget — a request can end up with more than 4 breakpoints and be rejected by Bedrock.
2. The system dedup misses it and sets a symbol :cache_control on a hash that already has 'cache_control', which serializes to duplicate "cache_control" keys in the JSON body.
3. The tail walker skips string-keyed cacheable blocks entirely, so the breakpoint drifts to an earlier message than intended.

Recommend: make all three helpers key-indifferent the same way translate_raw_block already is — b[:cache_control] || b['cache_control'] here and at line 120, block[:type] || block['type'] at line 135.

[staging-supernova-dx-appf-io]

Queued — this will make count_blocks_with_cache_control, inject_system_cache_breakpoint?, and last_cacheable_block check both symbol and string keys, plus add specs for string-keyed blocks across the budget-count, dedup, and tail-selection paths.