Skip to content

refactor(session-ingest): prepare direct ingest path#4472

Open
pandemicsyn wants to merge 2 commits into
mainfrom
florian/chore/session-ingest-direct-prep
Open

refactor(session-ingest): prepare direct ingest path#4472
pandemicsyn wants to merge 2 commits into
mainfrom
florian/chore/session-ingest-direct-prep

Conversation

@pandemicsyn

@pandemicsyn pandemicsyn commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

This preparation PR extracts the parser/validation, metadata projection, staging cleanup, and tombstone handling that will be needed by the follow-up ,while keeping the existing R2 + queue ingest path in place. Basically, breeaking out all the bits that ill need/reuse.

PR 2 will add a gated direct path for the conservative common case: requests with a known body size under 4 MiB that fully validate, contain at most 128 valid items, and require no R2 item offloading will be written to SessionIngestDO in one synchronous RPC. A 200 will then mean the items are immediately available to exports and cloud-agent restores. Missing/oversized/multi-chunk payloads and direct RPC failures will continue through the existing R2 + queue path.

PR2 will ship with the bypass behind a user allowlist and percentage gate with path/fallback observability so that we can slowly roll it out, without changing the existing metadata or lifecycle consistency model.

  • Add a side-effect-free ingest parser and validator that distinguishes missing, malformed, and wrong-shaped payloads while reporting valid-item count and byte budgets.
  • Extract metadata projection and R2 staging/queue cleanup into reusable ingest modules while preserving the existing queue parser and post-staging ingestedAt ordering.
  • Add a version-skew-safe Durable Object tombstone result and make queue processing acknowledge tombstoned sessions while still deleting staging objects.
  • Centralize the existing one-RPC item and byte limits for the later direct-ingest implementation.

Verification

  • No manual runtime verification was performed because this preparation PR intentionally leaves the request path and observable queue behavior unchanged.

Visual Changes

N/A

Reviewer Notes

  • This is preparation only: it does not add a direct ingest route, rollout gate, configuration, migration, or client change.
  • Strict structural JSON validation is opt-in for the new buffered validator; the existing queue path retains lexical parser behavior.
  • Legacy staging captures ingestedAt after the R2 upload, matching existing conflict ordering. PR 2 will explicitly pass request-arrival time for direct fallback.
  • Tombstone handling cancels remaining streaming work, avoids duplicate follow-up RPCs, and still deletes the staging object before acknowledgment.

@pandemicsyn pandemicsyn marked this pull request as ready for review July 9, 2026 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants