Skip to content

[SOURCE-324] Harden AI review workflow - #29

Draft
xopham wants to merge 3 commits into
mainfrom
christoph.hamsen/SOURCE-324/harden-ai-review-workflow
Draft

[SOURCE-324] Harden AI review workflow#29
xopham wants to merge 3 commits into
mainfrom
christoph.hamsen/SOURCE-324/harden-ai-review-workflow

Conversation

@xopham

@xopham xopham commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • pin reusable-workflow support code, trusted guidelines, action revisions, Codex/Gemini runtimes, the Gemini sandbox image, and PR commit inputs
  • generate and size-check a complete local diff for the gate-pinned SHAs, with stale-PR checks before provider execution and review submission
  • retain the pinned anthropics/claude-code-action agent flow with read-only tools, project/local settings disabled, a 10-turn ceiling, and schema-backed output that is rejected rather than recovered from execution logs
  • harden Codex and Gemini with read-only tool boundaries, minimal environments, package/bootstrap controls, and fail-closed output handling
  • ensure the write-capable post job treats provider artifacts only as bounded data and independently checks out its trusted validator
  • make fork reviews comment-only, fetch fork heads through the base pull ref, and report superseded runs as cancelled instead of failed

Security model

Model execution remains isolated from GitHub write permissions. Trusted code and schemas come from the exact reusable-workflow revision; review guides come from a gate-pinned default-branch commit; PR content and provider output are treated as untrusted data. The post job performs strict shape validation and a second secret/anomaly scan before calling GitHub.

review_event: COMMENT_ONLY remains the secure default. ALL is an explicit relaxation for same-repository PRs and is documented as such. Gemini still requires network access to reach its API, but exposes no model-callable network, shell, write, MCP, or extension tool.

Reliability

  • rejects moved or closed PRs instead of posting stale reviews
  • fails closed for incomplete changed-file lists and oversized complete diffs
  • bounds provider execution and artifact sizes
  • avoids false technical-failure comments and reactions when concurrency cancels a superseded run
  • reports provider failures through a structured warning review when possible

Validation

  • npm test (101 passing, 99.39% statement coverage)
  • actionlint 1.7.12 (with the compatibility ignore for GitHub's documented job.workflow_repository / job.workflow_sha contexts)
  • YAML parse and git diff --check
  • independent security review of the restored Claude provider boundary

Operational notes

A live reusable-workflow run with real provider secrets was not performed. The pinned Claude action installs its fixed CLI version through Anthropic's mutable installer endpoint, so the action SHA does not pin the installer response. Datadog's internal guidance also requires coordinating direct-provider usage and costs because this path does not use centralized AI Gateway observability.

Ticket

SOURCE-324

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Jul 28, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d805a09 | Docs | Datadog PR Page | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant