fix(pr-risk): a repo-owned GitHub App is a runbook candidate, not external (BE-6715)

.github/workflows/pr-risk.yml

@@ -258,8 +258,13 @@ on:
       bot_logins:
         description: >-
           Extra logins treated as bots, on top of any `[bot]`-suffixed login
-          (comma-separated). A bot with no runbook registry entry grades as
-          human — identity alone never buys trust.
+          and any author GitHub itself types as a `Bot` (comma-separated). A
+          real GitHub App needs no entry here; a machine USER account does. A
+          bot with no runbook registry entry grades as human — identity alone
+          never buys trust. LOAD-BEARING, not a hint: a listed login skips the
+          first-time-contributor test, moving a non-fork NONE PR from `external`
+          (R3) to `human` (R1), and nothing validates that the login really is a
+          machine account. List only accounts you control; prune retired ones.
         type: string
         required: false
         default: github-actions,dependabot,renovate,coderabbitai,cursor,comfy-pr-bot,web-flow

docs/callers/pr-risk.md

@@ -78,8 +78,8 @@ statuses: read
 | Input | Default | Notes |
 |---|---|---|
 | `workflows_ref` | — (**required**) | Pin to the SAME full commit SHA as `uses:`. No default on purpose: a floating default let a caller SHA-pin `uses:` and still load the grader from HEAD of main. |
-| `fleet_logins` | `mattmillerai` | Logins whose PRs grade provenance `agent-supervised` alongside `agent-coded`. |
-| `bot_logins` | `github-actions,dependabot,renovate,coderabbitai,cursor,comfy-pr-bot,web-flow` | Extra logins treated as bots. A bot with no runbook entry still grades as human — identity alone buys no trust. |
+| `fleet_logins` | `mattmillerai` | Logins whose PRs grade provenance `agent-supervised` alongside `agent-coded`. Both are read for **human** authors only: an author GitHub types as a `Bot` is a runbook candidate regardless, so listing a bot here (or labelling its PR) buys it nothing — only a registry entry that asserts can promote it. |
+| `bot_logins` | `github-actions,dependabot,renovate,coderabbitai,cursor,comfy-pr-bot,web-flow` | Extra logins treated as bots. Needed only for **machine USER accounts** — a real GitHub App is recognized from GitHub's own actor type, no list entry required. A bot with no runbook entry still grades as human — identity alone buys no trust. **This list is load-bearing, not a hint:** a listed login skips the first-time-contributor test, so it moves a non-fork `NONE`/`FIRST_TIME_CONTRIBUTOR` PR from `external` (R3) to `human` (R1). Nothing validates that a listed login is really a machine account, so add one only for an account you control, and remove it when it is retired. |
 | `label_map` | `''` | Rename the five grader-owned labels as `tier=label` pairs. Tier keys are fixed; only the label text is yours. |
 | `wait_for_checks_minutes` | `10` | How long to wait for the rest of the check rollup to settle before labeling (clamped to 25 — what a 30-minute job can spend waiting). `0` labels immediately, expect R2 floors from still-pending checks. |
 | `repo_map_path` | `.github/risk.json` | Consumer risk-map override, read from the PR **base ref**. |

scripts/pr-risk/README.md

@@ -34,9 +34,16 @@ record.
 2. **Provenance** — what PROCESS produced the diff: `runbook` (a registered
    producer in [`runbook-registry.v0.json`](runbook-registry.v0.json) whose
    identity AND diff shape both assert), `agent-supervised`, `human`, or
-   `external` (fork / first-time contributor — R3, no exceptions, even when a
-   runbook shape matches). Identity is the server-attributed author login,
-   never the forgeable commit author string.
+   `external` (any fork, or a first-time **human** contributor — R3, no
+   exceptions, even when a runbook shape matches). Identity is the
+   server-attributed author login, never the forgeable commit author string.
+   A **non-fork bot is never `external`**, even though every GitHub App authors
+   with `author_association: NONE` — it is a runbook candidate, judged by the
+   registry's shape assertion, and falls back to `human` when no entry asserts.
+   "Is this a bot?" is answered from GitHub's own actor type (`Bot`) plus the
+   `bot_logins` list, so an App needs no list entry and a machine *user* account
+   does. The fork half is unconditional, so a bot opening a PR *from a fork* is
+   still `external` R3.
 3. **Reversibility** — mutates persistent state or deletes data → R3; **removes**
    a file under a sensitive class → R3 (a delete, or a rename out of that class);
    no green check rollup → R2; green but no test file touched → R1; green with
@@ -177,7 +184,14 @@ them by committing:
 - `.github/risk.json` — the repo's own path→tier map (same schema as
   [`risk-map.v0.json`](risk-map.v0.json))
 - `.github/risk-runbooks.json` — the repo's own producer registry (same schema
-  as [`runbook-registry.v0.json`](runbook-registry.v0.json))
+  as [`runbook-registry.v0.json`](runbook-registry.v0.json)). **Name a GitHub App
+  by its suffixed login** — `"my-bot[bot]"`, the form REST and the web UI show
+  you. The grader reads the PR over GraphQL, which reports a Bot login
+  **unsuffixed**, and restores the suffix before matching — one way only, and
+  only for an author GitHub's own actor type calls a `Bot`. A bare slug
+  (`"my-bot"`) still matches literally, because a machine **user** account has no
+  suffix and needs it; but a bare slug is equally matchable by a same-named
+  *human*, who would then inherit that runbook — so never name an App with one.
 
 Both are read from the PR's **base ref**, so a PR cannot edit the rules that
 judge it (editing them — or the grader — at all is R3 by the map's own first

scripts/pr-risk/grade-pr-risk.sh

@@ -30,7 +30,9 @@
 #     is the WORST tier over every rule any changed path matches.
 #   AXIS 2 — PROVENANCE. runbook / agent-supervised / human / external. A PR whose identity
 #     matches a runbook but whose DIFF SHAPE does not is not a runbook, and falls back to its
-#     underlying class. `external` (fork / first-time contributor) is R3 on provenance alone.
+#     underlying class. `external` (any fork, or a first-time HUMAN contributor) is R3 on
+#     provenance alone; a non-fork bot is a runbook candidate, not an outsider, because every
+#     GitHub App authors with `author_association: NONE`.
 #   AXIS 3 — REVERSIBILITY. Single clean revert? Mutates persistent state or deletes data?
 #     Did tests covering the touched lines actually run? Answered from the changed-path list
 #     + change types + the PR's own check rollup.
@@ -126,8 +128,8 @@ read_map() { # <file> <kind: map|runbooks> -> JSON on stdout, rc 0; rc 1 + reaso
         then "provenance_tiers carries a tier outside \(known)"
       # EVERY provenance class must be MAPPED, not just well-typed. Checking only the values let
       # a map that OMITS `external` pass, and the lookup then fell back to a tier of its own
-      # choosing — silently retiring the "external (fork / first-time contributor) is R3, no
-      # exceptions" invariant that both the default map comment and the README promise. A class
+      # choosing — silently retiring the "external (a fork, or a first-time human contributor) is
+      # R3, no exceptions" invariant that the default map comment and the README promise. A class
       # nobody mapped is a routing decision nobody made, so it is refused at load time.
       # (No apostrophes in here: the whole shape program is a single-quoted shell string.)
       elif ((["runbook","agent-supervised","human","external"] - [.provenance_tiers | keys[]]) | length) > 0
@@ -252,6 +254,32 @@ cat <<'JQ'
     # runbook — provenance alone is never sufficient.
     | ($r.author // null) as $author
     | ($author | classify_login($fleetl; $botl)) as $cls
+    # IS THE AUTHOR A BOT? The shared resolver answers from the login STRING — a `[bot]` suffix or
+    # the caller's `bot_logins` list — and on the live path that string is not enough on its own.
+    # GraphQL reports a Bot actor's login WITHOUT the `[bot]` suffix (`cloud-code-bot`, where REST
+    # and the web UI both show `cloud-code-bot[bot]`), so the suffix test can only ever fire on a
+    # REST-shaped or offline-corpus record. `author.__typename == "Bot"` is GitHub's OWN actor
+    # type for the same account and is not forgeable by the PR, so it is collected alongside the
+    # login and ORed in here. It is additive: a record without the field (every pre-schema-4
+    # corpus row) falls back to the resolver's answer exactly as before, and `bot_logins` still
+    # works for machine USER accounts, which are typed `User` and carry no suffix at all.
+    # Deliberately kept OUT of classify_login: that jq is shared verbatim with agent-work's
+    # grade-collect.sh, and a field only this collector emits would fork the shared definition.
+    #
+    # `== true`, NOT jq truthiness. This is the one field that can switch the `external` guard
+    # off, and in jq the string "false", 0 and "" are all truthy — so a foreign collector, or a
+    # corpus row someone hand-edited, could turn a stringly-typed "false" into a confident
+    # not-an-outsider. Only the literal boolean counts; anything else falls back to the login.
+    #
+    # NO STATUS TWIN HERE, deliberately, and it is the only provenance input without one. The
+    # twins exist where the un-collected default would ANSWER — reading an absent `is_fork` as
+    # "not a fork" retires `external => R3`. This default runs the other way: absent means "not
+    # known to be a bot", which sends the author BACK through the association test and grades it
+    # the stricter way, exactly as a pre-schema-4 record graded before. Nothing is retired, so
+    # there is nothing to refuse to answer. A schema-3 corpus row therefore still grades an App
+    # PR `external` where a live schema-4 grade now says `human`: that is a versioned behaviour
+    # change, which is what `schema_version` and `map_version` are on the record to express.
+    | ((($r.author_is_bot // false) == true) or $cls == "bot") as $is_bot
     | (($r.labels // []) | index("agent-coded") != null) as $agent_coded
     # The label list has a STATUS TWIN for the same reason the file list does: `agent-coded` is
     # read from it, and a TRUNCATED label list answers "is this agent-coded?" with a confident no
@@ -264,11 +292,47 @@ cat <<'JQ'
     # class never routed unattended — silently unreachable. Unread is `unknown`, and `unknown`
     # refuses to grade the axis.
     | ($r.provenance_status // (if ($r | has("is_fork")) then "ok" else "absent" end)) as $pvst
+    # ORDER IS THE RULE HERE, AND IT IS NOT REARRANGEABLE. The fork test runs FIRST and stays
+    # unconditional: a fork PR is `external` no matter who authored it, so a bot on a fork can
+    # never escape R3 by presenting a bot login. Only AFTER that does the author-association half
+    # get narrowed to authors the shared resolver does NOT read as a bot.
+    #
+    # WHY THE NARROWING. A GitHub App is never an org member, so EVERY PR opened by a repo-owned
+    # App arrives with `author_association: NONE` — and testing that string before the login
+    # classification pinned every such PR `external` => R3 regardless of what it changed (measured
+    # on one consumer: 14 of 19 R3 grades in a 24-PR sample came from this, not from the diff).
+    # It was also unfixable from the consumer side, because both levers sit further down: the
+    # `bot_logins` input feeds `classify_login`, and `.github/risk-runbooks.json` is read by the
+    # shape assertion below. A non-fork bot now falls through to `runbook-candidate`, where the
+    # registry and its shape assertion decide — and a `runbook-candidate` with no asserting entry
+    # still lands on `human` below, so identity alone continues to buy no trust.
+    # The first-time-contributor guard is UNCHANGED for humans: `$cls` is "human" (or "unknown")
+    # for anyone the resolver does not classify as a bot, so a human's NONE still reads external.
+    #
+    # AND THE BOT TEST RUNS BEFORE THE LABEL/FLEET TEST, for the same reason `classify_login`
+    # answers "bot" before it answers "fleet": a bot is a bot no matter what else is true of it.
+    # Both halves of that branch are reachable for a bot and neither is evidence about a bot:
+    #   * `agent-coded` is a LABEL — anyone with write access applies it, and what it claims is
+    #     that a human supervised an agent. That is a claim about a human author, not about an App.
+    #   * `$cls == "fleet"` is reachable for a bot at all only because `$is_bot` no longer comes
+    #     from the login string alone: a Bot actor whose GraphQL login arrives unsuffixed
+    #     (`cloud-code-bot`) and also appears in `fleet_logins` classifies "fleet" while
+    #     `author_is_bot` says Bot. Leaving `fleet` first lets this use site contradict the
+    #     resolver's own documented bot-beats-fleet ordering.
+    # What this does NOT change: a REGISTERED producer was never at risk here, because the shape
+    # assertion below resolves `$rbk != null` to `runbook` ahead of `$base_class` either way.
+    # What it fixes is the UNREGISTERED bot, which either half would classify `agent-supervised`
+    # — contradicting this axis's stated promise (and the map's `_why`) that a non-fork bot with
+    # no asserting entry falls back to `human`. The default map tiers both R1, so nothing moves
+    # there; the two classes exist so a CONSUMER map can tier them apart, and a consumer that
+    # trusts its own supervised agents at R0 would otherwise hand R0 to any bot PR someone
+    # labelled `agent-coded`. Trust in this axis is earned by the shape assertion, never by a label.
     | (if $pvst != "ok" or $lbst != "ok" then "unknown"
-       elif ($r.is_fork // false) or (($r.author_association // "") | IN("FIRST_TIME_CONTRIBUTOR","FIRST_TIMER","NONE"))
+       elif ($r.is_fork // false) then "external"
+       elif ($is_bot | not) and (($r.author_association // "") | IN("FIRST_TIME_CONTRIBUTOR","FIRST_TIMER","NONE"))
        then "external"
+       elif $is_bot then "runbook-candidate"
        elif $agent_coded or $cls == "fleet" then "agent-supervised"
-       elif $cls == "bot" then "runbook-candidate"
        elif $cls == "human" then "human"
        else "unknown" end) as $base_class
     # The shape assertion: identity match AND every changed path inside permitted_paths AND the
@@ -281,8 +345,23 @@ cat <<'JQ'
     # login test is therefore REQUIRED, and head_ref_patterns are an ADDITIONAL condition where
     # the producer declares them. The parentheses are load-bearing — do not let this collapse
     # back into a bare or/and chain.
+    #
+    # AND THE APP'S SUFFIXED LOGIN IS RESTORED BEFORE MATCHING, in ONE direction only. GraphQL
+    # hands us `cloud-code-bot` where REST and the web UI say `cloud-code-bot[bot]`, so without
+    # this an entry listing the form a human can actually SEE never asserts on a live grade, and
+    # the miss is silent — `shape_failures` only records entries whose identity DID match. The
+    # direction matters and is the whole safety argument: the suffixed form is SYNTHESIZED, never
+    # stripped, and only when `author_is_bot` — GitHub's own actor type, not the caller's
+    # `bot_logins` — says the author is a Bot. So an App reaches an `"x[bot]"` entry, and a USER
+    # account named `x` still cannot: it can only ever present `x`, and nothing here turns
+    # `"x[bot]"` into `"x"`. That asymmetry is why this is safe where normalizing both sides is
+    # not. Consequence for registries: an App should be listed by its SUFFIXED login alone. A
+    # bare slug still matches literally — machine USER accounts have no suffix and need it — but
+    # a bare slug is also matchable by a same-named human, so never use one to name an App.
+    | (if $author != null and (($r.author_is_bot // false) == true) and ($author | endswith("[bot]") | not)
+       then [$author, ($author + "[bot]")] else [$author] end) as $author_forms
     | ([$RB.runbooks[]? | . as $bk
-        | select((($bk.identity.logins // []) | any(. as $l | ($author // "") | ascii_downcase == ($l | ascii_downcase)))
+        | select((($bk.identity.logins // []) | any(. as $l | $author_forms | any(. // "" | ascii_downcase == ($l | ascii_downcase))))
                  and ((($bk.identity.head_ref_patterns // []) | length) == 0
                       or (($r.head_ref // "") | matches_any($bk.identity.head_ref_patterns // []))))
         | {id: $bk.id, lane: $bk.lane, daily_cap: $bk.daily_cap,
@@ -467,7 +546,7 @@ fetch_pr_record() { # <repo> <num> -> record JSON on stdout, rc 1 on an unreadab
   q='query($owner:String!,$name:String!,$num:Int!){
   repository(owner:$owner,name:$name){ pullRequest(number:$num){
     number title state isDraft createdAt updatedAt closedAt mergedAt
-    author{ login } authorAssociation baseRefName headRefName isCrossRepository
+    author{ login __typename } authorAssociation baseRefName headRefName isCrossRepository
     additions deletions changedFiles
     labels(first:100){ pageInfo{ hasNextPage } nodes{ name } }
     commits(last:1){ nodes{ commit{ oid statusCheckRollup{ state
@@ -613,7 +692,14 @@ fetch_pr_record() { # <repo> <num> -> record JSON on stdout, rc 1 on an unreadab
        then {list: null, status: "unknown",
              reason: "changed-file list is short (\($files | length) of \(.changedFiles)) — GitHub caps the files API at 3000 files"}
        else {list: $files, status: "ok", reason: null} end) as $fread
-    | {schema_version:3, repo:$repo, pr:.number, title:.title, author:(.author.login // null),
+    | {schema_version:4, repo:$repo, pr:.number, title:.title, author:(.author.login // null),
+       # The actor type GitHub itself assigns the author, recorded because the LOGIN cannot answer
+       # it here: GraphQL reports a Bot login UNSUFFIXED, so `cloud-code-bot[bot]` arrives as
+       # `cloud-code-bot` and the `[bot]` test in the shared resolver never fires on this path.
+       # Additive (schema 3 -> 4): the grader reads it as `// false`, so a schema-3 corpus record
+       # still grades exactly as it did.
+       # (No apostrophes in here: this whole jq program is a single-quoted shell string.)
+       author_is_bot:((.author.__typename // "") == "Bot"),
        author_association:.authorAssociation, is_fork:(.isCrossRepository // false),
        labels:$labels, agent_coded:($labels | index("agent-coded") != null),
        labels_status:(if $labels_trunc then "unknown" else "ok" end),

scripts/pr-risk/risk-map.v0.json

@@ -68,7 +68,7 @@
     "agent-supervised": "R1",
     "human": "R1",
     "external": "R3",
-    "_why": "external (a fork / first-time contributor) is R3 on provenance alone — an unreviewed outside diff is the one provenance class never routed unattended. runbook is R0 ONLY when the shape assertion also holds (see the runbook registry); a claimed-but-shape-failed runbook falls back to its underlying class."
+    "_why": "external (ANY fork, or a first-time HUMAN contributor) is R3 on provenance alone — an unreviewed outside diff is the one provenance class never routed unattended. A non-fork bot is NOT external even though every GitHub App authors with author_association NONE: it is a runbook candidate, graded by the registry's shape assertion, and falls back to human when no entry asserts. runbook is R0 ONLY when the shape assertion also holds (see the runbook registry); a claimed-but-shape-failed runbook falls back to its underlying class."
   },
 
   "reversibility": {