Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ Review precision: confidence now decides where a finding lands, newly-added pars
- **Mock-fidelity review check** (#111): when a PR changes tests, the review prompt compares stubs/mocks (`when`/`doThrow`/`doReturn`) against the real collaborator's contract when that definition is already in the provided material, and flags contradictions (e.g. a mock that throws an exception the real method swallows) at low/medium confidence instead of treating the green test as proof. Softens the related-tests framing that previously reinforced unfaithful stubs; the skeptical verifier gains a matching rule. Broader cross-file collaborator retrieval remains with #55
- **Structured skip transparency for automatic reviews** (#341): every path that skips an automatic review (draft, base-branch and label gates, `/pause`, rate window, duplicate webhook delivery, rejected dispatch) now emits a structured reason code β€” a `Automatic review skipped [reason=...]` log line, a `thrillhouse.review.skips` OTLP counter tagged with `reason` and `repository`, and per-reason counts on the dashboard summary endpoint (`skippedReviewsByReason`). README gains a "PR opened but no review posted" troubleshooting checklist
- **Bug-fix efficacy check** (#110): when a PR declares itself a bug fix (the PR-template "Bug fix" checkbox or a `Fixes/Closes/Resolves #N` reference), the review prompt now extracts the concrete failure trigger from the PR description and the linked issues' text (fetched best-effort, up to 3 issues) and verifies the change actually alters behavior on that trigger's path β€” a locally-correct fix that the stated trigger never reaches is reported as a finding instead of passing silently, and when the deciding code is outside the diff the verdict is held at low confidence as a verification request rather than approved
- **Finding feedback capture for the learnings pipeline**: records maintainer πŸ‘/πŸ‘Ž reactions (and conservative "not useful" reply heuristics) on bot finding comments into a `finding_feedback` table, with per-repo aggregates on `GET /api/dashboard/feedback` and a documented data model/retention policy (`docs/FEEDBACK.md`). GitHub Apps have no `reaction` webhook, so reactions are polled via the Reactions API on review-thread replies and follow-up reviews. Precursor to cross-review learnings (#38); does not yet feed prompts (#324)
- **Adversarial failure-mode characterization for new heuristics** (#123): when a diff introduces parsing, regex, validation, or heuristic code β€” a `Pattern.compile`, a tokenizer, a normalize step, a scope window or threshold constant β€” the reviewer switches those hunks from happy-path reading into decision-boundary characterization: it synthesizes the inputs that probe the rule's edges (a lambda against a paren-counting regex, an NBSP against an ASCII whitespace class, a token inside a fenced code block against a "starts with" check, a present-but-wrong-shape value against a presence check), names them literally, and reports false negatives as well as false positives. The triggering input is absent from the diff by definition, so the verifier gains a matching exemption from the verbatim-quote rule β€” without it the quote requirement rejects this entire class. Findings land at low/medium confidence as verification requests; executing the synthesized inputs is #96. Gated by a deliberately narrow detector so ordinary `substring`/`indexOf`/`trim`/`parseInt` code does not trigger it (dogfood: three review rounds on PR #114 returned "no issues" while human review found ~10 precision/recall defects in exactly this kind of code)
- **Finding feedback capture for the learnings pipeline**: records maintainer πŸ‘/πŸ‘Ž reactions (and conservative "not useful" reply heuristics) on bot finding comments into a `finding_feedback` table, with per-repo aggregates on `GET /api/dashboard/feedback` and a documented data model/retention policy (`docs/FEEDBACK.md`). GitHub Apps have no `reaction` webhook, so reactions are polled via the Reactions API on review-thread replies and follow-up reviews. Feedback is permission-gated in both directions: only verified write-capable collaborators reacting on bot-authored finding threads are recorded, and `GET /api/dashboard/feedback` returns only repositories the authenticated user can access β€” so the endpoint cannot be used to enumerate activity on repos you are not a member of. Capture runs on a bounded executor that prioritizes newer findings and drops work rather than queueing without limit, and review logs no longer carry source-line contents at INFO. Precursor to cross-review learnings (#38); does not yet feed prompts (#324)
- **Adversarial failure-mode characterization for new heuristics** (#123): when a diff introduces parsing, regex, validation, or heuristic code β€” a `Pattern.compile`, a tokenizer, a normalize step, a scope window or threshold constant β€” the reviewer switches those hunks from happy-path reading into decision-boundary characterization: it synthesizes the inputs that probe the rule's edges (a lambda against a paren-counting regex, an NBSP against an ASCII whitespace class, a token inside a fenced code block against a "starts with" check, a present-but-wrong-shape value against a presence check), names them literally, and reports false negatives as well as false positives. The triggering input is absent from the diff by definition, so the verifier gains a matching exemption from the verbatim-quote rule β€” without it the quote requirement rejects this entire class. Findings land at low/medium confidence as verification requests, and a synthesized failure is only confirmed when the expected domain or contract is visible in the material β€” a contract-free probe stays a low-confidence verification request. Executing the synthesized inputs is #96. The trigger spans languages: explicit regex construction (Java, JS/TS, Python, Go, C#, Kotlin), JS/TS regex literals and `function`/arrow validators, declared parse/validate/tokenize members including package-private Java methods, and construction split across lines via a bounded, file-reset-aware window. It stays deliberately narrow so ordinary `substring`/`indexOf`/`trim`/`parseInt` code does not trigger it (dogfood: three review rounds on PR #114 returned "no issues" while human review found ~10 precision/recall defects in exactly this kind of code)
- **Configurable CI gating strictness for APPROVE** (#322): `REVIEW_CI_GATING` chooses how CI status factors into approval β€” `strict` (default) holds APPROVE while required CI is pending, failing, or unreadable; `warn` allows APPROVE but records the CI uncertainty in the summary and check run; `off` decides on findings alone
- **Configurable blocking strictness** (#323): `REVIEW_BLOCKING_STRICTNESS` sets when findings escalate to `REQUEST_CHANGES` β€” `balanced` (default: CRITICAL/HIGH at high confidence), `strict` (any CRITICAL/HIGH), or `lenient` (CRITICAL at high confidence only), so a repo can tune how loudly the bot blocks without editing prompts
- **Model generation parameters** (#64): `frequency-penalty`, `presence-penalty` and `seed` join the existing per-model settings under `thrillhousebot.ai.models.<model>.*`, sent on every chat call when set. `seed` makes a provider's sampling reproducible where supported, which is what lets the #113 eval corpus compare runs meaningfully
Expand All @@ -23,8 +23,9 @@ Review precision: confidence now decides where a finding lands, newly-added pars

- **Passing in-diff tests must exercise the claimed path before suppressing a finding** (#116): the review and verifier prompts no longer treat a green test in the same diff as automatic disproof. A test may invalidate (generator) or reject (verifier) a finding only when it demonstrably exercises the claimed path β€” asserting on the path's output and stubbing collaborators into the relevant state, not leaving them unmocked so a default bypasses it (and not with stubs that contradict the real collaborator β€” see #111). When that exercise cannot be shown from the provided material, confidence is lowered with an explanation instead of dropping the finding (dogfood: PR #99 approve-path test green because `getPullRequest` was unmocked)
- **Low-confidence findings post in the summary instead of inline** (#105): inline review comments are now gated on confidence β€” a low-confidence finding is listed in the summary body rather than anchored to a line. Uncertain claims still surface, but they stop occupying the position that reads as "this line is wrong", and they can no longer request changes on their own
- **Exact-arithmetic and "this test fails" claims are capped and hedged** (#97): a finding that rests on line-count, array-length, or index arithmetic the model performed by counting, or that asserts a specific test fails, is now at most low confidence and must be phrased as a verification request ("CI will confirm") rather than settled fact. The verifier gains the symmetric rule and rejects such a claim stated as fact with no execution or CI signal behind it β€” it cannot settle the question by recounting, being the same modality reading the same diff. Mirrors the existing "why would this test still pass" guard for the failure direction (dogfood: PR #84 produced a definitively-worded off-by-one β€” "the section is 7 lines, so 7 βˆ’ 3 = 4 omitted" β€” against a test that passes as written, and the verifier re-did the same arithmetic and agreed)
- **Exact-arithmetic and "this test fails" claims are capped and hedged** (#97): a finding that rests on line-count, array-length, or index arithmetic the model performed by counting, or that asserts a specific test fails, is now capped at low confidence when nothing but counting backs it, and must be phrased as a verification request ("CI will confirm") rather than settled fact. The verifier gains the symmetric rule and rejects such a claim stated as fact with no execution or CI signal behind it β€” it cannot settle the question by recounting, being the same modality reading the same diff. The cap is evidence-dependent, not absolute: when an execution or CI signal in the provided material actually shows the failure, the finding keeps the confidence that evidence justifies instead of being forced low. Mirrors the existing "why would this test still pass" guard for the failure direction (dogfood: PR #84 produced a definitively-worded off-by-one β€” "the section is 7 lines, so 7 βˆ’ 3 = 4 omitted" β€” against a test that passes as written, and the verifier re-did the same arithmetic and agreed)
- **Dual-gate merge policy for contributors** (#342): merging now requires both the static analysis gate and a ThrillhouseBot review, so neither signal alone can carry a change in. Affects anyone opening a PR against this repo rather than anyone running the bot
- **Wider default ignored-files patterns** (#52): `thrillhousebot.review.ignored-files` now skips far more generated and vendored output out of the box β€” `pnpm-lock.yaml`, `go.sum`, protobuf output (`*.pb.go`, `*_pb2.py`), minified bundles and sourcemaps (`*.min.js`, `*.min.css`, `*.map`), and the directories `node_modules/`, `dist/`, `build/`, `out/`, `.next/`, `vendor/`, `__pycache__/`, `.venv/`, `bin/`, `obj/`, alongside the previous lockfile/`pom.xml`/`target/` set. Less budget burned on artifacts nobody reviews. Worth checking on upgrade: the directory globs match by name, so a repository that keeps handwritten source under `build/`, `bin/`, `out/` or `vendor/` will stop having it reviewed β€” narrow the list per deployment if that applies to you

### Fixed

Expand All @@ -41,6 +42,7 @@ Review precision: confidence now decides where a finding lands, newly-added pars
### Dependencies

- Bumped the Quarkus platform group, the frontend npm-minor-patch group, `@testing-library/jest-dom` to 7.0.0, the website docs-minor-patch group, `dompurify` to 3.4.12, `svgo` to 4.0.2, and the GitHub Actions minor/patch group
- The docs site now requires Node 22 and pins PostCSS 8.5.23 and Sharp 0.35.3, so a clean `npm ci` on npm 11 resolves without known high-severity advisories

## [0.4.0] β€” 2026-07-12

Expand Down
Loading