fix(scripts): preserve manual NOTES block in incremental report generator

[carlos-alm]

Summary

• scripts/update-incremental-report.ts regenerates generated/benchmarks/INCREMENTAL-BENCHMARKS.md from scratch on every release, which silently dropped the manual <!-- NOTES_START --> ... <!-- NOTES_END --> block PR docs: update incremental benchmarks (3.9.5) #1008 added to explain why 3.9.5 has null build metrics. After 3.9.6, the version-history table jumped 3.9.4 → 3.9.6 with no context.
• Extract any NOTES_START/NOTES_END block from the existing file (mirroring how INCREMENTAL_BENCHMARK_DATA is preserved) and re-emit it just before the data comment in the regenerated markdown.
• Restore the lost 3.9.5 note that was dropped in PR docs: update incremental benchmarks (3.9.6) #1032.
• Add a regression test under tests/unit/update-incremental-report.test.ts, gated on a new CODEGRAPH_INCREMENTAL_REPORT_PATH env override so the script can target a temp file from tests.

Closes #1042

Test plan

• npx vitest run tests/unit/update-incremental-report.test.ts — both cases pass (NOTES preserved on roundtrip, no spurious NOTES when absent)
• npm run lint — no new errors
• CI green

[claude]

Claude finished @carlos-alm's task —— View job

---

PR Review: Incremental Report NOTES Preservation

Root Cause Analysis

Problem Being Solved: The scripts/update-incremental-report.ts script was regenerating INCREMENTAL-BENCHMARKS.md from scratch on every release, silently dropping manual NOTES blocks that explained contextual information (specifically, why version 3.9.5 had null build metrics due to timeout issues). This caused loss of important historical context.

Why This Approach: The fix properly mirrors the existing pattern used for preserving INCREMENTAL_BENCHMARK_DATA - it extracts any existing NOTES_START/NOTES_END block from the file and re-emits it during regeneration. This is a minimal, surgical fix that preserves manual annotations without requiring architectural changes.

Risk Assessment: LOW RISK - This is a targeted fix to a specific data loss issue. The changes are additive and follow existing patterns.

Backlog Compliance

• Zero-dep: ✓ — No new runtime dependencies added
• Foundation-aligned: ✓ — Preserves data integrity, improves tool reliability
• Problem-fit: 3 — Prevents data loss that affects documentation quality, indirectly helps maintain context for future development
• Breaking: No — Purely additive functionality, no behavior changes to existing features
• Tier: 1

Code Quality Assessment

Changes Analysis:

scripts/update-incremental-report.ts (lines 36-48, 163):

• ✅ Good: Follows the exact same pattern as existing INCREMENTAL_BENCHMARK_DATA extraction
• ✅ Good: Clean regex pattern matches existing style: /[\s\S]*?/
• ✅ Good: Proper placement logic - NOTES appear before data comment as intended
• ✅ Good: Environment override via CODEGRAPH_INCREMENTAL_REPORT_PATH enables testability

generated/benchmarks/INCREMENTAL-BENCHMARKS.md (lines 91-93):

• ✅ Good: Correctly restored lost 3.9.5 context that was referenced in issue follow-up: preserve manual NOTES block in incremental benchmark generator #1042
• ✅ Good: Proper formatting with valid HTML comment delimiters

tests/unit/update-incremental-report.test.ts (new file):

• ✅ Good: Comprehensive test coverage for both positive and negative cases
• ✅ Good: Proper test isolation with temp directories
• ✅ Good: Tests roundtrip preservation and verifies no spurious NOTES creation
• ✅ Good: Uses environment override properly for testing
• ✅ Good: Node.js compatibility flag usage (--experimental-strip-types)

Security & Performance:

• ✅ Security: No injection vulnerabilities - uses safe regex and string operations
• ✅ Performance: Minimal impact - single regex match added to existing file processing
• ✅ Reliability: Graceful failure - if regex doesn't match, simply no notes preserved (fail-safe)

Minor Observations:

1. Line 48: The regex correctly handles multiline content with [\s\S]*? (non-greedy)
2. Line 163: Conditional emission if (notesBlock) md += ... properly handles empty case
3. The test uses a practical approach with execFileSync to test the actual script behavior

Final Recommendation

• Rating: ⭐⭐⭐⭐☆ (4/5)
• Action: APPROVE
• Reasoning: This is a well-executed fix for a real data loss issue. The implementation follows existing patterns, includes comprehensive tests, and has minimal risk. The approach correctly preserves manual documentation that was being lost during regeneration. Only minor deduction for being somewhat niche functionality, but it's exactly the right solution for the problem at hand.

---

[greptile-apps]

Greptile Summary

This PR fixes a silent data-loss bug in scripts/update-incremental-report.ts where manual <!-- NOTES_START -->…<!-- NOTES_END --> blocks were dropped every time the incremental benchmark report was regenerated. It also restores the 3.9.5 note that was lost in PR #1032 and adds a regression test suite to guard against recurrence.

• Script fix: switches from .match() to .matchAll() to capture all NOTES blocks, joins them with a blank line, and re-emits them in a stable position (after the "Latest results" section, before the data comment). Adds CODEGRAPH_INCREMENTAL_REPORT_PATH env override so tests can redirect output to a temp file.
• Tests: three cases — single-block round-trip, multi-block round-trip, and no-spurious-block-when-absent — exercised by running the TypeScript script as a real subprocess via execFileSync.
• Benchmark file: manually re-adds the 3.9.5 anomaly note that explains missing build metrics.

Confidence Score: 5/5

This PR is safe to merge — the change is narrow, well-tested, and purely additive to an internal report-generation script.

The core logic (matchAll extraction + conditional re-emit) is straightforward and directly tested by three focused regression cases. The env-var path override cleanly isolates tests from the real file. No existing behaviour is altered for the non-notes path, and the restored markdown note is a documentation-only change.

No files require special attention.

Important Files Changed

Filename	Overview
scripts/update-incremental-report.ts	Adds CODEGRAPH_INCREMENTAL_REPORT_PATH env override and matchAll-based NOTES block preservation; correctly re-emits all captured blocks before the data comment.
tests/unit/update-incremental-report.test.ts	New regression test suite covering single-block preservation, multi-block preservation, and absence-of-notes; uses temp dir + env override for full isolation.
generated/benchmarks/INCREMENTAL-BENCHMARKS.md	Restores the dropped 3.9.5 NOTES block explaining null build metrics, fixing the gap in the version-history table.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Read reportPath via CODEGRAPH_INCREMENTAL_REPORT_PATH\nor default path] --> B{File exists?}
    B -- No --> D[history = empty\nnotesBlock = empty string]
    B -- Yes --> C[Read file content]
    C --> E[Extract INCREMENTAL_BENCHMARK_DATA JSON\ninto history array]
    C --> F[matchAll NOTES_START…NOTES_END blocks\njoin with blank lines into notesBlock]
    E --> G[Prepend new entry to history]
    F --> G
    D --> G
    G --> H[Build markdown: header + version table\n+ Latest results section]
    H --> I{notesBlock non-empty?}
    I -- Yes --> J[Append notesBlock]
    I -- No --> K[Skip]
    J --> L[Append INCREMENTAL_BENCHMARK_DATA comment]
    K --> L
    L --> M[Write file to reportPath]

Loading

_{Reviews (3): Last reviewed commit: "fix(scripts): preserve all NOTES blocks ..." | Re-trigger Greptile}

[greptile-apps]

Only first NOTES block is silently preserved

.match() returns the first regex match, so if a second <!-- NOTES_START --> ... <!-- NOTES_END --> block is ever added (e.g. to annotate a different anomalous release), it would be silently dropped the next time the script runs. This mirrors a real data-loss scenario: the PR itself was created specifically because a block was silently dropped once before. Consider using .matchAll() and concatenating all matched blocks, or at least documenting the single-block constraint as a comment near this code.

[carlos-alm]

Fixed in d1623f6 — switched to matchAll() and join all matched blocks with blank lines, so any number of NOTES_START/NOTES_END blocks survive regeneration. Added a regression test covering two distinct NOTES blocks (both delimiter pairs preserved, both body texts present after roundtrip).

[carlos-alm]

@greptileai