Fix empty section content + surface richer citations

[hallelx2]

@

What

Two engine fixes reported from the dashboard playground / section browser.

1. Empty section content (root cause + safety net)

Multi-line display headings (a hero title wrapped across lines, e.g. THE SYSTEM, END-TO-END / How Vectorless / actually works.) were each detected as a separate heading, each producing a "purely structural" leaf Section with empty Content. Those sections returned empty content to the browser and /v1/sections/{id}.

• Row-merge (parseDoc) — consecutive same-level heading rows on the same page with no body between them merge into one heading; the following body attaches to the whole title.
• foldEmptyLeafSections — lossless post-parse pass removes any remaining empty-content leaf and carries its title onto the sibling body it heads (as a bold markdown heading line), unioning page ranges. Empty internal nodes (headings with real children) are left untouched. Applied on both the font-heuristic and outline parse paths.

2. Richer citations

defaultTreeWalkMaxCitations 3 → 6. The cap is a ceiling that only bites on over-emission; the prompt + confidence gating still drive the minimal sufficient set, so precision-critical single-pick answers are unchanged, while genuinely multi-location answers surface every location they rely on.

Tests

• New pkg/parser/empty_sections_test.go (fold behaviour, accumulation, internal-node preservation, trailing/lone-leaf edge cases).
• Updated cap-default assertions (3→6) in config tests.
• Full go test ./... green.
  @

Summary by CodeRabbit

• New Features

  • Increased the default limit for cited page ranges in TreeWalk answers from 3 to 6, allowing broader multi-page responses.
  • Improved PDF section handling so continuation headings and empty fragments are folded into cleaner, more readable sections.

• Bug Fixes

  • Reduced cases where PDFs produced separate empty heading-only sections.
  • Preserved citation coverage when empty sections are merged across page ranges.

[sourcery-ai]

Sorry @hallelx2, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR increases the default TreeWalk max_citations value from 3 to 6 across config defaults, retrieval strategy, and associated tests. Separately, it introduces a foldEmptyLeafSections post-processing pass in the PDF parser that merges empty leaf section titles into sibling content, along with heading flatting adjustments and new tests.

Changes

TreeWalk max_citations default increase

Layer / File(s)	Summary
Default value and strategy update pkg/config/config.go, pkg/retrieval/treewalk_strategy.go	Built-in default and citation cap for max_citations raised from 3 to 6, with updated comment explaining the ceiling.
Config and env-override test updates pkg/config/config_test.go, internal/config/config_treewalk_test.go	Test assertions updated to expect default of 6, including no-env and invalid-env-value cases.

Empty leaf section folding in PDF parser

Layer / File(s)	Summary
Heading level and multi-line merge logic pkg/parser/pdf.go	Heading flatting derives newLevel without mutating base lvl, and merges multi-line display heading continuations into the current heading fragment when conditions match.
foldEmptyLeafSections implementation and wiring pkg/parser/pdf.go	New function folds empty leaf section titles into next/previous sibling content with page range unioning; wired into both heuristic and outline-based parse paths before page propagation.
Fold behavior tests pkg/parser/empty_sections_test.go	New TestFoldEmptyLeafSections covers merging, accumulation, preservation of internal nodes, trailing fragments, and lone empty leaves.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• hallelx2/vectorless-engine#35: Prior PR wired TreeWalk max-citations env forwarding and tests that this PR's default-value change builds upon.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the two main changes: empty section handling and richer citation output.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/empty-section-content-and-citations

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

pkg/config/config.go (1)

468-476: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Stale doc comment still says "Default: 3".

The MaxCitations field comment says Default: 3 but Default() now sets it to 6 (Line 763). Update the comment to avoid confusing future readers/config authors.

📝 Proposed fix

 	// MaxCitations caps how many distinct page ranges the final
-	// answer may cite. Default: 3. Set to 0 to use the strategy's
+	// answer may cite. Default: 6. Set to 0 to use the strategy's
 	// built-in default. This is a confidence backstop, not a

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/config/config.go` around lines 468 - 476, The MaxCitations doc comment is
stale and still claims the default is 3 even though Default() sets it to 6.
Update the comment attached to the MaxCitations field in config.go so it matches
the actual default value and remains consistent with the Default function.

pkg/parser/empty_sections_test.go (1)

14-88: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a case for an empty leaf whose next sibling is an empty internal node.

The suite doesn't exercise the scenario where the only carrier following an empty leaf is an internal node (children, empty Content). Given the folding pass attaches pending titles to that node's Content, a test here would pin down the behavior flagged in pdf.go and prevent silent title loss.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/parser/empty_sections_test.go` around lines 14 - 88, Add a test case in
TestFoldEmptyLeafSections for an empty leaf followed by an empty internal node,
since foldEmptyLeafSections currently isn’t pinned down for that path. Create a
Section sequence where the first item has empty Content and the next item has
Children with empty Content, then assert the folded result preserves the pending
title by attaching it to that internal node’s Content while keeping its children
intact. Use the existing foldEmptyLeafSections and Section symbols to place the
new case alongside the other subtests.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/config/config.go`:
- Around line 468-476: The MaxCitations doc comment is stale and still claims
the default is 3 even though Default() sets it to 6. Update the comment attached
to the MaxCitations field in config.go so it matches the actual default value
and remains consistent with the Default function.

In `@pkg/parser/empty_sections_test.go`:
- Around line 14-88: Add a test case in TestFoldEmptyLeafSections for an empty
leaf followed by an empty internal node, since foldEmptyLeafSections currently
isn’t pinned down for that path. Create a Section sequence where the first item
has empty Content and the next item has Children with empty Content, then assert
the folded result preserves the pending title by attaching it to that internal
node’s Content while keeping its children intact. Use the existing
foldEmptyLeafSections and Section symbols to place the new case alongside the
other subtests.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d27098c0-0bfd-4224-8217-57a9608cfa94

📥 Commits

Reviewing files that changed from the base of the PR and between 8d074fa and d9aa153.

📒 Files selected for processing (6)

• internal/config/config_treewalk_test.go
• pkg/config/config.go
• pkg/config/config_test.go
• pkg/parser/empty_sections_test.go
• pkg/parser/pdf.go
• pkg/retrieval/treewalk_strategy.go