Skip to content

Fix JSONL loader handling of blank/invalid lines#2434

Open
natoverse wants to merge 3 commits into
mainfrom
fix/2424-jsonl-loader-blank-lines
Open

Fix JSONL loader handling of blank/invalid lines#2434
natoverse wants to merge 3 commits into
mainfrom
fix/2424-jsonl-loader-blank-lines

Conversation

@natoverse

Copy link
Copy Markdown
Collaborator

Summary

  • Skip blank/whitespace-only lines in JSONL input loader
  • Skip malformed JSON rows safely
  • Skip non-object JSON rows
  • Add regression test and fixture coverage
  • Add semversioner patch entry

Fixes #2424

@natoverse natoverse requested a review from a team as a code owner July 10, 2026 22:20
@natoverse

Copy link
Copy Markdown
Collaborator Author

This replaces #2423 to include more handling of edge cases

Comment thread tests/unit/indexing/input/test_jsonl_loader.py

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves robustness of the JSONL input reader so that a single blank or invalid line doesn’t cause the entire file to be skipped, aligning behavior with the reported issue (#2424).

Changes:

  • Updated the JSONL reader to ignore blank/whitespace-only lines, skip malformed JSON rows, and skip non-object JSON values.
  • Added a regression test and a JSONL fixture covering blank, invalid, and non-object rows.
  • Added a semversioner patch note documenting the behavior change.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
packages/graphrag-input/graphrag_input/jsonl.py Makes JSONL parsing resilient by skipping blank/invalid/non-object rows and continuing through the file.
tests/unit/indexing/input/test_jsonl_loader.py Adds regression coverage to ensure only valid object rows become documents.
tests/unit/indexing/input/data/jsonl-with-invalid-and-blank-lines/input.jsonl Provides a fixture with blank, invalid JSON, and non-object rows interleaved with valid objects.
.semversioner/next-release/patch-20260710221345787890.json Records the patch-level change in JSONL loader behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/graphrag-input/graphrag_input/jsonl.py
Comment thread packages/graphrag-input/graphrag_input/jsonl.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSONL reader skips entire file when input contains blank lines

4 participants