Skip to content

Fix end-edge attribution (off-by-one), node-id collisions, and stars - #1

Open
deerdear wants to merge 3 commits into
BeksOmega:mainfrom
deerdear:fix/end-edge-attribution
Open

Fix end-edge attribution (off-by-one), node-id collisions, and stars#1
deerdear wants to merge 3 commits into
BeksOmega:mainfrom
deerdear:fix/end-edge-attribution

Conversation

@deerdear

@deerdear deerdear commented Jul 28, 2026

Copy link
Copy Markdown

Hi! I'm using this dataset for a visualization project — thank you for publishing it. While validating it against two independent transcriptions of the book, I found a systematic extraction bug and some smaller ones. This PR fixes them.

The main bug: end edges are attributed one pattern forward

In addEdges, lines[i - 2] is the paragraph before pattern N's heading — which is pattern N−1's closing links — but the code attributes those references to N:

for string in re.findall(r'\((\d+)\)', lines[i - 2]):
    e = g.add_edge(node, nodesDict[int(string)], True)   # node = N, should be N-1
    e['origin'] = 'end'

Evidence (validated against the apl-md hand transcription and the iwritewordsgood.com mirror):

  • As attributed, only 141/1053 end edges match the book's link structure. Re-attributed to the previous pattern, 1008/1053 (96%) match.
  • It also explains all 66 self-edges in the file (pattern N−1's closing referencing N becomes N→N).
  • Whole-graph agreement (undirected Jaccard vs the apl-md transcription): 0.34 → 0.81 after the fix.

The fix tracks the previous heading's node and attributes lines[i - 2] to it, plus a guard against self-references and duplicate edges (the book declares most links in both patterns' texts).

Smaller fixes

  • "A ROOM OF ONE's OWN" (lowercase s) — pattern 141's heading never matched, silently dropping its links and stars.
  • Includes the three fixes from your unmerged fix-data-errors branch (merged in, credited to that commit): (73, 'THE FAMILY') → 75, (201, 'THICKENING THE OUTER WALLS') → 211, and the range(26, 233) subsection typo. The id collisions also silently misrouted both patterns' edges and stars via the nodesDict overwrite.
  • .gitignore for __pycache__ (the branch had committed bytecode).

The committed patterns.graphml

I don't have a .txt of the book, so I couldn't re-run the pipeline. Instead the committed file was mechanically repaired with an equivalent transformation: rename the duplicate nodes (73→75, 201→211 + section/subsection), shift every origin=end edge's source back by one, drop self-edges/duplicates, and correct 64 stars values cross-checked against the book mirror (231/253 headings parseable; apl-md confidence tags as fallback). Edges for the four patterns affected by the id collisions were unrecoverable from the file (both patterns' links landed on one node id) and were restored from the apl-md transcription's link metadata.

Result: 253 nodes with unique ids 1–253, 1,474 directed edges, zero self-edges/duplicates, Jaccard 0.81 vs apl-md.

Known remaining limitation (out of scope)

Even after the fix, edge recall is ~60% — the source text conversion missed a large share of the book's cross-references (the book has ~1,750+ links). That's a property of the input .txt, not the script; I can share my comparison tooling if useful.

Happy to split this differently (e.g. rebase onto fix-data-errors) if you'd prefer.

google-labs-jules Bot and others added 3 commits October 9, 2025 21:05
Corrects three data errors in the `process.py` script:
- Duplicate pattern ID 73 ('THE FAMILY') is changed to 75.
- Duplicate pattern ID 201 ('THICKENING THE OUTER WALLS') is changed to 211.
- Incorrect subsection range for 'STRUCTURAL SUBSIDIARIES' is changed from range(26, 233) to range(226, 233).

Adds a new test file `test_process.py` with tests to verify that there are no duplicate pattern IDs and no overlapping subsection ranges, preventing similar issues in the future.
The paragraph before each pattern's heading is the PREVIOUS pattern's
closing links; addEdges attributed it to the current pattern, shifting
every 'end' edge one pattern forward and creating 66 self-edges.
Validated against two independent transcriptions of the book: as
attributed, only 141/1053 end edges matched; shifted to the previous
pattern, 1008/1053 match. Whole-graph agreement (undirected Jaccard vs
the apl-md transcription) goes 0.34 -> 0.81.

Also: fix the 'A ROOM OF ONE'S OWN' case typo (pattern 141's heading
never matched, dropping its links and stars), add a self/duplicate edge
guard, and repair the committed patterns.graphml (node renames 73->75,
201->211 from the fix-data-errors branch, end-edge shift, dedupe, and
64 star corrections cross-checked against the book).

The committed graphml was repaired by transforming the previous file
(the pipeline needs a book .txt we don't have); links for the four
patterns affected by the id collisions were restored from the apl-md
transcription's link metadata.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant