docs: migrate remaining trees + man-page generation (Phase 2+3 of N)

docs-src/PLAN.md

@@ -116,14 +116,54 @@ the site (and can attach to GitHub releases).
 
 ## Execution phases
 
-1. **Scaffold**: `docs-src/` skeleton, `site.toml` (version from `dist/RELEASE`),
-   templates, `build.py` (HTML first), one hand-migrated sample page end-to-end.
-2. **Extractor**: `extract.py` on the API-reference C tree; diff-verify; iterate
-   the cleanup until content matches. Then C++/STL, then the guides.
-3. **Man pages**: per-API `*.3` + the `libdb.3` overview; `mandoc -Tlint` clean.
-4. **PDF**: pandoc per book (programmer_reference, the GSGs, api_reference).
-5. **CI**: `docs.yml` with all validators + the completeness gate.
-6. **Publish**: wire gh-pages to the generated HTML; update the landing page.
+1. **Scaffold** (DONE, PR #119): `docs-src/` skeleton, `site.toml` (version from
+   `dist/RELEASE`), templates, `build.py` (HTML first).
+2. **Extractor + trees** (DONE): `extract.py` + `verify.py` no-loss gate on the
+   C API tree (100.00%), then STL (100.00%) and every guide tree. See the
+   retention table below. `migrate_tree.py` drives chaptered guides (order from
+   the index TOC, image copy); `fix_xrefs.py` remaps cross-tree links to the new
+   `docs-src/` layout.
+3. **Man pages** (DONE): `build_man()` -> 787 `*.3` (785 C+STL refentry pages +
+   `libdb.3` overview + utilities), `mandoc -Tlint` = 0 errors. `man_coverage.py`
+   reports API coverage (measure-only; the CI gate is phase 5).
+4. **PDF** (TODO): pandoc per book (programmer_reference, the GSGs, api_reference)
+   with a shared LaTeX header. `build.py build_pdf()` is the stubbed seam.
+5. **CI** (TODO): `docs.yml` with all validators + the completeness gate (wire
+   `verify.py` per tree + `man_coverage.py` as hard gates).
+6. **Publish** (TODO): wire gh-pages to the generated HTML; update the landing
+   page.
+
+### Phase 2 retention (verify.py, mean word retention; 0 hard drops on all)
+
+| tree                    | pages | retention |
+|-------------------------|-------|-----------|
+| api/c                   | 470   | 100.00%   |
+| api/stl                 | 322   | 100.00%   |
+| guides/programmer_reference | 203 | 99.98%  |
+| guides/upgrading        | 180   | 100.00%   |
+| guides/installation     | 101   | 99.99%    |
+| guides/porting          | 16    | 100.00%   |
+| guides/gsg (C)          | 37    | 100.00%   |
+| guides/gsg_txn (C)      | 38    | 100.00%   |
+| guides/gsg_db_rep (C)   | 26    | 100.00%   |
+| guides/collections      | 37    | 99.98%    |
+| guides/bdb-sql          | 30    | 100.00%   |
+| guides/articles         | 2     | 100.00%   |
+
+### Deferred: docs/csharp (38 MB) and docs/java (12 MB)
+
+These are LANGUAGE-BINDING docs and are NOT DocBook, so the reverse-DocBook
+extractor does not apply:
+- **csharp** — a compiled Sandcastle/MS-Help-Viewer tree (`.chm` + `.aspx` +
+  JS/PNG, 2457 files, only 1 real `.html`). Would need a bespoke extractor.
+- **java** — standard Javadoc HTML (525 files: allclasses-frame, package-frame,
+  index-all). Different structure; the `refentry`/`chapter` isolation is moot.
+
+Both are enormous and lower-value for the core C engine. Deferred to a future
+phase (regenerate from the C#/Java sources with their native doc tools, or write
+a per-format extractor) rather than sink budget reverse-engineering rendered
+help output. The gsg/gsg_txn/gsg_db_rep CXX/JAVA sub-variants are likewise
+deferred; the C variants are migrated.
 
 ## Non-negotiables
 

docs-src/_data/site.toml

@@ -8,3 +8,7 @@ short_name = "libdb"
 base_url   = "https://libdb.org/docs/"
 copyright  = "Copyright (c) 1990, 2013 Oracle and/or its affiliates. All rights reserved."
 tagline    = "The Berkeley DB reference documentation"
+# Man-page .TH date field (the DocBook source is dated 9/9/2013). Use the
+# `Month D, YYYY` form so mandoc parses it (hyphenated ISO gets roff-escaped by
+# pandoc into `2013\-09\-09`, which mandoc then can't read). Single source.
+man_date   = "September 9, 2013"

docs-src/_migrate/extract.py

@@ -20,6 +20,14 @@
 Usage:  extract.py [SRC_HTML_DIR] [OUT_MD_DIR]
 Defaults: docs/api_reference/C  ->  docs-src/api/c
 Requires: pandoc on PATH (run under `nix shell nixpkgs#pandoc`).
+
+Works for both the flat `refentry` API trees (C, STL) and the chaptered
+`chapter`/`sect1` guide trees (programmer_reference, gsg, ...): every page has a
+single top-level content div in CONTENT_CLASSES. A few auxiliary guide pages
+(embedded.html, witold.html) put prose straight under <body> with no content
+div — for those a body-level fallback captures everything except the stable
+boilerplate divs. The C API tree never hits the fallback (its only div-less
+pages are frameset stubs with no prose), so it is regression-safe.
 """
 import html.parser
 import re
@@ -30,6 +38,11 @@
 REPO = Path(__file__).resolve().parents[2]
 SRC = Path(sys.argv[1]) if len(sys.argv) > 1 else REPO / "docs/api_reference/C"
 OUT = Path(sys.argv[2]) if len(sys.argv) > 2 else REPO / "docs-src/api/c"
+# source: path shown in front-matter; derived from SRC relative to the repo.
+try:
+    SRC_REL = str(SRC.resolve().relative_to(REPO))
+except ValueError:
+    SRC_REL = str(SRC)
 
 # DocBook classes whose top-level <div> IS the real content.
 CONTENT_CLASSES = {"sect1", "chapter", "book", "preface", "appendix",
@@ -119,6 +132,72 @@ def inner_html(self):
         return "".join(self.buf)
 
 
+# Body-level fallback: for the handful of guide pages that carry prose directly
+# under <body> (no content div), capture everything except the stable
+# boilerplate divs (navheader/libver/navfooter). Only used when BodyExtractor
+# found no content div, so it cannot alter div-having pages.
+BOILERPLATE_CLASSES = {"navheader", "libver", "navfooter"}
+
+
+class BodyFallbackExtractor(html.parser.HTMLParser):
+    def __init__(self):
+        super().__init__(convert_charrefs=False)
+        self.in_body = False
+        self.depth = 0
+        self.skip_depth = None    # depth at which we entered a boilerplate div
+        self.buf = []
+
+    def handle_starttag(self, tag, attrs):
+        ad = dict(attrs)
+        if tag == "body":
+            self.in_body = True
+            self.depth = 0
+            return
+        if not self.in_body:
+            return
+        if self.skip_depth is None and tag == "div":
+            cls = (ad.get("class") or "").split()
+            if any(c in BOILERPLATE_CLASSES for c in cls):
+                self.skip_depth = self.depth
+                self.depth += 1
+                return
+        if self.skip_depth is None:
+            self.buf.append(BodyExtractor._fmt_start(tag, attrs))
+        self.depth += 1
+
+    def handle_startendtag(self, tag, attrs):
+        if self.in_body and self.skip_depth is None:
+            self.buf.append(BodyExtractor._fmt_start(tag, attrs, self_closing=True))
+
+    def handle_endtag(self, tag):
+        if tag == "body":
+            self.in_body = False
+            return
+        if not self.in_body:
+            return
+        self.depth -= 1
+        if self.skip_depth is not None and self.depth == self.skip_depth:
+            self.skip_depth = None
+            return
+        if self.skip_depth is None:
+            self.buf.append(f"</{tag}>")
+
+    def handle_data(self, data):
+        if self.in_body and self.skip_depth is None:
+            self.buf.append(data)
+
+    def handle_entityref(self, name):
+        if self.in_body and self.skip_depth is None:
+            self.buf.append(f"&{name};")
+
+    def handle_charref(self, name):
+        if self.in_body and self.skip_depth is None:
+            self.buf.append(f"&#{name};")
+
+    def inner_html(self):
+        return "".join(self.buf)
+
+
 # Wrapper <div>s that carry no content — just DocBook layout scaffolding. We
 # unwrap them (drop the tags, keep children) so pandoc doesn't emit a wall of
 # empty `<div>` noise. Headings inside survive and carry the structure.
@@ -205,10 +284,16 @@ def extract_one(path):
     raw = path.read_text(encoding="utf-8", errors="replace")
     ex = BodyExtractor()
     ex.feed(raw)
-    inner = preprocess_html(ex.inner_html())
+    inner = ex.inner_html()
     title = clean_title(ex.title or path.stem)
+    if not inner.strip():
+        # No content div: fall back to body-level capture (guide article pages).
+        fb = BodyFallbackExtractor()
+        fb.feed(raw)
+        inner = fb.inner_html()
+    inner = preprocess_html(inner)
     md = pandoc_html_to_gfm(inner)
-    md = cleanup(md, title, title, f"docs/api_reference/C/{path.name}")
+    md = cleanup(md, title, title, f"{SRC_REL}/{path.name}")
     return md, title
 
 

docs-src/_migrate/fix_xrefs.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Rewrite cross-tree links from the OLD docs/ layout to the NEW docs-src/ one.
+
+The extractor rewrites *same-tree* links (`foo.html` -> `foo.md`). Cross-tree
+links keep their old relative shape, e.g. from an API page:
+    ../../programmer_reference/env_db_config.html#frag
+    ../api_reference/C/dbget.html
+Those still address the OLD `docs/` tree. This pass maps the old tree segment to
+the new `docs-src/` location and re-computes the relative prefix from each page's
+own depth, emitting `.md` targets (build.py turns `.md` -> `.html`).
+
+Trees not migrated yet (CXX, TCL, java, csharp) are left as-is: their links stay
+`.html` pointing at the archived `docs/` tree, which still resolves for readers.
+
+Idempotent: only rewrites links whose old tree segment is in OLD_TO_NEW.
+Usage:  fix_xrefs.py            # rewrite under docs-src/api + docs-src/guides
+"""
+import re
+import sys
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+DOCS_SRC = HERE.parent            # docs-src/
+ROOTS = [DOCS_SRC / "api", DOCS_SRC / "guides"]
+
+# Old docs/ tree segment  ->  new docs-src/ tree dir (relative to docs-src root).
+OLD_TO_NEW = {
+    "api_reference/C": "api/c",
+    "api_reference/STL": "api/stl",
+    "programmer_reference": "guides/programmer_reference",
+    "upgrading": "guides/upgrading",
+    "installation": "guides/installation",
+    "porting": "guides/porting",
+    "gsg/C": "guides/gsg",
+    "gsg_txn/C": "guides/gsg_txn",
+    "gsg_db_rep/C": "guides/gsg_db_rep",
+    "collections/tutorial": "guides/collections",
+    "bdb-sql": "guides/bdb-sql",
+}
+# Longest keys first so "api_reference/C" wins over a bare "C" etc.
+OLD_KEYS = sorted(OLD_TO_NEW, key=len, reverse=True)
+
+# href="<any ../ prefix><rest>.html<#frag>"  (cross-tree = has a "/" in rest)
+HREF = re.compile(r'href="((?:\.\./)+)([A-Za-z0-9_./\-]+)\.html(#[^"]*)?"')
+
+
+def new_root_prefix(page):
+    """`../` * (depth of page below docs-src) -> reach docs-src root."""
+    depth = len(page.relative_to(DOCS_SRC).parts) - 1
+    return "../" * depth
+
+
+def rewrite_one(page):
+    text = page.read_text(encoding="utf-8")
+    root = new_root_prefix(page)
+
+    def sub(m):
+        rest = m.group(2)            # e.g. programmer_reference/env_db_config
+        frag = m.group(3) or ""
+        for key in OLD_KEYS:
+            if rest == key or rest.startswith(key + "/"):
+                tail = rest[len(key):].lstrip("/")   # page stem within the tree
+                if not tail:
+                    tail = "index"
+                return f'href="{root}{OLD_TO_NEW[key]}/{tail}.md{frag}"'
+        return m.group(0)            # untouched (unmigrated tree)
+
+    new = HREF.sub(sub, text)
+    if new != text:
+        page.write_text(new, encoding="utf-8")
+        return True
+    return False
+
+
+def main():
+    changed = 0
+    for root in ROOTS:
+        for p in root.rglob("*.md"):
+            if rewrite_one(p):
+                changed += 1
+    print(f"cross-tree xrefs rewritten in {changed} pages")
+
+
+def _selfcheck():
+    # from a depth-2 page (api/c/x.md, root=../../), an old ../../programmer_reference
+    # link becomes ../../guides/programmer_reference/*.md
+    import tempfile
+    d = Path(tempfile.mkdtemp())
+    (d / "api/c").mkdir(parents=True)
+    global DOCS_SRC, ROOTS
+    DOCS_SRC, ROOTS = d, [d / "api"]
+    pg = d / "api/c/x.md"
+    pg.write_text('a <a href="../../programmer_reference/env.html#f">E</a> '
+                  'b <a href="../api_reference/C/dbget.html">G</a> '
+                  'c <a href="../api_reference/CXX/foo.html">X</a>')
+    assert rewrite_one(pg)
+    out = pg.read_text()
+    assert 'href="../../guides/programmer_reference/env.md#f"' in out, out
+    assert 'href="../../api/c/dbget.md"' in out, out
+    assert '../api_reference/CXX/foo.html' in out, out   # unmigrated: untouched
+    print("selfcheck ok")
+
+
+if __name__ == "__main__":
+    if "--selfcheck" in sys.argv:
+        _selfcheck()
+    else:
+        main()