Skip to content

docs: modernization pipeline + C API tree (Phase 1 of N) - #119

Merged
gburd merged 1 commit into
masterfrom
docs/modernize
Jul 31, 2026
Merged

docs: modernization pipeline + C API tree (Phase 1 of N)#119
gburd merged 1 commit into
masterfrom
docs/modernize

Conversation

@gburd

@gburd gburd commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Phase 1 of N — the machinery + one proven tree

This is the foundation of the docs modernization in docs-src/PLAN.md:
a Markdown-source + generator pipeline that reverses the lost-source DocBook
HTML into maintainable Markdown and renders it to HTML. It is proven
end-to-end on one doc tree — the C API reference (470 pages) — with a
real no-loss content gate. The other 16 trees, man pages, PDF, and CI are
deliberately left for follow-up phases (clean seams left in place).

What's in this PR

Scaffold

  • docs-src/_data/site.toml — project name, copyright, base URL in ONE place.
    The version is read live from dist/RELEASE at build time (→ 5.3.33),
    never hard-coded.
  • docs-src/_templates/page.html.tmpl — the single HTML shell (header, nav,
    footer, version, copyright) that replaces the boilerplate previously
    duplicated across 2703 generated pages.

Generator — docs-src/build.py (stdlib + pandoc, no framework)

  • loads site.toml + live version, walks docs-src/**/*.md, renders each via
    pandoc + the template into docs-build/html/ (boilerplate injected once).
  • build_pdf() / build_man() are explicit phase-3/4 stubs so the shape
    is fixed and a follow-up wires pandoc without reshaping the file.

Extractor — docs-src/_migrate/extract.py (one-time reverse-DocBook)

  • strips the boilerplate (navheader, navfooter, Library Version 11.2.5.3,
    generator meta, stylesheet link, Prev/Next) by stable DocBook classes;
  • converts the semantic body to GFM (pandoc) + a cleanup pass that restores the
    API schema: front-matter (title/api-name/source) then Description /
    Parameters / Errors / Class / See-Also as headings;
  • rewrites foo.htmlfoo.md cross-links; keeps programlisting as verbatim
    fenced c code.

No-loss gate — docs-src/_migrate/verify.py

  • per page compares OLD visible prose vs NEW Markdown prose as a normalized word
    multiset, and separately asserts no code block or parameter/error sub-section
    was dropped; exits non-zero on any hard structural drop (CI-ready).

Completeness-verifier results (C API tree, 470 pages)

pages compared: 470
mean word retention: 99.98%
outliers (< 97% words OR structural drop): 0
no hard drops: every code block and parameter/error sub-section retained.

Before / after evidence

Before (old DocBook HTML) After (generated)
Version string Library Version 11.2.5.3 (hard-coded on every page) Version 5.3.33 (from dist/RELEASE, via template)
Boilerplate navheader/navfooter/libver/generator meta per page injected once from page.html.tmpl
dbget sections Description → Parameters → Errors → Class → See Also same, as ##/### headings
dbt struct code in <pre class="programlisting"> verbatim fenced ```c block
index method table 353 xref rows preserved, links rewritten .html

Tooling

  • flake.nix: added pandoc to the devShell (python3 already present) so the
    pipeline is reproducible under nix develop. Verified nix develop still
    evaluates.

Scope / safety

  • Touches only docs-src/, flake.nix (devShell input), .gitignore.
  • Does not touch src/ or test/ (a concurrent security review owns those).
  • Does not delete the old docs/ (kept until full sign-off).
  • docs-build/ is git-ignored (generated).

Honest status

  • Solid: the extractor + generator + no-loss gate, proven at 99.98%
    retention with zero drops on the full C API tree; version/copyright/header
    now single-sourced.
  • Follow-up phases must finish: C++/STL + guide trees (retarget
    extract.py, tune per-tree cleanup), man pages (build_man stub), PDF
    (build_pdf stub), the docs.yml CI (spelling/links/mandoc/completeness gate
    against the full db.h API surface), and publishing.

Reverse the lost-source DocBook HTML into maintainable Markdown, rendered to
HTML by a small stdlib+pandoc generator. Version/copyright/boilerplate now live
in ONE place instead of duplicated across 2703 generated pages.

Scaffold:
- docs-src/_data/site.toml        project/copyright/base-url (version read LIVE
                                  from dist/RELEASE, not hard-coded)
- docs-src/_templates/page.html.tmpl  the ONE HTML shell (header/nav/footer/
                                  version/copyright) replacing per-page boilerplate

Generator (docs-src/build.py):
- loads site.toml + live version (5.3.33 from dist/RELEASE)
- walks docs-src/**/*.md, renders each via pandoc + the template -> docs-build/html
- PDF and man outputs stubbed as clear phase-3/4 seams (build_pdf/build_man)

Extractor (docs-src/_migrate/extract.py):
- strips DocBook boilerplate (navheader/navfooter/libver/generator/Prev-Next) by
  stable classes, converts the semantic body to GFM, restores the API schema
  (title front-matter + Description/Parameters/Errors/Class/See-Also sections),
  rewrites foo.html->foo.md cross-links, keeps programlisting code verbatim

No-loss gate (docs-src/_migrate/verify.py):
- per-page OLD-prose vs NEW-prose word-multiset retention + code-block and
  parameter/error sub-section drop detection; exits non-zero on hard drop

Proven end-to-end on the C API tree (470 pages): extract -> build ->
verify reports mean word retention 99.98%, 0 outliers, 0 hard drops. Generated
pages carry version 5.3.33 (not the old 11.2.5.3) via the shared template.

flake.nix: add pandoc to the devShell so the pipeline is reproducible.
.gitignore: ignore docs-build/ (generated) and __pycache__.

Does NOT touch src/ or the old docs/ (kept until full sign-off). C++/STL and
the guide trees, man pages, PDF, and CI are follow-up phases.
@github-actions

Copy link
Copy Markdown

Coccinelle convention checks

No new violations. ✅

Resolved since baseline (2) -- update dist/cocci/baseline.txt to lock these in.
rule_mutex_unbalanced|MUTEX_UNBALANCED|src/crypto/mersenne/mt19937db.c|return (ret);
rule_mutex_unbalanced|MUTEX_UNBALANCED|src/mp/mp_register.c|return (ret);

@gburd
gburd merged commit e2e5ac3 into master Jul 31, 2026
47 of 48 checks passed
@gburd
gburd deleted the docs/modernize branch July 31, 2026 15:35
@github-actions

Copy link
Copy Markdown

ABI diff vs v5.3.33 (libabigail — authoritative)

Removed exported symbols (nm -D, _NNNN version suffix normalized)

None.


Advisory: libabigail/nm is the authoritative binary-ABI check; Coccinelle is complementary source-level early warning. See dist/cocci/README.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant