docs: Phase 4+5 — per-book PDF output + docs validation CI - #123
Closed
gburd wants to merge 3 commits into
Closed
Conversation
Implement build_pdf(): render each top-level book (2 API refs + 9 guides + articles' 2 sub-books = 13 PDFs) to docs-build/pdf/<book>.pdf. Chapters are concatenated in _meta.toml order (API refs: index first, then alphabetical); a CSS title page carries the live version (dist/RELEASE, 5.3.33) + copyright, with a running header/footer via CSS paged-media (_templates/pdf-print.css). Engine: pandoc(html) -> weasyprint. Chosen over LaTeX because it needs no TeX toolchain, is deterministic, and reuses the existing HTML path. All 13 books render (api_c 655pp, programmer_reference 370pp, ...) in ~3.5 min total. Also fixes two real bugs the link-check surfaced: pandoc_md_to_html now rewrites cross-tree .md links (../../api/c/foo.md) to .html, and build_html copies tree images (docs-src/<tree>/img/*) into the built site so <img> links resolve. Adds real stub pages for the 2 genuinely-undocumented public APIs (db_env_set_func_assert, db_env_set_win_security) written from their db.h prototypes + source comments, so the completeness gate can be a hard 100%. flake.nix devShell gains the docs validation toolchain (weasyprint, poppler-utils, mandoc, codespell, lychee, write-good) so CI matches local.
Add .github/workflows/docs.yml (modeled on ci.yml/fuzz.yml; nix devShell for
tool parity). Triggers: push to master, PRs touching docs-src/**, dispatch,
and a weekly schedule.
HARD gates (block PRs):
- build: build.py --no-pdf (HTML + man, 0 errors) + self-check
- no-loss: verify_all.py runs verify.py over all 13 migrated trees; fails on
any content drop (retention < threshold or a code/section hard drop)
- completeness: man_coverage.py --ci; 28/28 public functions documented, and
every uncovered method must be on the frozen allowlist (a NEW undocumented
API fails). Matcher also taught the dbsite/set_-drop DocBook stem shapes.
- spelling: spellcheck.py runs codespell keyed on (path, word), baselined to
the ~153 legacy typos so only newly-introduced typos fail
- internal link-check: lychee --offline + lychee.toml (deferred-tree and
un-migrated-asset links excluded); every migrated internal link must resolve
- man-lint: mandoc -Tlint over all .3, 0 ERRORS
ADVISORY (continue-on-error): prose (write-good passive/wordiness counts).
BEST-EFFORT/scheduled: pdf build + validate_pdf.py (slow), external link-check.
Every command was dry-run inside 'nix develop' and passes.
ABI diff vs
|
Coccinelle convention checksNo new violations. ✅ Resolved since baseline (2) -- update dist/cocci/baseline.txt to lock these in. |
am_second.md points at 'second.javas' (a malformed link to a Java API example in the deferred java tree); the file is absent in the upstream DocBook too, so it is a pre-existing dead link, not a migration regression. The flake-pinned lychee 0.24.1 (what CI's nix develop uses) extracts it where an older lychee did not, so add it to the exclude set. Full gate suite re-verified inside 'nix develop' (the exact CI env): 0 link errors, all gates pass.
Collaborator
Author
|
Superseded by docs/phase4-final (rebased onto master to drop the security fixes #122 already merged; now docs-only). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phases 4 (PDF) and 5 (CI) of the docs modernization. Builds on phases 1-3
(pipeline + all trees migrated + 787 man pages).
Phase 4 — PDF
build_pdf()renders one PDF per book — 13 books: 2 API refs (api/c,api/stl) + 9 guides +articles' 2 sub-books (inmemory,mssgtxt) — todocs-build/pdf/<book>.pdf. Chapters concatenated in_meta.tomlorder; aCSS title page carries the live version (
dist/RELEASE= 5.3.33) +copyright, with a running header/footer.
Engine: pandoc(html) → weasyprint. Chosen over LaTeX: no TeX toolchain,
deterministic, reuses the existing HTML path. All 13 render in ~3.5 min.
validate_pdf.pyasserts each PDF is non-empty, has a sane page count, andcarries
Berkeley DB 5.3.33on the title page — all 13 PASS.Two real bugs the link-check surfaced, fixed at the root:
pandoc_md_to_htmlnow rewrites cross-tree.md→.htmllinks (wassame-dir only; 1408 dangling internal links → 0).
build_htmlcopies tree images (docs-src/<tree>/img/*) into the site so<img>links resolve.Phase 5 — CI (
.github/workflows/docs.yml)Nix devShell for tool parity with local. Triggers: push→master, PRs touching
docs-src/**, dispatch, weekly schedule.HARD gates: build (HTML+man, 0 errors) · no-loss (
verify_all.py, all 13trees, 0 hard drops) · completeness (
man_coverage.py --ci, 28/28 functions +method allowlist) · spelling (
spellcheck.py, codespell baselined) · internallink-check (
lychee, 0 errors on migrated content) · man-lint (mandoc, 0ERRORS).
Advisory: prose (write-good). Best-effort/scheduled: PDF build+validate,
external link-check.
Completeness resolution: added real stub pages for the only 2
genuinely-undocumented APIs (
db_env_set_func_assert,db_env_set_win_security,from their db.h prototypes) → functions gate is a hard 100%. The remaining
uncovered methods (getters documented on their setter pages, callback/vtable
slots) are a frozen allowlist; a NEW undocumented method fails CI.
Every CI command was dry-run inside
nix developand passes.Not in scope
lychee.toml.no-loss-protected content); the spelling gate blocks only new typos.