Migrate registry-docgen to registry-dev

[thomashoneyman]

Closes #704. This moves @natefaubion's purescript-registry-docgen into this repository as the top-level docgen package and makes it the foundation for registry-owned package documentation, as discussed on Discord.

The goal is to build doc artifacts that we can store in a bucket, and those become the source of truth for Pursuit. HTML, search indexes, and other data can be rebuilt from the docs without needing to reach back out to GitHub or any dependency on purs publish directly.

What changed

The new docgen package includes lots of good things built by Nathan:

• A canonical docs schema, which is versioned and has the associated codecs
• Functions to convert docs from compiler/historical Pursuit documentation
• Deterministic package generation using the registry manifest and dependency resolutions
• Re-export resolution including selective imports, hidden imports, aliases, etc.
• Package and module HTML rendering, README rendering
• Exact packag eversion links for dependencies and re-exports
• Test covering codecs, generation, etc.

We use the registry manifest for almost everything, but since we have all the Pursuit backups we can use that for the documentation content itself. 107 package versions out of the ~6000 total have different dependency ranges in the registry and the docs used technically invalid versions, so we'd want to regenerate those.

Documentation storage

The main registry change in here, beyond the docgen target, is adding a separate DocsStorage effect with filesystem and S3 interpreters. Documentation is derived and replaceable, unlike immutable package tarballs, so it supports checking for existing artifacts, downloading, create-only uploading, explicitly replacing, and deleting docs.

The existing package storage effect has been renamed from Storage to PackageStorage to make the distinction clearer.

The production Nix configuration now supplies DOCS_BUCKET, defaulting to the purescript-registry-docs space that I created last week. DigitalOcean spaces cannot make the existence check and upload atomic, so production generation will still need to ensure there is only one writer for a given package version. That constraint is tracked in #525.

Historical documentation

This adds scripts for rendering documentation locally and validating historical Pursuit artifacts against registry manifests.

The migration audit found:

• 6,543 historical Pursuit artifacts;
• 4,240 artifacts corresponding to registry versions considered for migration;
• 4,133 which decode and agree with their authoritative registry manifests;
• 107 which should be regenerated from registry tarballs rather than imported;

All 107 regeneration candidates have stored tarballs and at least one recorded compatible compiler. This PR adds the conversion and validation machinery but does not perform the production migration.

Scope

This PR is for defining, validating, rendering, and storing documentation artifacts, but it doesn't do any of the following:

• generate or upload them during package publication;
• add retry or regeneration jobs;
• perform the production migration;
• replace the existing Pursuit application;
• add search or compiler-versioned Prim documentation;
• change package-source handling for monorepos.

Those are covered by #525, #789, #790, and #750.

The imported and adapted code is included under the repository's BSD-3-Clause license with permission from Nathan.

[thomashoneyman]

The Docgen tree is brought in basically as-is, which includes modules like this one — not sure if we need this separate JSON decoding module, but I didn't want to futz around with the implementation.

[natefaubion]

This was because:

• Legacy decoder doesn't need to round-trip. That would bloat the decoders significantly since there's a lot of sum types.
• We aren't depending on argonaut, AFAIK

There's not a great ecosystem for purescript-json, other than codec.

[greptile-apps]

Greptile Summary

This PR migrates purescript-registry-docgen into the monorepo as the docgen package and introduces a separate DocsStorage effect for reading and writing canonical documentation artifacts. The existing Storage effect is renamed to PackageStorage to make the distinction explicit.

• docgen package: Brings in a versioned documentation schema (Docs.purs), codecs, compiler-output conversion, re-export resolution (with full RefSet algebra for selective/hiding imports), and HTML rendering. Generation is driven by the registry manifest and exact dependency resolutions rather than the compiler's unconstrained output.
• DocsStorage effect: Provides upload (create-or-fail), replace (atomic overwrite), download, exists, and delete with two interpreters — a filesystem handler using link/rename for atomic operations, and an S3 handler targeting a separate purescript-registry-docs bucket. Read-only mode passes reads through to S3 while silently no-oping writes.
• PackageStorage rename: Mechanical rename of the previous Storage effect across API.purs, MatrixBuilder.purs, Server/Env.purs, and tests; no logic changes.

Confidence Score: 5/5

Safe to merge; all production-facing code paths are well-guarded and the non-atomic S3 upload constraint is explicitly documented and tracked separately.

The DocsStorage implementation uses OS-level atomicity (link for create-or-fail, rename for replace) in the filesystem handler, and explicitly acknowledges the S3 preflight-check limitation in both code comments and an open issue. The PackageStorage rename is purely mechanical. The docgen package carries extensive tests covering codecs, generation, and the complete storage lifecycle. The one observable gap (DeleteObjectsCommand errors silently discarded in deleteAllObjectsImpl) only affects E2E test teardown, not production paths.

Files Needing Attention: foreign/src/Foreign/S3.js — the new deleteAllObjectsImpl ignores per-key errors in the DeleteObjects response; low risk since it is only called from E2E test teardown.

Important Files Changed

Filename	Overview
app/src/App/Effect/DocsStorage.purs	New DocsStorage effect with filesystem (atomic via link/rename) and S3 interpreters; clean design with explicit read-only delegation to S3 for reads
app/src/App/Effect/PackageStorage.purs	Mechanical rename of Storage → PackageStorage throughout; no logic changes
foreign/src/Foreign/S3.js	Adds deleteAllObjectsImpl with V1 ListObjects pagination; errors from DeleteObjectsCommand (Quiet: true) are discarded silently
foreign/src/Foreign/S3.purs	Adds deleteAllObjects FFI binding; types match JS implementation correctly
docgen/src/Registry/Docgen/Generate.purs	Deterministic package generation from manifest and resolved dependencies; validates module names, source paths, and dependency membership before generating DocPackage
docgen/src/Registry/Docgen/Reexports.purs	Re-export resolution with RefSet algebra; Semigroup instances for RefAll/RefSet/RefHiding correctly model selective imports, hiding, and aliases
docgen/src/Registry/Docgen/Docs.purs	Canonical doc schema with versioning (schemaVersion = 1); all types are newtypes or records with clear derivations
app/src/App/Server/Env.purs	Wires DocsStorage into ServerEffects and runEffects; docsBucket defaults to purescript-registry-docs; shared S3 credentials for both storage effects
app-e2e/src/Test/E2E/DocsStorage.purs	Full lifecycle test (upload→exists→download→duplicate-rejection→replace→delete) with WireMock request verification; well-structured
app/test/App/Effect/DocsStorage.purs	Unit tests for handleFs covering the complete replaceable lifecycle; verifies identity mismatch rejection and parallel isolation
scripts/src/RenderDocs.purs	Script to render a canonical doc JSON artifact to Pursuit-compatible HTML paths; straightforward file I/O with proper error handling
scripts/src/VerifyLegacyDocs.purs	Verification script for historical Pursuit artifacts; uses an explicit allowlist for known-incompatible packages rather than silently skipping
docgen/src/Registry/Docgen/Codec.purs	Versioned codec for the canonical doc schema; uses deferred/lazy codecs for the mutually recursive DocType; taggedSum pattern is consistent

Sequence Diagram

sequenceDiagram
    participant Caller
    participant DocsStorage
    participant FS as Filesystem (handleFs)
    participant S3 as S3 (handleS3)

    Note over DocsStorage,FS: Upload (atomic create-or-fail)
    Caller->>DocsStorage: upload(docs)
    DocsStorage->>FS: writeDocsFile(link, path, docs)
    FS->>FS: ensureDirectory(parent)
    FS->>FS: mkdtemp(parent/.docs-)
    FS->>FS: writeJsonFile(tempPath, docs)
    FS->>FS: link(tempPath, path)  [EEXIST → already exists]
    FS->>FS: finally: remove(tempDir)
    FS-->>DocsStorage: Right unit / Left error

    Note over DocsStorage,S3: Upload (S3 - best-effort, non-atomic)
    Caller->>DocsStorage: upload(docs)
    DocsStorage->>S3: objectExists (listObjects preflight)
    S3-->>DocsStorage: exists?
    DocsStorage->>S3: putObject(key, buffer)
    S3-->>DocsStorage: Right unit / Left error

    Note over DocsStorage,FS: Replace (atomic overwrite)
    Caller->>DocsStorage: replace(docs)
    DocsStorage->>FS: writeDocsFile(rename, path, docs)
    FS->>FS: writeJsonFile(tempPath, docs)
    FS->>FS: rename(tempPath, path)  [atomic on same FS]
    FS->>FS: finally: remove(tempDir)
    FS-->>DocsStorage: Right unit

Loading

_{Reviews (3): Last reviewed commit: "Address documentation review feedback" | Re-trigger Greptile}

[natefaubion]

Note that all of these refer to existing deployed assets. I think for "production" these might need some alternative way of specifying these. Especially if spago is ever going to generate docs completely locally.

[natefaubion]

I don't remember what this is about exactly, but it probably needs some consideration. Right now only absolute uri's are allowed but I don't know if we want to do any other kind of linking, or even munging of links.

[natefaubion]

Some amount of maintenance risk here, though it's relatively benign. The converter automatically brackets things for presentation. If anything new is ever added, this needs to be addressed. It's also possible there are bugs lurking here (precedence is hard!). Bug fixes here will need docs regenerated, of course.