diff --git a/contributor_docs/reference_generation_process.md b/contributor_docs/reference_generation_process.md new file mode 100644 index 0000000000..917eddac6c --- /dev/null +++ b/contributor_docs/reference_generation_process.md @@ -0,0 +1,382 @@ + + +# Reference-generation process + +This document will describe the documentation-generation process which takes the p5.js v2.x source code and generates from it the pages of the [reference website](https://beta.p5js.org/reference/). + +## Need to know? + +You don't need to know _any_ of this to successfully write and maintain the reference comments in the p5.js library! If you've questions about that, see [Contributing to the p5.js reference](./contributing_to_the_p5js_reference.md). + +## Table of Contents + + * [The reference-generation process](#ref-gen-process) + * [Search index gen process (WIP)](#search-index-gen-process) + +## A warning: stale information? + +This document describes how things work at time of writing, but might be out of date! Check for changes in the key scripts listed in the following section, if in doubt. + +(Please consider contributing to bring this up-to-date if you notice it is incorrect.) + +## Key scripts + +Here are the most important documents in the process, currently. + +| Repo | Path | +| :------------ | :--- | +| p5.js-website | [/package.json](https://github.com/processing/p5.js-website/blob/2.0/package.json) | +| p5.js-website | [/src/scripts/builders/reference.ts](https://github.com/processing/p5.js-website/blob/2.0/src/scripts/builders/reference.ts)| +| p5.js | [/package.json](https://github.com/processing/p5.js/blob/dev-2.0/package.json) | +| p5.js | [/utils/convert.mjs](https://github.com/processing/p5.js/blob/dev-2.0/utils/convert.mjs) | + +## What's generated from the source? + +There are some other artifacts generated from the same source: + +* Type declarations supporting autocompletion, intellisense, and type-checking. See [type-generation process](./type_generation_process.md) + +* Parameter validation data for the Friendly Error System (FES). + +```mermaid +--- +title: What's generated from p5.js source? +--- +graph LR + js[*.js files] + js --> ref[Reference pages] + js --> typedecls[Type declarations
for type-checking
& intellisense] + js --> fesParamData[Parameter data for FES] +``` + +## The reference-generation process + + + +```mermaid +--- +title: Flow diagram of reference-generation +--- +graph TD + + %% nodes + + websiteNpmBuildReference[[website repo
npm run
build:reference]] + + cloneP5Repo[clones p5 repo] + + npmRunDocs[[p5 repo
npm run
docs]] + + docBuild[[documentation
build
/src/**/*.js]] + + convertDocsToMDX[[website repo
convertDocsToMDX]] + + contentReferenceMDX["/src/content
/reference/**/*.mdx"]@{shape: docs} + + + + + %% connections + + websiteNpmBuildReference --> cloneP5Repo + websiteNpmBuildReference --> npmRunDocs + + npmRunDocs --> docBuild + + docBuild --> /docs/data.json@{ shape: doc } + + /docs/data.json --> convert[[convert.mjs]] + + convert --> /docs/reference/data.json@{ shape: doc } + + convert --> /docs/reference/data.min.json@{ shape: doc } + + /docs/reference/data.json --> convertDocsToMDX + + convertDocsToMDX --> contentReferenceMDX + + contentReferenceMDX --> npmRunBuild[[npm run
build]] + + subgraph "Production build only" + contentReferenceHTML["/dist/reference/**/*.html"]@{shape: docs} + + npmRunBuild --> astroBuild[[astro build]] + astroBuild --> contentReferenceHTML + end + + + subgraph legend + p5RepoActionExample[[p5 repo
action]] + websiteRepoActionExample[[website repo
action]] + end + + %% styling - class declarations and assignments + + class npmRunDocs,docBuild,convert,p5RepoActionExample p5RepoAction; + class websiteNpmBuildReference,convertDocsToMDX,npmRunBuild,astroBuild,websiteRepoActionExample websiteRepoAction; + + classDef p5RepoAction stroke:#ED225D,stroke-width:2px; + classDef websiteRepoAction stroke:#000000,stroke-width:2px; + +``` + +### How the process starts: + +For a production deploy, the process _currently_ happens in [a release process workflow](https://github.com/processing/p5.js/blob/dev-2.0/.github/workflows/release-workflow-v2.yml) on Github Actions CI. However, you can follow these steps on a local clone of the p5.js-website repo. + +The process is started with the execution of the following command from a check-out of p5.js-website repo: + +```bash +npm build:reference +``` + +This calls [p5.js-website:/src/scripts/builders/reference.ts](https://github.com/processing/p5.js-website/blob/2.0/src/scripts/builders/reference.ts), which starts by: +1. cloning the _p5.js_ repo into a temporary working directory, and +2. from there running: + +```bash +npm run docs +``` + +### p5.js: npm run docs + +In turn this command runs the following single-line command: + +```bash +"documentation build + ./src/**/*.js ./src/**/**/*.js + -o ./docs/data.json + && node ./utils/convert.mjs" +``` + +This is actually _two_ commands in sequence: +* First, `documentation build ...`, and then - if the first command was successful +* `node ./utils/convert.mjs`. + +We'll look at these in turn. + +#### Step 1: `documentation build...` command + +This command turns JSDoc comment blocks across all the javascript files into a single structured JSON file ready for further processing. + +In more detail: + +```bash +documentation build + ./src/**/*.js ./src/**/**/*.js + -o ./docs/data.json +``` + +`documentation build` is the [standard way](https://github.com/documentationjs/documentation?tab=readme-ov-file#user-guide) to invoke the tool [documentation.js](https://documentation.js.org/) to build documentation. + +Next we tell it which files to work on. + +`./src/**/*.js ./src/**/**/*.js` is an expression which will expand to match all .js files under the src directory at depths of one and two subdirectories. That covers all the .js files that currently have documentation comment blocks in them. + +Examples of files which will match: + +* `src/image/pixels.js` +* `src/color/color_spaces/hsb.js` + +The `-o` is used to specify the output file. `documentation.js`'s default output format is JSON, so JSON content is written into the output file. + +Aside: It's important to note that the tool _does not_ generate HTML pages, even though that's a common use for tools like documentation.js. + +This generated JSON file will be converted further by the next step, to prepare the data for the website build process. + +It will _also_ be used by other build processes - e.g. to build the types - see [the type-generation process](#type-gen-process). + +You can run this command yourself locally and inspect the output file to see the gathered data. + +You can also run `npx documentation build --help` to read the manual page for documentation.js. + +#### Step 2: `node ./utils/convert.mjs` command + +The second command is +```bash +node ./utils/convert.mjs +``` + +Note this will _only_ run if the documentation build command runs successfully. + +Here's the [/utils/convert.mjs](https://github.com/processing/p5.js/blob/dev-2.0/utils/convert.mjs) from the p5 repo. + +The `./utils/convert.mjs` script reads the output file generated by the `documentation build` command (the JSON file `/docs/data.json`), and creates three new output files: +* `/docs/reference/data.json` - Used by the p5.js-website to generate the reference pages +* `/docs/reference/data.min.json` - A smaller version of the above +* `/docs/parameterData.json` - Used by the FES\* + +The file generated initially by `documentation build` is _very_ large with a lot of data our pages won't need. Further, pieces of information are also scattered over the file - we'd like to group them together into a structure that will make it easier to use in later webpage generation. +The convert script has taken only what's necessary and grouped it together into the new data file `/docs/reference/data.json` and its minified version. + +This file will be used by the p5.js-website to generate the final pages. + +\* The convert script has _also_ generated `/docs/parameterData.json` which is used by the friendly error system (FES) at runtime to validate function parameters. We won't discuss that further here. + +#### Excerpt of /docs/reference/data.json + +Here's an excerpt from `/docs/reference/data.json`, showing how our [noFill()](https://beta.p5js.org/reference/p5/noFill/) function documentation is now represented at the end of both steps taken by `npm run docs`: + +```json +{ + "name": "noFill", + "file": "src/color/setting.js", + "line": 1342, + "itemtype": "method", + "description": "

Disables setting the fill color for shapes.

\n

Calling noFill() is the same as making the fill completely transparent,\nas in fill(0, 0). If both noStroke() and\nnoFill() are called, nothing will be drawn to the screen.

\n", + "example": [ + "function setup() {\n createCanvas(100, 100);\n\n square(32, 10, 35);\n\n noFill();\n square(32, 55, 35);\n}", + "...second example goes here..." + ], + "overloads": [ + { + "params": [], + "chainable": 1 + } + ], + "class": "p5", + "static": false, + "module": "Color", + "submodule": "Setting" + }, +``` + +(I've shortened the examples) + +### after p5.js npm run docs - p5.js-website continues + +Once the p5.js `npm run docs` script has finished, the p5.js-website's `npm run build:reference` command continues, as follows: + +#### Summary: + +* Loads the generated `./docs/reference/data.json` +* Generates an MDX file for each entry in the data (e.g. each p5 function or p5 variable gets its own .mdx file). +* That's all! The production of these MDX files is as far as `npm run build:reference` goes. + +Who consumes this work? + + * For developers, the astro dev server can now read these MDX files directly to show a preview of the website. + * In CI, a separate step will commit the MDX files into git. + * In CI, a separate step will run `astro build` to convert the MDX files into HTML. + +#### More detail on MDX generation: + +From the input `./docs/reference/data.json`... + +For each module: + * generates the file path to which generated pages will be stored +For each feature: + * generates the MDX content for the relevant page + + +Example +We've seen previously the entry for `noFill` in the docs/reference/data.json. + +Here's the path decided for its mdx file: +`src/content/reference/en/p5/noFill.mdx` + +And here's what the contents of noFill.mdx looks like: + + +```mdx +--- +title: noFill +module: Color +submodule: Setting +file: src/color/setting.js +description: > +

Disables setting the fill color for shapes.

+ +

Calling noFill() is the same as making the fill completely + transparent, + + as in fill(0, 0). If both noStroke() and + + noFill() are called, nothing will be drawn to the screen.

+line: 1340 +isConstructor: false +itemtype: method +example: + - |- + function setup() { + createCanvas(100, 100); + + background(200); + + // Draw the top square. + square(32, 10, 35); + + // Draw the bottom square. + noFill(); + square(32, 55, 35); + + describe('A white square on above an empty square. Both squares have black outlines.'); + } + - |- + function setup() { + //second example cuts short for brevity + } + +class: p5 +overloads: + - params: [] + chainable: 1 +chainable: false +--- + + +# noFill + +``` +#### Run npm run dev + +For developers who are running the dev server to preview their work (e.g. with `npm run dev` or `npm run custom:dev`), the artifacts have now been generated that allow them to do that. + +Commands like `npm run dev` run astro's dev server which reads our generated MDX files as it runs, and will even monitor them for changes, allowing you to make quick experimental edits. + +If you haven't before, try running +```bash +npm run dev +``` + +visiting `/reference/p5/noFill/` on the url that appears, + +and then editing (and saving!) `src/content/reference/en/p5/noFill.mdx` to see your changes reflected. + +For many p5 contributors, this is enough. but there is a final step! + +### Final astro build: converting MDX to HTML + +For final preparation of the production website, there is one last conversion step for our data to go through. + +`npm run build` delegates to the astro tool with `astro build`. + +The "astro build" process converts all MDX files to HTML files (applying common page layouts and more), preparing the HTML files into a `dist/` folder, ready for deployment to a web host (like netlify or cloudflare). + +(Feasibly, you could also distribute that folder of HTML files on a USB drive, say, for users to have _offline_ access to the p5 reference). + +## Search index generation + +More info is needed in this section. Consider contributing! + + +On the p5js-website repo, we run `npm run build:search` + +```mermaid +graph TD + contentReferenceMDX["/src/content/reference"]@{shape: docs} + contentExamples["/src/content/examples"]@{shape: docs} + + npmBuildSearch[[npm run
build:search]] + npmBuildSearch --> buildSearchIndices[[buildSearchIndices]] --> genSearchIndexForExamples[[generateSearchIndex
examples]] + buildSearchIndices --> genSearchIndexForReference[[generateSearchIndex
reference]] + contentReferenceMDX --> genSearchIndexForReference + contentExamples --> genSearchIndexForExamples + buildSearchIndices --> saveSearchIndex[[saveSearchIndex]] --> outDir[[public/search-indices en.json, ja.json, ...]]@{shape: docs} +``` + + +## Contributing to this document + +Notice any errors or places for improvement? Please consider contributing to this document! It lives at [/contributor_docs/reference_generation_process.md](https://github.com/processing/p5.js/blob/dev-2.0/contributor_docs/reference_generation_process.md) in the p5.js repo. Alternatively, [let us know](https://github.com/processing/p5.js/issues)! \ No newline at end of file diff --git a/contributor_docs/type_generation_process.md b/contributor_docs/type_generation_process.md new file mode 100644 index 0000000000..bd2020fd48 --- /dev/null +++ b/contributor_docs/type_generation_process.md @@ -0,0 +1,132 @@ + + +# The type-generation process + +This document outlines the process that generates type-declaration files for the p5 v2 library. (Types for v1 are not discussed here). + +## What are the type declarations useful for? + +The type-declaration files contain lists of every function, variable, and class in the p5.js library, the description and details of those elements. + +These can be used by any compatible editor or code to provide (for example): + +* Auto-completion & intellisense, including... + - On-hover documentation of the P5 functions and variables + - Completion suggestions: e.g. display of available methods and properties on the current object + - Parameter hints: the name and required type of the current argument being passed to a function +* Type-checking of typescript OR javascript sketches + +It is *not* the case that the declarations are only for TypeScript! The above functionality can be provided when the sketches in question are in JavaScript, too! + +## Inputs + +There are currently *two* inputs for the library's type declarations: +* The JSDoc comments in the javascript files in the source directory +* Ad-hoc overrides and additions for certain cases, stored in a patching script + +## Outputs + +* `types/p5.d.ts` +* `types/global.d.ts` + +## Flow diagram for type-generation + +```mermaid +--- +title: Generation of p5.js type-declarations +--- +graph TD + p5dts1["/types/p5.d.ts"]@{shape: doc} + p5dts2["/types/p5.d.ts"]@{shape: doc} + globaldts1["/types/global.d.ts"]@{shape: doc} + globaldts2["/types/global.d.ts"]@{shape: doc} + + npmRunDocs[[npm run
docs]] --> docBuild[[documentation
build
/src/**/*.js]] + docBuild --> /docs/data.json@{ shape: doc } + /docs/data.json --> typescript[[typescript.mjs]] + + npmRunGenerateTypes[[npm run
generate-types]] + npmRunGenerateTypes --> typescript + + typescript --> p5dts1 + typescript --> globaldts1 + typescript --> patch[["patch.mjs
applyPatches()"]] + p5dts1 --> patch + globaldts1 --> patch + patch --> globaldts2@{shape: doc} + patch --> p5dts2@{shape: doc} + +``` +## The process described + +### Starting the process + +The process is run with the following command on the p5js repo: + +`npm run generate-types` runs: +```bash +"npm run docs && node utils/typescript.mjs" +``` + +### Step 1. npm run docs + +The first step in the process is common to the reference-generation process, and so is documented in: [Reference-generation process](./reference_generation_process.md) + +Specifically, the JSON file `/docs/data.json` is generated by `documentation build` from the p5.js source code. This will be used as an input to the next step. + +### Step 2. utils/typescript.mjs + +A better name for this script might be `generate-types`. + +This script: + +1. takes an initial pass at generating types from `docs/data.json`, producing: + +* `types/p5.d.ts` +* `types/global.d.ts` + +2. it then delegates to the patching script, which amends the above files in place, into a final form. + +### Step 2.1. Patching types +In a few specialized cases, the normal JSDoc way of documenting function parameter and return types fails. + +To deal with these situations, a patching mechanism is in place which allows us to inject the correct types directly into the type-declaration files. + +This is done in [./utils/patch.mjs](./utils/patch.mjs) through a series of calls to `replace()` which replaces the generated type coming out of the first phase of type-generation with the correct type. + +#### Examples of patching: + +In the following example, we correct the type for the `mouseButton` variable, which is initially generated incompletely from the JSDoc as: `'mouseButton: object;'` +This call to replace will look through the `p5.d.ts` file, looking for the above line, and replacing it with the corrected type - an object with three boolean properties. + +```js +replace( + 'p5.d.ts', + 'mouseButton: object;', + 'mouseButton: { left: boolean; center: boolean; right: boolean };' + ); +``` +The next example replaces, in _both_ files, the generated type of the `shuffle` function with a corrected generic version which states the input and output arrays will have the same type of elements. + +```js + replace( + ['p5.d.ts', 'global.d.ts'], + 'shuffle(array: any[], modify?: boolean): any[];', + 'shuffle(array: T[], modify?: boolean): T[];' + ); +``` + + +## What about p5 v1 types? + +The process described in this document does not apply to the types for the p5 v1 library. There are (often out-of-date) types at [definitely-typed](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/p5), packaged to npm as @types/p5. This document only discusses the types for p5 v2. + + +## Aside: "Why not just do X..." + +It is more common in industry for such type-declarations to be generated by the TypeScript compiler (tsc) or similar. However, in prioritising the ease-of-use experience for the library end user, the p5 library does some things in ways that mean this standard process is unsuitable. + +## See also + +The [reference-generation process](./reference_generation_process.md). +