Introduce incremental `gen-api-docs` command to support watch functionality

[StephenBeirlaen]

Is your feature request related to a problem?

Currently, with the existing gen-api-docs and clean-api-docs commands at our disposal I am unable to provide a workable "watch" functionality to enable quick iterative development. Due to how these commands are designed, they do not regenerate updated MDX files after changes have been made to the source OpenAPI schema (this has already been described in #615).

So, to end up with up-to-date MDX files, we have to clean all files and regenerate them.

Our problem with this design choice is that our Docusaurus dev server seems to be stalling and going OOM after the watcher sees so many MDX files being removed and re-added by this plugin. Even adding a timeout to the watcher does not prevent this: webpackConfig.watchOptions.aggregateTimeout = 10000;. The server just hangs and ultimately crashes. This means that we have to restart the Docusaurus dev server, which means that every development iteration takes at least 30 seconds, depending on the size of your site.

As a workaround, we found that when we generate files to a temporary directory, and then use rsync -r -v --size-only to copy only the changed files from the temporary directory back to the intentional directory, this strain on Webpack does not occur and the builds are working fine. This solution however is impractical, as the info_path of the generated mdx files then contains references to our temporary build directory.

Describe the solution you'd like

I would like to ask to reconsider the idea that @shmichael proposed in #615, which is to build the full file tree & touch only the actually modified files, to reduce IO and improve watcher performance.

Additionally, to remove all deleted files, one could keep track of the generated file tree in a temporary cache file, and on the next build, remove all existing files that were not part of this tree.

I am willing to help contribute towards an implementation, however I am unsure how this fits alongside the existing workflow of the gen & clean commands. Maybe a gen-api-docs:watch command could be introduced?

This would enable a quick regeneration & development cycle.

Describe alternatives you've considered

• Build using gen-api-docs to a temporary dir, and use rsync -r -v --size-only to copy only the changed files from the temporary directory back to the intentional directory
• Use Nodemon to provide a watch function nodemon --delay 1000ms --watch <my-openapi-spec-folder> --ext 'yaml' --exec 'yarn api-docs:tmp-dir:clean && yarn gen-api-docs && yarn api-docs:tmp-dir:rsync'
• Prevent stalling the Docusaurus dev server using webpackConfig.watchOptions.aggregateTimeout = 10000;

Additional context

• gen-api-docs does not regenerate files when the openAPI spec changes #615

Thanks in advance!