-
Notifications
You must be signed in to change notification settings - Fork 0
docs: add docs/infrastructure.md, README contents #96
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
threexc
wants to merge
6
commits into
main
Choose a base branch
from
tgamblin/docs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
8af5536
docs/infrastructure: add guide
threexc d817c9d
docs: convert old docs for GitHub Pages
threexc b156680
README: add link to GitHub Pages deployment
threexc 5bda505
docs: add requirements files for building
threexc 411a5d4
docs/packages/generate_packages_doc: fix Markdown-in-raw-HTML parsing
threexc 7b49389
docs: packages: add updated docs list, index.md
threexc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| title: RISE Python Wheels | ||
| description: riscv64 binary wheels for popular Python projects | ||
| remote_theme: just-the-docs/just-the-docs | ||
|
|
||
| url: https://riseproject-dev.github.io | ||
| baseurl: /python-wheels | ||
|
|
||
| plugins: | ||
| - jekyll-remote-theme | ||
|
|
||
| color_scheme: light | ||
| search_enabled: true | ||
| heading_anchors: true | ||
|
|
||
| aux_links: | ||
| "GitHub": | ||
| - "https://github.com/riseproject-dev/python-wheels" | ||
|
|
||
| exclude: | ||
| - packages/generate_packages_doc.py | ||
| - packages/*.yaml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| --- | ||
| title: Home | ||
| layout: default | ||
| nav_order: 1 | ||
| --- | ||
|
|
||
| # RISE Python Wheels | ||
|
|
||
| RISE Python Wheels is a public project enabling RISC-V support for the Python | ||
| ecosystem. It uses the RISE [RISC-V | ||
| Runners](https://riscv-runners.riseproject.dev/) project to build wheels on | ||
| native riscv64 hardware, with the goal of maintaining a riscv64-specific | ||
| package repository for projects where upstream are not yet ready or able to | ||
| perform builds themselves. | ||
|
|
||
| This work continues the earlier | ||
| [wheel_builder](https://gitlab.com/riseproject/python/wheel_builder) project, | ||
| which hosts binary wheels for a variety of Python modules. | ||
|
|
||
| For more information about RISE, visit the [project | ||
| website](https://riseproject.dev/). | ||
|
|
||
| ## Sections | ||
|
|
||
| - [Motivation](motivation.html) | ||
| - [Installing the Packages](install.html) | ||
| - [Manylinux_2_35 and Manylinux_2_39](manylinux.html) | ||
| - [Supported Packages](packages/) | ||
| - [Infrastructure](infrastructure.html) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| --- | ||
| title: Infrastructure | ||
| layout: default | ||
| nav_order: 6 | ||
| --- | ||
|
|
||
| # Infrastructure | ||
|
|
||
| The [python-wheels](https://github.com/riseproject-dev/python-wheels) project | ||
| makes use of some supplemental infrastructure available both on GitHub, and | ||
| externally. The following sections outline these services and how to | ||
| configure/maintain them. | ||
|
|
||
| ## RISE RISC-V Runners | ||
|
|
||
| For RISE-hosted builds of Python wheels and upstream submissions, the | ||
| `python-wheels` repository uses the [RISC-V | ||
| Runners](https://riscv-runners.riseproject.dev/) project. It is already enabled | ||
| for the repository; to reconfigure, see the [installation | ||
| guide](https://riscv-runners.riseproject.dev/docs/getting-started/install). | ||
|
|
||
| ## GitLab | ||
|
|
||
| The `python-wheels` project is a continuation of | ||
| [wheel_builder](https://gitlab.com/riseproject/python/wheel_builder), which is | ||
| hosted on GitLab. This project leverages the GitLab CI/CD framework to create | ||
| pipelines for building, testing, and deploying wheels, along with automatic | ||
| documentation updates and periodic checks for new package versions upstream. | ||
|
|
||
| While development and maintenance now primarily occur on GitHub, the existing | ||
| GitLab infrastructure (particularly the package registry) must be maintained. | ||
| The original wheel_builder | ||
| [documentation](https://riseproject.gitlab.io/python/wheel_builder/index.html) | ||
| is maintained for reference purposes, and provides a detailed explanation of how | ||
| the project was originally structured. | ||
|
|
||
| ### GitLab for Open Source | ||
|
|
||
| We apply for the [GitLab for Open | ||
| Source](https://about.gitlab.com/solutions/open-source/join/) program. This | ||
| provides us with the GitLab Ultimate tier at no cost. Membership requires annual | ||
| renewal. | ||
|
|
||
| ### CI/CD Pipeline Tokens and Access | ||
|
|
||
| The GitLab CI/CD pipelines make use of project tokens, generated at different | ||
| project scopes. Two tokens are defined: | ||
|
|
||
| 1. `PYTHON_CI_GROUP_TOKEN` - Defined at the "Group" | ||
| ([Python](https://gitlab.com/riseproject/python)) level. Used for triggering | ||
| pipelines for merge requests and most automatic updates and checks. The value | ||
| of this token is inherited by `wheel_builder` as `PYTHON_CI_GROUP_TOKEN` | ||
| under `Settings -> CI/CD Settings -> Variables` (see the [CI/CD variables | ||
| panel](https://gitlab.com/riseproject/python/wheel_builder/-/settings/ci_cd#js-cicd-variables-settings)). | ||
| 2. `rise-github-package-registry` - Defined at the "Repository" | ||
| ([wheel_builder](https://gitlab.com/riseproject/python/wheel_builder/-/settings/repository#js-deploy-tokens)) | ||
| level. Used for pushing wheels to the GitLab package registry from GitHub. | ||
|
|
||
| These tokens are generated with a 1-year expiry timeline, so they need to be | ||
| regenerated by a project administrator/owner. | ||
|
|
||
| #### Token Scope Reference | ||
|
|
||
| | Token | Level | Role | Scopes | Used For | Expiry | | ||
| |-------|-------|------|--------|----------|--------| | ||
| | `PYTHON_CI_GROUP_TOKEN` | Group ([Python](https://gitlab.com/riseproject/python)) | Maintainer | `api`, `read_api`, `read_repository`, `write_repository` | MR pipeline triggers, auto updates/checks | 1 year (Unknown expiry) | | ||
| | `rise-github-package-registry` | Repository deploy token ([wheel_builder](https://gitlab.com/riseproject/python/wheel_builder/-/settings/repository#js-deploy-tokens)) | n/a (deploy token) | `read_package_registry`, `write_package_registry` | Push wheels GitHub → GitLab package registry | 1 year (June 29th, 2027) | | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| --- | ||
| title: Installing the Packages | ||
| layout: default | ||
| nav_order: 3 | ||
| --- | ||
|
|
||
| # Installing the Packages | ||
|
|
||
| The built wheels are hosted in the [package | ||
| registry](https://gitlab.com/riseproject/python/wheel_builder/-/packages) | ||
| associated with the riseproject/python/wheel_builder project. To install them, | ||
| first upgrade pip to the latest version, e.g., | ||
|
|
||
| ```bash | ||
| python -m pip install --upgrade pip | ||
| ``` | ||
|
|
||
| and then pass the `--index-url` option to the install command to tell pip to | ||
| pull packages from the registry associated with this project, e.g., | ||
|
|
||
| ```bash | ||
| python -m pip install scipy --index-url https://gitlab.com/api/v4/projects/riseproject%2Fpython%2Fwheel_builder/packages/pypi/simple | ||
| ``` | ||
|
|
||
| {: .note } | ||
| > Some riscv64 packages are now built and published upstream on PyPI. We will | ||
| > no longer build, upgrade and publish them as part of the wheel_builder | ||
| > project. | ||
| > | ||
| > To make sure to always install the latest version available, use | ||
| > | ||
| > ```bash | ||
| > python -m pip install scipy --prefer-binary --extra-index-url https://gitlab.com/api/v4/projects/riseproject%2Fpython%2Fwheel_builder/packages/pypi/simple | ||
| > ``` | ||
| > | ||
| > This will: | ||
| > | ||
| > - search both PyPI and the internal registry. | ||
| > - pick the highest available version. | ||
| > - prefer binary wheels over source distributions. | ||
| > | ||
| > This ensures we get wheels from PyPI when available, while falling back to | ||
| > our registry for packages without riscv64 wheels, avoiding unnecessary source | ||
| > builds. | ||
|
|
||
| {: .warning } | ||
| > In general, `--extra-index-url` should be used very carefully (see | ||
| > <https://peps.python.org/pep-0708/#motivation>). | ||
| > | ||
| > The wheel_builder registry only contains wheels for packages that have | ||
| > pre-existing counterparts in PyPI, so it is safe to use it in that context. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| --- | ||
| title: Manylinux_2_35 and Manylinux_2_39 | ||
| layout: default | ||
| nav_order: 4 | ||
| --- | ||
|
|
||
| # Manylinux_2_35 and Manylinux_2_39 | ||
|
|
||
| The packages distributed in this registry are a mix of manylinux_2_35 and | ||
| manylinux_2_39 wheels. Support for manylinux wheels for riscv64 was added to | ||
| pip in version 24.1 so you must have pip 24.1 or greater installed locally in | ||
| order to install the packages. This is why we recommend upgrading pip above | ||
| before installing anything. Prior to the release of pip 24.1, some wheels with | ||
| the linux tag were created for a select set of packages and uploaded to this | ||
| package registry. If your version of pip is older than 24.1 pip will install | ||
| these older wheels instead of the manylinux wheels, if they are available. | ||
| This is probably not what you want so do upgrade pip. | ||
|
|
||
| These older packages in this registry with the 'linux' platform tag, e.g., | ||
| `numpy-1.26.4-cp39-cp39-linux_riscv64.whl`, actually behave like manylinux_2_35 | ||
| wheels. These wheels were generated because we didn't want to wait for the | ||
| release of pip 24.1 to begin distributing riscv64 wheels, so while we were | ||
| waiting, we used a modified auditwheel that generates wheels with the linux | ||
| platform tag that mostly behave like manylinux_2_35 wheels. This means that | ||
| the wheels will work on any Linux distribution with glibc 2.35 or greater. The | ||
| wheels also vendor all of their dependencies that are not on the | ||
| manylinux_2_35 whitelist, e.g., OpenBLAS and libgfortran. One downside of | ||
| misusing the platform tag in this way is that pip will not warn you when you | ||
| install the wheels on distributions that use a glibc older than 2.35. In this | ||
| case the wheels will install but will be unlikely to work. As most riscv64 | ||
| users are expected to use recent distributions to benefit from the latest | ||
| riscv64 support from the kernel and toolchains, this hopefully won't be too | ||
| much of an issue. | ||
|
|
||
| The manylinux_2_39 wheels are built with the upstream manylinux_2_39_riscv64 | ||
| image. There are a few exceptions that were built with a custom | ||
| manylinux_2_39_riscv64 based on RockyLinux 10 which we used for a short period | ||
| of time before the official manylinux_2_39_riscv64 images were published. | ||
|
|
||
| All earlier wheels are built using a custom manylinux_2_35 based on Ubuntu | ||
| 22.04. See the local | ||
| [riseproject/python/manylinux](https://gitlab.com/riseproject/python/manylinux) | ||
| fork for more details. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| title: Motivation | ||
| layout: default | ||
| nav_order: 2 | ||
| --- | ||
|
|
||
| # Motivation | ||
|
|
||
| Python is commonly used in the fields of scientific computing, data analysis and | ||
| machine learning. However, the Python packages used in these disciplines aren't | ||
| wholly written in Python - they also contain a lot of code written in C/C++ or | ||
| other languages which needs to be built as part of the module. Such projects | ||
| typically create binary wheels for each of their releases and upload these | ||
| wheels to pypi, the Python Package Index. This allows users to easily and | ||
| quickly install tested, prebuilt versions of their favourite projects using the | ||
| pip (or [uv](https://docs.astral.sh/uv/)) tool. | ||
|
|
||
| Until recently, Python packaging infrastructure like | ||
| [auditwheel](https://github.com/pypa/auditwheel), | ||
| [cibuildwheel](https://github.com/pypa/cibuildwheel), and | ||
| [manylinux](https://github.com/pypa/manylinux) did not support riscv64, and no | ||
| native runners for GitHub Actions were available. This made supporting the | ||
| architecture difficult for open-source projects without complicated build | ||
| processes and emulated systems. However, the aforementioned infrastructure now | ||
| supports riscv64, and with the RISE [RISC-V | ||
| Runners](https://riscv-runners.riseproject.dev/) project, maintainers have the | ||
| option of building binary wheels on native riscv64 platforms. The RISE Python | ||
| Wheels project's goal is to accelerate this adoption and ensure that the riscv64 | ||
| architecture is fully-supported for data science and machine learning | ||
| applications. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| --- | ||
| title: aiohttp | ||
| layout: default | ||
| parent: Supported Packages | ||
| --- | ||
|
|
||
| <!-- Auto-generated by generate_packages_doc.py. Do not edit manually. --> | ||
|
|
||
| # aiohttp | ||
|
|
||
| {: .warning } | ||
| > This package is deprecated. PyPI now publishes newer versions of | ||
| > this package for riscv64, and we will no longer maintain this | ||
| > package. Please use the version from [PyPI](https://pypi.org/project/aiohttp/) instead. | ||
| > | ||
| > If you need a specific version between the latest available here | ||
| > and the first available on PyPI, please open an | ||
| > [Issue](https://gitlab.com/riseproject/python/wheel_builder/-/issues). | ||
|
|
||
| - **Source Code:** [https://github.com/aio-libs/aiohttp](https://github.com/aio-libs/aiohttp) | ||
| - **Supported versions:** | ||
|
|
||
| <details markdown="1" open> | ||
| <summary><strong>3.12.14 (latest)</strong></summary> | ||
|
|
||
| ```bash | ||
| pip install aiohttp --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple | ||
| ``` | ||
|
|
||
| - **License:** Apache-2.0 | ||
| - **Download files:** [https://gitlab.com/riseproject/python/wheel_builder/-/packages/42690263](https://gitlab.com/riseproject/python/wheel_builder/-/packages/42690263) | ||
| - **Patch applied for this version:** [https://gitlab.com/riseproject/python/wheel_builder/-/tree/main/wheel_builder/aiohttp/patches/v3.12.14](https://gitlab.com/riseproject/python/wheel_builder/-/tree/main/wheel_builder/aiohttp/patches/v3.12.14) | ||
| </details> | ||
|
|
||
| <details markdown="1"> | ||
| <summary><strong>3.11.18</strong></summary> | ||
|
|
||
| ```bash | ||
| pip install aiohttp==3.11.18 --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple | ||
| ``` | ||
|
|
||
| - **License:** Apache-2.0 | ||
| - **Download files:** [https://gitlab.com/riseproject/python/wheel_builder/-/packages/40999682](https://gitlab.com/riseproject/python/wheel_builder/-/packages/40999682) | ||
| - **Patch applied for this version:** [https://gitlab.com/riseproject/python/wheel_builder/-/tree/main/wheel_builder/aiohttp/patches/v3.11.18](https://gitlab.com/riseproject/python/wheel_builder/-/tree/main/wheel_builder/aiohttp/patches/v3.11.18) | ||
| </details> | ||
|
|
||
|
|
||
| {: .warning } | ||
| > We disable the tests in tests/autobahn/test_autobahn as they depend on | ||
| > docker_on_whales and we don't want to be testing docker builds as part of | ||
| > the pipeline. | ||
| > | ||
| > We also disable a flaky uvloop test, and tests that contain the following | ||
| > patterns, involving consistent failures related to networking or permissions: | ||
| > | ||
| > - test_no_warnings | ||
| > - test_static_directory_without_read_permission | ||
| > - test_static_file_without_read_permission | ||
| > - test_uvloop_secure_https_proxy |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| --- | ||
| title: argon2-cffi-bindings | ||
| layout: default | ||
| parent: Supported Packages | ||
| --- | ||
|
|
||
| <!-- Auto-generated by generate_packages_doc.py. Do not edit manually. --> | ||
|
|
||
| # argon2-cffi-bindings | ||
|
|
||
| - **Source Code:** [https://github.com/hynek/argon2-cffi-bindings](https://github.com/hynek/argon2-cffi-bindings) | ||
| - **Supported versions:** | ||
|
|
||
| <details markdown="1" open> | ||
| <summary><strong>25.1.0 (latest)</strong></summary> | ||
|
|
||
| ```bash | ||
| pip install argon2-cffi-bindings --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple | ||
| ``` | ||
|
|
||
| - **License:** MIT/CC0(libargon2) | ||
| - **Download files:** [https://gitlab.com/riseproject/python/wheel_builder/-/packages/50575538](https://gitlab.com/riseproject/python/wheel_builder/-/packages/50575538) | ||
| </details> | ||
|
|
||
| <details markdown="1"> | ||
| <summary><strong>21.2.0</strong></summary> | ||
|
|
||
| ```bash | ||
| pip install argon2-cffi-bindings==21.2.0 --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple | ||
| ``` | ||
|
|
||
| - **License:** MIT/CC0(libargon2) | ||
| - **Download files:** [https://gitlab.com/riseproject/python/wheel_builder/-/packages/32318987](https://gitlab.com/riseproject/python/wheel_builder/-/packages/32318987) | ||
| </details> |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.