Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ native riscv64 hardware, with the goal of maintaining a riscv64-specific package
repository for projects where upstream are not yet ready or able to perform
builds themselves.

[python-wheels Documentation](https://riseproject-dev.github.io/python-wheels/)

This work is a continuation of the earlier
[wheel_builder](https://gitlab.com/riseproject/python/wheel_builder) project,
which hosts binary wheels for a variety of Python modules.
Expand Down
21 changes: 21 additions & 0 deletions docs/_config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
title: RISE Python Wheels
description: riscv64 binary wheels for popular Python projects
remote_theme: just-the-docs/just-the-docs

url: https://riseproject-dev.github.io
baseurl: /python-wheels

plugins:
- jekyll-remote-theme

color_scheme: light
search_enabled: true
heading_anchors: true

aux_links:
"GitHub":
- "https://github.com/riseproject-dev/python-wheels"

exclude:
- packages/generate_packages_doc.py
- packages/*.yaml
29 changes: 29 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: Home
layout: default
nav_order: 1
---

# RISE Python Wheels

RISE Python Wheels is a public project enabling RISC-V support for the Python
ecosystem. It uses the RISE [RISC-V
Runners](https://riscv-runners.riseproject.dev/) project to build wheels on
native riscv64 hardware, with the goal of maintaining a riscv64-specific
package repository for projects where upstream are not yet ready or able to
perform builds themselves.

This work continues the earlier
[wheel_builder](https://gitlab.com/riseproject/python/wheel_builder) project,
which hosts binary wheels for a variety of Python modules.

For more information about RISE, visit the [project
website](https://riseproject.dev/).

## Sections

- [Motivation](motivation.html)
- [Installing the Packages](install.html)
- [Manylinux_2_35 and Manylinux_2_39](manylinux.html)
- [Supported Packages](packages/)
- [Infrastructure](infrastructure.html)
67 changes: 67 additions & 0 deletions docs/infrastructure.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
title: Infrastructure
layout: default
nav_order: 6
---

# Infrastructure

The [python-wheels](https://github.com/riseproject-dev/python-wheels) project
makes use of some supplemental infrastructure available both on GitHub, and
externally. The following sections outline these services and how to
configure/maintain them.

## RISE RISC-V Runners

For RISE-hosted builds of Python wheels and upstream submissions, the
`python-wheels` repository uses the [RISC-V
Runners](https://riscv-runners.riseproject.dev/) project. It is already enabled
for the repository; to reconfigure, see the [installation
guide](https://riscv-runners.riseproject.dev/docs/getting-started/install).

## GitLab

The `python-wheels` project is a continuation of
[wheel_builder](https://gitlab.com/riseproject/python/wheel_builder), which is
hosted on GitLab. This project leverages the GitLab CI/CD framework to create
pipelines for building, testing, and deploying wheels, along with automatic
documentation updates and periodic checks for new package versions upstream.

While development and maintenance now primarily occur on GitHub, the existing
GitLab infrastructure (particularly the package registry) must be maintained.
Comment thread
threexc marked this conversation as resolved.
The original wheel_builder
[documentation](https://riseproject.gitlab.io/python/wheel_builder/index.html)
is maintained for reference purposes, and provides a detailed explanation of how
the project was originally structured.

### GitLab for Open Source

We apply for the [GitLab for Open
Source](https://about.gitlab.com/solutions/open-source/join/) program. This
provides us with the GitLab Ultimate tier at no cost. Membership requires annual
renewal.

### CI/CD Pipeline Tokens and Access

The GitLab CI/CD pipelines make use of project tokens, generated at different
project scopes. Two tokens are defined:

1. `PYTHON_CI_GROUP_TOKEN` - Defined at the "Group"
([Python](https://gitlab.com/riseproject/python)) level. Used for triggering
pipelines for merge requests and most automatic updates and checks. The value
of this token is inherited by `wheel_builder` as `PYTHON_CI_GROUP_TOKEN`
under `Settings -> CI/CD Settings -> Variables` (see the [CI/CD variables
panel](https://gitlab.com/riseproject/python/wheel_builder/-/settings/ci_cd#js-cicd-variables-settings)).
2. `rise-github-package-registry` - Defined at the "Repository"
([wheel_builder](https://gitlab.com/riseproject/python/wheel_builder/-/settings/repository#js-deploy-tokens))
level. Used for pushing wheels to the GitLab package registry from GitHub.

These tokens are generated with a 1-year expiry timeline, so they need to be
regenerated by a project administrator/owner.

#### Token Scope Reference

| Token | Level | Role | Scopes | Used For | Expiry |
|-------|-------|------|--------|----------|--------|
| `PYTHON_CI_GROUP_TOKEN` | Group ([Python](https://gitlab.com/riseproject/python)) | Maintainer | `api`, `read_api`, `read_repository`, `write_repository` | MR pipeline triggers, auto updates/checks | 1 year (Unknown expiry) |
| `rise-github-package-registry` | Repository deploy token ([wheel_builder](https://gitlab.com/riseproject/python/wheel_builder/-/settings/repository#js-deploy-tokens)) | n/a (deploy token) | `read_package_registry`, `write_package_registry` | Push wheels GitHub → GitLab package registry | 1 year (June 29th, 2027) |
51 changes: 51 additions & 0 deletions docs/install.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
title: Installing the Packages
layout: default
nav_order: 3
---

# Installing the Packages

The built wheels are hosted in the [package
registry](https://gitlab.com/riseproject/python/wheel_builder/-/packages)
associated with the riseproject/python/wheel_builder project. To install them,
first upgrade pip to the latest version, e.g.,

```bash
python -m pip install --upgrade pip
```

and then pass the `--index-url` option to the install command to tell pip to
pull packages from the registry associated with this project, e.g.,

```bash
python -m pip install scipy --index-url https://gitlab.com/api/v4/projects/riseproject%2Fpython%2Fwheel_builder/packages/pypi/simple
```

{: .note }
> Some riscv64 packages are now built and published upstream on PyPI. We will
> no longer build, upgrade and publish them as part of the wheel_builder
> project.
>
> To make sure to always install the latest version available, use
>
> ```bash
> python -m pip install scipy --prefer-binary --extra-index-url https://gitlab.com/api/v4/projects/riseproject%2Fpython%2Fwheel_builder/packages/pypi/simple
> ```
>
> This will:
>
> - search both PyPI and the internal registry.
> - pick the highest available version.
> - prefer binary wheels over source distributions.
>
> This ensures we get wheels from PyPI when available, while falling back to
> our registry for packages without riscv64 wheels, avoiding unnecessary source
> builds.

{: .warning }
> In general, `--extra-index-url` should be used very carefully (see
> <https://peps.python.org/pep-0708/#motivation>).
>
> The wheel_builder registry only contains wheels for packages that have
> pre-existing counterparts in PyPI, so it is safe to use it in that context.
43 changes: 43 additions & 0 deletions docs/manylinux.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
---
title: Manylinux_2_35 and Manylinux_2_39
layout: default
nav_order: 4
---

# Manylinux_2_35 and Manylinux_2_39

The packages distributed in this registry are a mix of manylinux_2_35 and
manylinux_2_39 wheels. Support for manylinux wheels for riscv64 was added to
pip in version 24.1 so you must have pip 24.1 or greater installed locally in
order to install the packages. This is why we recommend upgrading pip above
before installing anything. Prior to the release of pip 24.1, some wheels with
the linux tag were created for a select set of packages and uploaded to this
package registry. If your version of pip is older than 24.1 pip will install
these older wheels instead of the manylinux wheels, if they are available.
This is probably not what you want so do upgrade pip.

These older packages in this registry with the 'linux' platform tag, e.g.,
`numpy-1.26.4-cp39-cp39-linux_riscv64.whl`, actually behave like manylinux_2_35
wheels. These wheels were generated because we didn't want to wait for the
release of pip 24.1 to begin distributing riscv64 wheels, so while we were
waiting, we used a modified auditwheel that generates wheels with the linux
platform tag that mostly behave like manylinux_2_35 wheels. This means that
the wheels will work on any Linux distribution with glibc 2.35 or greater. The
wheels also vendor all of their dependencies that are not on the
manylinux_2_35 whitelist, e.g., OpenBLAS and libgfortran. One downside of
misusing the platform tag in this way is that pip will not warn you when you
install the wheels on distributions that use a glibc older than 2.35. In this
case the wheels will install but will be unlikely to work. As most riscv64
users are expected to use recent distributions to benefit from the latest
riscv64 support from the kernel and toolchains, this hopefully won't be too
much of an issue.

The manylinux_2_39 wheels are built with the upstream manylinux_2_39_riscv64
image. There are a few exceptions that were built with a custom
manylinux_2_39_riscv64 based on RockyLinux 10 which we used for a short period
of time before the official manylinux_2_39_riscv64 images were published.

All earlier wheels are built using a custom manylinux_2_35 based on Ubuntu
22.04. See the local
[riseproject/python/manylinux](https://gitlab.com/riseproject/python/manylinux)
fork for more details.
30 changes: 30 additions & 0 deletions docs/motivation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
title: Motivation
layout: default
nav_order: 2
---

# Motivation

Python is commonly used in the fields of scientific computing, data analysis and
machine learning. However, the Python packages used in these disciplines aren't
wholly written in Python - they also contain a lot of code written in C/C++ or
other languages which needs to be built as part of the module. Such projects
typically create binary wheels for each of their releases and upload these
wheels to pypi, the Python Package Index. This allows users to easily and
quickly install tested, prebuilt versions of their favourite projects using the
pip (or [uv](https://docs.astral.sh/uv/)) tool.

Until recently, Python packaging infrastructure like
[auditwheel](https://github.com/pypa/auditwheel),
[cibuildwheel](https://github.com/pypa/cibuildwheel), and
[manylinux](https://github.com/pypa/manylinux) did not support riscv64, and no
native runners for GitHub Actions were available. This made supporting the
architecture difficult for open-source projects without complicated build
processes and emulated systems. However, the aforementioned infrastructure now
supports riscv64, and with the RISE [RISC-V
Runners](https://riscv-runners.riseproject.dev/) project, maintainers have the
option of building binary wheels on native riscv64 platforms. The RISE Python
Wheels project's goal is to accelerate this adoption and ensure that the riscv64
architecture is fully-supported for data science and machine learning
applications.
59 changes: 59 additions & 0 deletions docs/packages/aiohttp.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
title: aiohttp
layout: default
parent: Supported Packages
---

<!-- Auto-generated by generate_packages_doc.py. Do not edit manually. -->

# aiohttp

{: .warning }
> This package is deprecated. PyPI now publishes newer versions of
> this package for riscv64, and we will no longer maintain this
> package. Please use the version from [PyPI](https://pypi.org/project/aiohttp/) instead.
>
> If you need a specific version between the latest available here
> and the first available on PyPI, please open an
> [Issue](https://gitlab.com/riseproject/python/wheel_builder/-/issues).

- **Source Code:** [https://github.com/aio-libs/aiohttp](https://github.com/aio-libs/aiohttp)
- **Supported versions:**

<details markdown="1" open>
<summary><strong>3.12.14 (latest)</strong></summary>

```bash
pip install aiohttp --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple
```

- **License:** Apache-2.0
- **Download files:** [https://gitlab.com/riseproject/python/wheel_builder/-/packages/42690263](https://gitlab.com/riseproject/python/wheel_builder/-/packages/42690263)
- **Patch applied for this version:** [https://gitlab.com/riseproject/python/wheel_builder/-/tree/main/wheel_builder/aiohttp/patches/v3.12.14](https://gitlab.com/riseproject/python/wheel_builder/-/tree/main/wheel_builder/aiohttp/patches/v3.12.14)
</details>

<details markdown="1">
<summary><strong>3.11.18</strong></summary>

```bash
pip install aiohttp==3.11.18 --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple
```

- **License:** Apache-2.0
- **Download files:** [https://gitlab.com/riseproject/python/wheel_builder/-/packages/40999682](https://gitlab.com/riseproject/python/wheel_builder/-/packages/40999682)
- **Patch applied for this version:** [https://gitlab.com/riseproject/python/wheel_builder/-/tree/main/wheel_builder/aiohttp/patches/v3.11.18](https://gitlab.com/riseproject/python/wheel_builder/-/tree/main/wheel_builder/aiohttp/patches/v3.11.18)
</details>


{: .warning }
> We disable the tests in tests/autobahn/test_autobahn as they depend on
> docker_on_whales and we don't want to be testing docker builds as part of
> the pipeline.
>
> We also disable a flaky uvloop test, and tests that contain the following
> patterns, involving consistent failures related to networking or permissions:
>
> - test_no_warnings
> - test_static_directory_without_read_permission
> - test_static_file_without_read_permission
> - test_uvloop_secure_https_proxy
34 changes: 34 additions & 0 deletions docs/packages/argon2-cffi-bindings.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
title: argon2-cffi-bindings
layout: default
parent: Supported Packages
---

<!-- Auto-generated by generate_packages_doc.py. Do not edit manually. -->

# argon2-cffi-bindings

- **Source Code:** [https://github.com/hynek/argon2-cffi-bindings](https://github.com/hynek/argon2-cffi-bindings)
- **Supported versions:**

<details markdown="1" open>
<summary><strong>25.1.0 (latest)</strong></summary>

```bash
pip install argon2-cffi-bindings --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple
```

- **License:** MIT/CC0(libargon2)
- **Download files:** [https://gitlab.com/riseproject/python/wheel_builder/-/packages/50575538](https://gitlab.com/riseproject/python/wheel_builder/-/packages/50575538)
</details>

<details markdown="1">
<summary><strong>21.2.0</strong></summary>

```bash
pip install argon2-cffi-bindings==21.2.0 --index-url https://gitlab.com/api/v4/projects/56254198/packages/pypi/simple
```

- **License:** MIT/CC0(libargon2)
- **Download files:** [https://gitlab.com/riseproject/python/wheel_builder/-/packages/32318987](https://gitlab.com/riseproject/python/wheel_builder/-/packages/32318987)
</details>
Loading
Loading