docs: user guide for the substrait.dataframe API (Zensical)

.github/workflows/docs-deploy.yml

@@ -0,0 +1,74 @@
+name: Deploy documentation
+
+# Publishes versioned docs to the `gh-pages` branch using the Zensical-compatible
+# fork of `mike` (https://github.com/squidfunk/mike): each version is committed
+# as a subdirectory of `site_url` (see zensical.toml), with a `latest` alias and
+# root redirect pointing at the newest release. Runs on release tags (v*.*.*),
+# and can be triggered manually.
+
+on:
+  push:
+    tags: [ 'v[0-9]+.[0-9]+.[0-9]+' ]
+  workflow_dispatch:
+    inputs:
+      version:
+        description: "Version label to publish (e.g. 0.29 or dev)"
+        required: true
+        default: "dev"
+      alias:
+        description: "Alias to update"
+        required: false
+        default: "latest"
+      set_default:
+        description: "Point the site root at this alias"
+        type: boolean
+        default: true
+
+permissions:
+  contents: write   # mike pushes the built docs to the gh-pages branch
+
+jobs:
+  deploy:
+    name: Deploy docs with mike
+    runs-on: ubuntu-latest
+    # Auto-deploy only on the canonical repo; manual runs are allowed anywhere
+    # (e.g. to bootstrap gh-pages on a fork while testing).
+    if: ${{ github.event_name == 'workflow_dispatch' || github.repository == 'substrait-io/substrait-python' }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0   # mike reads/writes the full gh-pages history
+      - name: Install uv with python
+        uses: astral-sh/setup-uv@v7
+        with:
+          python-version: "3.12"
+      - name: Install docs deps + mike
+        run: |
+          uv sync --frozen --group docs
+          uv pip install "git+https://github.com/squidfunk/mike.git"
+      - name: Configure git identity
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+      - name: Resolve version and alias
+        id: v
+        run: |
+          if [ "${{ github.event_name }}" = "push" ]; then
+            full="${GITHUB_REF_NAME#v}"      # 0.29.0
+            echo "version=${full%.*}" >> "$GITHUB_OUTPUT"   # 0.29 (minor series)
+            echo "alias=latest" >> "$GITHUB_OUTPUT"
+            echo "set_default=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "version=${{ inputs.version }}" >> "$GITHUB_OUTPUT"
+            echo "alias=${{ inputs.alias }}" >> "$GITHUB_OUTPUT"
+            echo "set_default=${{ inputs.set_default }}" >> "$GITHUB_OUTPUT"
+          fi
+      - name: Deploy version
+        run: |
+          uv run mike deploy --push --update-aliases \
+            "${{ steps.v.outputs.version }}" "${{ steps.v.outputs.alias }}"
+      - name: Set default version
+        if: ${{ steps.v.outputs.set_default == 'true' }}
+        run: |
+          uv run mike set-default --push "${{ steps.v.outputs.alias }}"

.github/workflows/docs.yml

@@ -0,0 +1,29 @@
+name: Build documentation
+
+on:
+  pull_request:
+  push:
+    branches: [ main ]
+
+permissions:
+  contents: read
+
+# Build-check only: this verifies the docs compile on every PR so broken pages
+# or a broken mkdocstrings reference are caught early. It does NOT publish.
+# Versioned publishing to GitHub Pages is handled separately by
+# .github/workflows/docs-deploy.yml (mike -> gh-pages branch).
+
+jobs:
+  build:
+    name: Build docs
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v6
+      - name: Install uv with python
+        uses: astral-sh/setup-uv@v7
+        with:
+          python-version: "3.12"
+      - name: Build the documentation site
+        run: |
+          uv run --frozen --group docs zensical build

CONTRIBUTING.md

@@ -7,9 +7,10 @@ cd substrait-python
 ```
 
 ## Development environment
-Activate environment with uv.
+Set up the environment with uv. This installs the default `dev` dependency
+group, which includes everything needed to run the tests.
 ```
-uv sync --extra test
+uv sync
 ```
 
 # Lint & Format
@@ -26,3 +27,19 @@ Run tests in the project's root dir.
 ```
 uv run pytest
 ```
+
+# Documentation
+The user guide lives in `docs/` and is built with [Zensical](https://zensical.org);
+the API reference under `docs/reference/` is generated from the package
+docstrings via mkdocstrings. Configuration is in `zensical.toml`.
+
+Preview it locally with live reload, or build the static site into `./site`:
+```
+pixi run docs-serve   # or: uv run --group docs zensical serve
+pixi run docs-build   # or: uv run --group docs zensical build
+```
+
+When editing docstrings that appear in the reference, prefer Markdown (fenced
+code blocks and backticked names) over reStructuredText so they render cleanly.
+Every pull request runs a docs build-check; versioned docs are published to
+GitHub Pages on release (see `.github/workflows/docs-deploy.yml`).

README.md

@@ -26,7 +26,43 @@ This project is not an execution engine for Substrait Plans.
 ## Status
 This is an experimental package that is still under development.
 
-# Example
+# Building plans with the DataFrame API
+
+The `substrait.dataframe` module is an ergonomic, fluent API for authoring
+Substrait plans — a Polars/PySpark-style DataFrame with operator-overloaded
+expressions, on top of the lower-level builders. It is the recommended way to
+build plans by hand:
+
+```python
+import substrait.dataframe as sub
+
+plan = (
+    sub.read_named_table("people", {"id": sub.i64, "age": sub.i64, "name": sub.string})
+    .filter(sub.col("age") > 25)
+    .with_columns(adult=sub.col("age") >= 18)
+    .select("id", "name", "adult")
+    .to_plan()
+)
+```
+
+`plan` is a `substrait.proto.Plan` ready to hand to a consumer such as DuckDB or
+DataFusion. Install the `extensions` extra so function overloads resolve against
+the standard Substrait extensions:
+
+```sh
+pip install "substrait[extensions]"
+```
+
+**📖 See the [documentation guide](docs/index.md)** for the full walkthrough —
+data sources, types, expressions, joins, aggregations, window functions,
+subqueries, DDL/writes, and custom extensions. Build it locally with
+`pixi run docs-serve` (or `uv run zensical serve`).
+
+# Example (low-level API)
+
+The examples below construct plans with the raw `substrait.proto` and
+`substrait.builders` layers. For most hand-authored plans, prefer the
+[DataFrame API](#building-plans-with-the-dataframe-api) above.
 
 ## Produce a Substrait Plan
 The ``substrait.proto`` module provides access to the classes

docs/aggregations.md

@@ -0,0 +1,76 @@
+# Aggregations
+
+Summarize rows with `group_by(...).agg(...)`. Aggregate functions come from the
+[`f` namespace](functions.md); name each measure with `.alias(...)`.
+
+## Group by and aggregate
+
+```python
+import substrait.dataframe as sub
+
+df.group_by("region").agg(
+    sub.f.sum(sub.col("amount")).alias("total"),
+    sub.f.count(sub.col("amount")).alias("n"),
+)
+```
+
+Group by multiple keys by passing several, and group the whole frame (a grand
+total) by passing none:
+
+```python
+df.group_by("region", "product").agg(sub.f.sum(sub.col("amount")).alias("total"))
+df.group_by().agg(sub.f.count(sub.col("id")).alias("rows"))
+```
+
+`group_by` keys may be column names or expressions.
+
+## One-shot form
+
+`aggregate` does the same thing in a single call — the grouping keys first, then
+the measures:
+
+```python
+df.aggregate("region", sub.f.sum(sub.col("amount")).alias("total"))
+df.aggregate(["region", "product"], sub.f.count(sub.col("id")).alias("n"))
+```
+
+## Modifying a measure
+
+Aggregate measures support several modifiers, which chain:
+
+```python
+sub.f.count(sub.col("customer")).distinct().alias("unique_customers")
+sub.f.sum(sub.col("amount")).filter(sub.col("status") == "paid").alias("paid_total")
+sub.f.string_agg(sub.col("name"), sub.lit(", ")).order_by("name").alias("names")
+```
+
+- **`.distinct()`** — operate on distinct inputs (`COUNT(DISTINCT x)`).
+- **`.filter(predicate)`** — restrict this aggregate to matching rows
+  (SQL `agg(x) FILTER (WHERE ...)`). Returns a `Measure`, only meaningful inside
+  `agg(...)`.
+- **`.order_by(*keys, descending=, nulls_last=)`** — order the inputs to an
+  order-sensitive aggregate.
+- **`.alias(name)`** — name the output column.
+
+## Grouping sets, rollup, and cube
+
+For multiple grouping levels in one aggregation, pass explicit `grouping_sets`
+(lists of key names or positions into the keys), or use the `rollup` / `cube`
+shortcuts:
+
+```python
+# explicit grouping sets: by (region, product), by (region), and the grand total
+df.group_by("region", "product", grouping_sets=[["region", "product"], ["region"], []]) \
+  .agg(sub.f.sum(sub.col("amount")).alias("total"))
+
+# ROLLUP: (region, product), (region), ()
+df.rollup("region", "product").agg(sub.f.sum(sub.col("amount")).alias("total"))
+
+# CUBE: every subset of the keys
+df.cube("region", "product").agg(sub.f.sum(sub.col("amount")).alias("total"))
+```
+
+## Next
+
+- [Window functions](window-functions.md) — aggregate without collapsing rows.
+- [The function namespace](functions.md) — the full set of aggregate functions.

docs/consuming-plans.md

@@ -0,0 +1,103 @@
+# Consuming plans
+
+substrait-python **produces** plans; it does not execute them. Once you have a
+`proto.Plan`, hand it to a Substrait consumer to run.
+
+## Materialize and serialize
+
+`.to_plan()` returns a `substrait.proto.Plan`; `SerializeToString()` turns it
+into the bytes a consumer accepts:
+
+```python
+import substrait.dataframe as sub
+
+plan = (
+    sub.read_named_table("customer", {"c_name": sub.string, "c_nationkey": sub.i32})
+    .filter(sub.col("c_nationkey") == 3)
+    .select("c_name")
+    .to_plan()
+)
+
+payload = plan.SerializeToString()
+```
+
+`DataFrame.to_substrait(registry=...)` is an alias kept for parity with the
+[Narwhals wrapper](#relationship-to-narwhals); it materializes against a given
+registry.
+
+## Inspecting a plan
+
+The bundled pretty printer renders the plan as a compact tree — far more
+readable than the raw protobuf text:
+
+```python
+from substrait.utils.display import pretty_print_plan
+
+pretty_print_plan(plan, use_colors=True)
+```
+
+## Handing off to an engine
+
+The [`examples/`](https://github.com/substrait-io/substrait-python/tree/main/examples)
+directory has runnable end-to-end scripts. In short, most engines take the
+serialized bytes:
+
+=== "DuckDB"
+
+    ```python
+    import duckdb
+
+    duckdb.install_extension("substrait")
+    duckdb.load_extension("substrait")
+    duckdb.sql("CALL from_substrait(?)", params=[plan.SerializeToString()])
+    ```
+
+    See [`examples/duckdb_example.py`](https://github.com/substrait-io/substrait-python/blob/main/examples/duckdb_example.py).
+
+=== "ADBC"
+
+    ```python
+    import adbc_driver_duckdb.dbapi
+
+    with adbc_driver_duckdb.dbapi.connect(":memory:") as conn, conn.cursor() as cur:
+        cur.executescript("INSTALL substrait;")
+        cur.executescript("LOAD substrait;")
+        cur.execute(plan.SerializeToString())
+        print(cur.fetch_arrow_table())
+    ```
+
+    See [`examples/adbc_example.py`](https://github.com/substrait-io/substrait-python/blob/main/examples/adbc_example.py).
+
+!!! tip "Matching a real table's schema"
+    The examples read a table's Arrow schema from the engine and convert it via
+    `pyarrow.substrait.serialize_schema(...).to_pysubstrait().base_schema`, then
+    feed that `NamedStruct` straight into `read_named_table`. That keeps the plan
+    schema exactly aligned with what the engine will resolve.
+
+## Round-tripping
+
+A serialized plan loads back with the generated protobuf class — useful for
+tests and for consuming plans other producers emit:
+
+```python
+from substrait.proto import Plan
+
+restored = Plan()
+restored.ParseFromString(payload)
+```
+
+## Relationship to Narwhals
+
+`substrait.dataframe` is the **native** fluent frame you call directly.
+`substrait.narwhals` is a separate **integration layer**: it implements the
+[Narwhals](https://narwhals-project.org) backend protocol and lets
+backend-agnostic Narwhals code compile down to a plan, delegating to the native
+frame underneath. Reach for it when you want to drive plan construction through
+Narwhals (`nw.from_native(...)`); reach for `substrait.dataframe` when you want
+to build plans directly. See
+[`examples/narwhals_example.py`](https://github.com/substrait-io/substrait-python/blob/main/examples/narwhals_example.py).
+
+## Next
+
+- [Getting started](getting-started.md) — back to the basics.
+- [API reference](reference/index.md) — the generated symbol reference.