Skip to content

ENG-1633: Implement pure-Python wheel packer for local source trees#3817

Open
mitchell-as wants to merge 2 commits into
version/0-48-1-RC2from
mitchell/eng-1633
Open

ENG-1633: Implement pure-Python wheel packer for local source trees#3817
mitchell-as wants to merge 2 commits into
version/0-48-1-RC2from
mitchell/eng-1633

Conversation

@mitchell-as

@mitchell-as mitchell-as commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

ENG-1633: Implement pure-Python wheel packer for local source trees

The first piece of the private ingredient publish side (ENG-1563). It turns a customer's plain Python source folder into the standard wheel file, entirely on their own machine — no Python tooling, no subprocess, and it never runs the customer's code. The same input always produces a byte-identical wheel, so the publish hash is stable across re-runs.

internal/python/wheel.Pack reads core metadata from pyproject.toml (caller-supplied values override), mirrors the source tree into the wheel root, and emits a spec-compliant {name}-{version}.dist-info with METADATA, WHEEL, and a PEP 376 RECORD. Compiled, platform-specific files are refused rather than silently packed. ENG-1634 (state publish --build) consumes this.

Note: a good chunk of this PR comes from vendoring the toml parser library, which moved from an indirect dependency to a direct one.

Deviations: Pack takes an output directory and returns the canonical wheel path (rather than a literal output path), so the packer owns the {name}-{version}-py3-none-any.whl filename. I also dropped the AC's "no os/exec import" test: it can't soundly enforce the no-execution property (the capability lives in os.StartProcess and in internal/osutils, so the import check is leaky and gives false confidence), so that property is upheld by the design and code review instead.

Base branch: targets mitchell/eng-1635 so this can be tested produce→consume before the consume side merges; GitHub will retarget it to version/0-48-1-RC2 once the upstream PRs land. The packer itself is independent of the other private-ingredient stories.

Tested for spec-compliant dist-info, RECORD hash/size correctness, determinism (build twice with perturbed mtimes → identical bytes), native-content and empty-tree rejection, metadata override precedence, and name normalization. Also verified end to end by pip install-ing a produced wheel into a fresh venv and importing it.

🤖 Generated with Claude Code

Add internal/python/wheel, a pure-Go routine that turns a local source
directory into a spec-compliant, byte-reproducible py3-none-any wheel. It only
relocates and zips files; it never executes the source it packs.

Pack reads core metadata from pyproject.toml's [project] table (caller values
override), mirrors the source tree into the wheel root (excluding the top-level
pyproject.toml, __pycache__, *.pyc/*.pyo, and version-control directories), and
emits {name}-{version}.dist-info with METADATA, WHEEL, and a PEP 376 RECORD.

Determinism comes from the stdlib archive/zip writer with fixed timestamps,
sorted entries, and fixed mode bits, so the same tree always produces the same
bytes. Compiled, platform-specific files (.so/.pyd/.dll/.dylib, matched
case-insensitively) are rejected rather than silently packed, and there is no
partial artifact on any failure path.

ENG-1633

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new Go package for building deterministic, pure-Python wheels from local source trees (for private-ingredient publishing), and vendors a TOML parser to read pyproject.toml metadata.

Changes:

  • Introduce internal/python/wheel.Pack to create byte-reproducible py3-none-any wheels (including .dist-info + RECORD) without executing user code.
  • Read [project] metadata from pyproject.toml with caller override precedence.
  • Vendor github.com/BurntSushi/toml and promote it to a direct dependency.

Reviewed changes

Copilot reviewed 6 out of 22 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
vendor/modules.txt Adds vendored package entries for BurntSushi/toml.
vendor/github.com/BurntSushi/toml/type_toml.go Vendored TOML type definitions used by parser/encoder.
vendor/github.com/BurntSushi/toml/type_fields.go Vendored reflection field-caching for struct decoding.
vendor/github.com/BurntSushi/toml/README.md Vendored upstream documentation.
vendor/github.com/BurntSushi/toml/parse.go Vendored TOML parser implementation.
vendor/github.com/BurntSushi/toml/meta.go Vendored metadata/key tracking utilities.
vendor/github.com/BurntSushi/toml/lex.go Vendored TOML lexer/state machine.
vendor/github.com/BurntSushi/toml/internal/tz.go Vendored internal timezone helpers.
vendor/github.com/BurntSushi/toml/error.go Vendored parse/lex error types and formatting.
vendor/github.com/BurntSushi/toml/encode.go Vendored TOML encoder.
vendor/github.com/BurntSushi/toml/doc.go Vendored package docs.
vendor/github.com/BurntSushi/toml/deprecated.go Vendored deprecated aliases for compatibility.
vendor/github.com/BurntSushi/toml/decode.go Vendored TOML decoder.
vendor/github.com/BurntSushi/toml/decode_go116.go Vendored go1.16 build-tag support (DecodeFS).
vendor/github.com/BurntSushi/toml/COPYING Vendored license.
vendor/github.com/BurntSushi/toml/.gitignore Vendored ignore rules from upstream.
internal/python/wheel/wheel.go Implements wheel packing, file collection, zip writing, hashing, and RECORD generation.
internal/python/wheel/wheel_test.go Validates wheel layout, RECORD correctness, determinism, and rejection conditions.
internal/python/wheel/metadata.go Reads/merges pyproject.toml metadata and normalizes wheel filename components.
internal/python/wheel/metadata_test.go Tests metadata resolution and normalization helpers.
internal/python/wheel/distinfo.go Builds METADATA, WHEEL, and RECORD contents.
go.mod Promotes BurntSushi/toml from indirect to direct dependency.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread internal/python/wheel/wheel_test.go Outdated
Comment thread internal/python/wheel/wheel.go
Comment thread internal/python/wheel/wheel.go
TestPackIsDeterministic ignored os.ReadFile errors, so two unreadable wheels
would compare equal as nil and pass. Fail the test if either read errors.

ENG-1633

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mitchell-as

Copy link
Copy Markdown
Collaborator Author

The Windows unit test failure is known and will be fixed in #3815 . The macOS test failures are sporadic and unrelated to this PR.

@mitchell-as mitchell-as requested a review from MDrakos June 23, 2026 21:09
Base automatically changed from mitchell/eng-1635 to version/0-48-1-RC2 June 23, 2026 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants