ENG-1633: Implement pure-Python wheel packer for local source trees#3817
Open
mitchell-as wants to merge 2 commits into
Open
ENG-1633: Implement pure-Python wheel packer for local source trees#3817mitchell-as wants to merge 2 commits into
mitchell-as wants to merge 2 commits into
Conversation
Add internal/python/wheel, a pure-Go routine that turns a local source
directory into a spec-compliant, byte-reproducible py3-none-any wheel. It only
relocates and zips files; it never executes the source it packs.
Pack reads core metadata from pyproject.toml's [project] table (caller values
override), mirrors the source tree into the wheel root (excluding the top-level
pyproject.toml, __pycache__, *.pyc/*.pyo, and version-control directories), and
emits {name}-{version}.dist-info with METADATA, WHEEL, and a PEP 376 RECORD.
Determinism comes from the stdlib archive/zip writer with fixed timestamps,
sorted entries, and fixed mode bits, so the same tree always produces the same
bytes. Compiled, platform-specific files (.so/.pyd/.dll/.dylib, matched
case-insensitively) are rejected rather than silently packed, and there is no
partial artifact on any failure path.
ENG-1633
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new Go package for building deterministic, pure-Python wheels from local source trees (for private-ingredient publishing), and vendors a TOML parser to read pyproject.toml metadata.
Changes:
- Introduce
internal/python/wheel.Packto create byte-reproduciblepy3-none-anywheels (including.dist-info+RECORD) without executing user code. - Read
[project]metadata frompyproject.tomlwith caller override precedence. - Vendor
github.com/BurntSushi/tomland promote it to a direct dependency.
Reviewed changes
Copilot reviewed 6 out of 22 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| vendor/modules.txt | Adds vendored package entries for BurntSushi/toml. |
| vendor/github.com/BurntSushi/toml/type_toml.go | Vendored TOML type definitions used by parser/encoder. |
| vendor/github.com/BurntSushi/toml/type_fields.go | Vendored reflection field-caching for struct decoding. |
| vendor/github.com/BurntSushi/toml/README.md | Vendored upstream documentation. |
| vendor/github.com/BurntSushi/toml/parse.go | Vendored TOML parser implementation. |
| vendor/github.com/BurntSushi/toml/meta.go | Vendored metadata/key tracking utilities. |
| vendor/github.com/BurntSushi/toml/lex.go | Vendored TOML lexer/state machine. |
| vendor/github.com/BurntSushi/toml/internal/tz.go | Vendored internal timezone helpers. |
| vendor/github.com/BurntSushi/toml/error.go | Vendored parse/lex error types and formatting. |
| vendor/github.com/BurntSushi/toml/encode.go | Vendored TOML encoder. |
| vendor/github.com/BurntSushi/toml/doc.go | Vendored package docs. |
| vendor/github.com/BurntSushi/toml/deprecated.go | Vendored deprecated aliases for compatibility. |
| vendor/github.com/BurntSushi/toml/decode.go | Vendored TOML decoder. |
| vendor/github.com/BurntSushi/toml/decode_go116.go | Vendored go1.16 build-tag support (DecodeFS). |
| vendor/github.com/BurntSushi/toml/COPYING | Vendored license. |
| vendor/github.com/BurntSushi/toml/.gitignore | Vendored ignore rules from upstream. |
| internal/python/wheel/wheel.go | Implements wheel packing, file collection, zip writing, hashing, and RECORD generation. |
| internal/python/wheel/wheel_test.go | Validates wheel layout, RECORD correctness, determinism, and rejection conditions. |
| internal/python/wheel/metadata.go | Reads/merges pyproject.toml metadata and normalizes wheel filename components. |
| internal/python/wheel/metadata_test.go | Tests metadata resolution and normalization helpers. |
| internal/python/wheel/distinfo.go | Builds METADATA, WHEEL, and RECORD contents. |
| go.mod | Promotes BurntSushi/toml from indirect to direct dependency. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
TestPackIsDeterministic ignored os.ReadFile errors, so two unreadable wheels would compare equal as nil and pass. Fail the test if either read errors. ENG-1633 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
b98ef85 to
771f733
Compare
Collaborator
Author
|
The Windows unit test failure is known and will be fixed in #3815 . The macOS test failures are sporadic and unrelated to this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ENG-1633: Implement pure-Python wheel packer for local source trees
The first piece of the private ingredient publish side (ENG-1563). It turns a customer's plain Python source folder into the standard wheel file, entirely on their own machine — no Python tooling, no subprocess, and it never runs the customer's code. The same input always produces a byte-identical wheel, so the publish hash is stable across re-runs.
internal/python/wheel.Packreads core metadata frompyproject.toml(caller-supplied values override), mirrors the source tree into the wheel root, and emits a spec-compliant{name}-{version}.dist-infowith METADATA, WHEEL, and a PEP 376 RECORD. Compiled, platform-specific files are refused rather than silently packed. ENG-1634 (state publish --build) consumes this.Note: a good chunk of this PR comes from vendoring the toml parser library, which moved from an indirect dependency to a direct one.
Deviations:
Packtakes an output directory and returns the canonical wheel path (rather than a literal output path), so the packer owns the{name}-{version}-py3-none-any.whlfilename. I also dropped the AC's "noos/execimport" test: it can't soundly enforce the no-execution property (the capability lives inos.StartProcessand ininternal/osutils, so the import check is leaky and gives false confidence), so that property is upheld by the design and code review instead.Base branch: targets
mitchell/eng-1635so this can be tested produce→consume before the consume side merges; GitHub will retarget it toversion/0-48-1-RC2once the upstream PRs land. The packer itself is independent of the other private-ingredient stories.Tested for spec-compliant dist-info, RECORD hash/size correctness, determinism (build twice with perturbed mtimes → identical bytes), native-content and empty-tree rejection, metadata override precedence, and name normalization. Also verified end to end by
pip install-ing a produced wheel into a fresh venv and importing it.🤖 Generated with Claude Code