Skip to content

Add hamilton.packaging for reusable, validatable dataflow subgraphs#1665

Open
btwitsshivanshu wants to merge 1 commit into
apache:mainfrom
btwitsshivanshu:reusable_package
Open

Add hamilton.packaging for reusable, validatable dataflow subgraphs#1665
btwitsshivanshu wants to merge 1 commit into
apache:mainfrom
btwitsshivanshu:reusable_package

Conversation

@btwitsshivanshu

Copy link
Copy Markdown

Adds a new hamilton.packaging module that lets a set of nodes from an existing
dataflow be bundled into a portable, versioned unit, exported as a JSON manifest,
and re-validated against a different host graph. Today there's no first-class way
to share a subgraph across projects: users copy modules and discover name, type,
and config mismatches only at runtime. This surfaces them as structured,
categorized errors before the driver is built.

Changes

NodePackage declares provides and requires as name-to-type mappings, along
with inter-node dependencies and config_keys. NodePackage.from_graph(...)
derives all of that from a live Driver.graph, typing provides by node output and
classifying each required input as a config key, an internal edge, or an external
requirement; include_upstream=True pulls in transitive non-input dependencies.
dependency_order() returns provided names dependencies-first, breaking ties
alphabetically.

PackageManifest is the serialization layer over the six MANIFEST_FIELDS, with
to_dict / to_json / from_dict / from_json round-tripping, sorted output, and
types rendered through hamilton.htypes.get_type_as_string. export_package and
import_manifest handle file I/O.

validate_package(package, graph, minimum_version=1) returns a ValidationReport
of ValidationProblems spanning VALIDATION_CATEGORIES: version_too_low,
node_collision, dependency_type_mismatch, missing_dependency, and
config_requirement_missing. Type checks reuse Hamilton's existing subclass rule,
so widening passes (bool feeding an int requirement) but narrowing does not.

find_conflicts(first, second) performs order-independent pairwise detection over
CONFLICT_CATEGORIES (duplicate_node, requirement_mismatch,
requirement_disagreement), and compose_packages(packages, name) checks all pairs
before merging provides, dependencies, and config keys, internalizing any requirement
satisfied within the set. Conflicts raise PackageConflictError carrying all of them.
diff_manifests(old, new) reports added, removed, and changed nodes and requirements
plus config-key deltas, for reviewing what a version bump actually changed.

Errors hang off a PackagingError base: PackageValidationError carries .report,
PackageConflictError carries .conflicts, and both stringify every underlying
message rather than just the first.

On the driver side, Builder.with_packages(*packages) registers packages, both
build() and async_driver's build_without_init() validate them against the built
graph, Driver.list_packages() and Driver.validate_packages() expose them, and
Builder.copy() carries them forward.

How I tested this

New unit tests live in tests/test_pkg.py. Constructor coverage exercises empty
and non-string names, non-integer and boolean versions, string values where types are
expected, a name appearing in both provides and requires, bare-string config_keys,
and single-pass iterables for both config_keys and dependencies. from_graph is
tested against real driver graphs: it rejects HamiltonGraph and plain dicts, rejects
unknown, input-node, empty, and non-string selections, accepts generators, correctly
classifies config keys versus internal edges versus external requirements, ignores
optional parameters, and handles include_upstream across shared ancestors and deep
chains without recursing.

Manifest tests cover sort stability, get_type_as_string rendering, dict and JSON round
trips, unknown and missing fields, malformed JSON, semantically invalid dependencies on
import, hand-written manifests, and cycle detection on both packages and manifests.
Validation tests hit every category in VALIDATION_CATEGORIES and pin the direction of
the subclass rule, along with generic aliases, typing.Any on either side,
config-satisfied requirements, version boundary equality, and aggregate multi-problem
reports.

Conflict and composition tests assert every CONFLICT_CATEGORIES entry in both argument
orders, merge semantics, internal satisfaction honoring subtypes, version defaulting to
the highest member, all-pairs checking even when a requirement is internally satisfied,
and identical serialized output across every input permutation. Error tests check the
hierarchy, the .report and .conflicts payloads, and that every underlying message
appears in both str() and repr(). Driver integration tests cover with_packages
varargs and accumulation, rejection of non-packages, validation on both build paths,
copy() isolation, and requirements satisfied across multiple modules.

Notes

Fully additive — existing driver behavior is unchanged when no packages are registered.
Type compatibility deliberately delegates to Hamilton's existing subclass rule rather
than introducing a second notion of type equality. Manifests are pinned by
MANIFEST_FIELDS, so adding a field later will need a compat path in from_dict.

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant