Skip to content

Add apply command to copy portable org state between instances#13

Open
andredlng wants to merge 4 commits into
mainfrom
apply-command
Open

Add apply command to copy portable org state between instances#13
andredlng wants to merge 4 commits into
mainfrom
apply-command

Conversation

@andredlng

@andredlng andredlng commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Depends on datamesh-manager/entropy-data#1521 — that app PR adds GET/PUT /api/organization/features, which this PR's organization-features resource requires. It must be merged and deployed before apply can copy organization features to a target instance. The rest of this PR works against the current API and is independent of #1521.

Summary

Adds a new apply command (plus supporting export/import dir work) that copies an organization's portable, declarative state from one Entropy Data instance to another — e.g. promoting a test environment's setup to production.

entropy-data export dir <path>  -c <source>                 # enumerate a source into a local YAML tree
entropy-data import dir <path>  -c <target> [--prune]       # upsert a tree into a target (zip form unchanged)
entropy-data apply --source <conn> --to <conn> [--prune] [--dry-run] [--keep <dir>]
  • apply exports the source into a staged directory and imports it into the target, so the transfer produces an auditable, re-runnable artifact. Supports --prune, --dry-run (per-resource create/update/prune counts), --include/--exclude, and --keep <dir> to retain the staged export. --source defaults to the global -c/--connection; --to is the required target.
  • export dir (new) enumerates a source into a local YAML tree, mirroring the app's organization-export layout so artifacts interchange.
  • import gains an import dir subcommand alongside import zip (the zip form now extracts to a temp dir and reuses the same engine). Both gain --prune (deletes target resources absent from the import, reverse dependency order, confirmation-gated unless --yes) and --include/--exclude.
  • Fix: import silently dropped sourcesystems/ from an org export — now imported in its correct dependency position (after policies, before assets).
  • Copies teams, tags, definitions, policies, sourcesystems, certifications, classification-schemes, assets, datacontracts, dataproducts, example-data, access, the semantics graph (namespaces + concepts/relationships nested per namespace), and the organization feature configuration (singleton).

One shared resources.py/sync.py engine drives export, import and apply so they cannot drift, with three resource shapes:

  • Flat/api/{path} list + /api/{path}/{id}; artifact <name>/<id>.yaml.
  • Nested (semantics) — a {parent} api_path template enumerated per parent; artifact <name>/<parent>/<id>.yaml; imported parent-first, pruned per parent.
  • Singleton (organization-features) — one object at /api/{path} (no id, no list); artifact <name>/<name>.yaml; applied last so a restrictive managedTagsPolicy it carries cannot reject earlier imports; never pruned.

All writes are idempotent PUT-by-id (or PUT-to-path for the singleton); the run is continue-on-error with a per-resource summary and non-zero exit on any failure.

Testing

ruff check/ruff format clean; pytest — 276 passed (25 in the export/import/apply suite, covering flat + nested semantics + singleton export/import/prune, upsert ordering, prune reverse-order + absent-only, apply orchestration, and the sourcesystems fix).

Notes / out of scope

  • dataproductbuilder is intentionally not wired yet — it needs an app-side schema migration to add a stable externalId before it can get an externalId-keyed API.

Add a new `apply --source <conn> --to <conn>` command that copies an
organization's portable, declarative state from one Entropy Data
instance to another by exporting the source into a staged directory and
importing it into the target.

- New `export dir <path>` enumerates a source into a local YAML tree
  (one directory per resource, one file per resource by id), mirroring
  the app's organization-export layout.
- `import` gains an `import dir <path>` subcommand; `import zip` now
  extracts and reuses the same dir-import engine. Both gain --prune
  (reverse-dependency-order delete of resources absent from the import,
  confirmation-gated unless --yes) and --include/--exclude; `import dir`
  and `apply` also support --dry-run.
- Fix `import` silently dropping sourcesystems from an org export.
- export/import/apply now also copy certifications, classification-schemes
  and example-data. Writes are idempotent PUT-by-id, team members are
  stripped on import, asset tag assignments are replayed, and audit
  fields are stripped before PUT. The experimental semantics API is not
  yet included.

Resource ordering, identity fields and per-resource handling live in one
shared module so export, import and apply cannot drift.
@andredlng andredlng self-assigned this Jul 14, 2026
Extend the export/import/apply engine to handle nested (parent-
parameterized) resources and include the semantics graph:
semantic-namespaces (flat), and semantic-concepts / semantic-
relationships nested per namespace.

A nested Resource carries a parent name and a {parent} api_path
template. Export lists the parent, then children per parent, writing
<name>/<namespace>/<id>.yaml; import upserts namespace-first; prune
enumerates parents and removes absent children per namespace (in
reverse dependency order). Flat behavior is unchanged.
Add singleton resource support to the copy engine and include the
organization feature configuration (GET/PUT /api/organization/features)
as an org-level singleton with no id and no collection.

Export reads the object and writes organization-features/
organization-features.yaml (skipping when unset); import PUTs it back;
it is applied last so a restrictive policy it carries cannot reject
tag/asset imports running earlier, and it is never pruned.

Requires the app-side endpoint (entropy-data#1521), which must be
merged and deployed to the target instance before this resource can be
applied.
Replace the per-concept/relationship nested handling with a single
per-namespace ontology document, using the app's new
GET/PUT /api/semantics/experimental/namespaces/{ns}/ontology.yaml
endpoint. The app imports the document in the correct internal order
(groups before members, concepts before relationships), so the client
no longer needs any concept-ordering logic.

Add a 'document' resource shape (one raw-YAML document per parent,
artifact <name>/<parent>.yaml, not prunable) and drop the now-unused
nested-collection machinery. The namespace row is still copied first as
a normal resource.

Requires the app-side ontology.yaml endpoint (entropy-data#1524).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant