Add apply command to copy portable org state between instances#13
Open
andredlng wants to merge 4 commits into
Open
Add apply command to copy portable org state between instances#13andredlng wants to merge 4 commits into
andredlng wants to merge 4 commits into
Conversation
Add a new `apply --source <conn> --to <conn>` command that copies an organization's portable, declarative state from one Entropy Data instance to another by exporting the source into a staged directory and importing it into the target. - New `export dir <path>` enumerates a source into a local YAML tree (one directory per resource, one file per resource by id), mirroring the app's organization-export layout. - `import` gains an `import dir <path>` subcommand; `import zip` now extracts and reuses the same dir-import engine. Both gain --prune (reverse-dependency-order delete of resources absent from the import, confirmation-gated unless --yes) and --include/--exclude; `import dir` and `apply` also support --dry-run. - Fix `import` silently dropping sourcesystems from an org export. - export/import/apply now also copy certifications, classification-schemes and example-data. Writes are idempotent PUT-by-id, team members are stripped on import, asset tag assignments are replayed, and audit fields are stripped before PUT. The experimental semantics API is not yet included. Resource ordering, identity fields and per-resource handling live in one shared module so export, import and apply cannot drift.
Extend the export/import/apply engine to handle nested (parent-
parameterized) resources and include the semantics graph:
semantic-namespaces (flat), and semantic-concepts / semantic-
relationships nested per namespace.
A nested Resource carries a parent name and a {parent} api_path
template. Export lists the parent, then children per parent, writing
<name>/<namespace>/<id>.yaml; import upserts namespace-first; prune
enumerates parents and removes absent children per namespace (in
reverse dependency order). Flat behavior is unchanged.
Add singleton resource support to the copy engine and include the organization feature configuration (GET/PUT /api/organization/features) as an org-level singleton with no id and no collection. Export reads the object and writes organization-features/ organization-features.yaml (skipping when unset); import PUTs it back; it is applied last so a restrictive policy it carries cannot reject tag/asset imports running earlier, and it is never pruned. Requires the app-side endpoint (entropy-data#1521), which must be merged and deployed to the target instance before this resource can be applied.
Replace the per-concept/relationship nested handling with a single
per-namespace ontology document, using the app's new
GET/PUT /api/semantics/experimental/namespaces/{ns}/ontology.yaml
endpoint. The app imports the document in the correct internal order
(groups before members, concepts before relationships), so the client
no longer needs any concept-ordering logic.
Add a 'document' resource shape (one raw-YAML document per parent,
artifact <name>/<parent>.yaml, not prunable) and drop the now-unused
nested-collection machinery. The namespace row is still copied first as
a normal resource.
Requires the app-side ontology.yaml endpoint (entropy-data#1524).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
applycommand (plus supportingexport/import dirwork) that copies an organization's portable, declarative state from one Entropy Data instance to another — e.g. promoting a test environment's setup to production.applyexports the source into a staged directory and imports it into the target, so the transfer produces an auditable, re-runnable artifact. Supports--prune,--dry-run(per-resource create/update/prune counts),--include/--exclude, and--keep <dir>to retain the staged export.--sourcedefaults to the global-c/--connection;--tois the required target.export dir(new) enumerates a source into a local YAML tree, mirroring the app's organization-export layout so artifacts interchange.importgains animport dirsubcommand alongsideimport zip(the zip form now extracts to a temp dir and reuses the same engine). Both gain--prune(deletes target resources absent from the import, reverse dependency order, confirmation-gated unless--yes) and--include/--exclude.importsilently droppedsourcesystems/from an org export — now imported in its correct dependency position (after policies, before assets).One shared
resources.py/sync.pyengine drivesexport,importandapplyso they cannot drift, with three resource shapes:/api/{path}list +/api/{path}/{id}; artifact<name>/<id>.yaml.{parent}api_pathtemplate enumerated per parent; artifact<name>/<parent>/<id>.yaml; imported parent-first, pruned per parent./api/{path}(no id, no list); artifact<name>/<name>.yaml; applied last so a restrictivemanagedTagsPolicyit carries cannot reject earlier imports; never pruned.All writes are idempotent PUT-by-id (or PUT-to-path for the singleton); the run is continue-on-error with a per-resource summary and non-zero exit on any failure.
Testing
ruff check/ruff formatclean;pytest— 276 passed (25 in the export/import/apply suite, covering flat + nested semantics + singleton export/import/prune, upsert ordering, prune reverse-order + absent-only, apply orchestration, and the sourcesystems fix).Notes / out of scope
externalIdbefore it can get an externalId-keyed API.