Add apply command to copy portable org state between instances

CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ## [Unreleased]
 
+- Add `entropy-data export dir <path>` to enumerate an organization's portable state into a local YAML tree (one directory per resource, one file per resource named by its id), mirroring the app's organization-export layout so the artifacts interchange.
+- Extend `entropy-data import` with an `import dir <path>` subcommand alongside `import zip` (the zip form now extracts to a temp directory and reuses the same dir-import logic). Both gain `--prune` to delete target resources absent from the import (in reverse dependency order, guarded by a confirmation prompt unless `--yes`), plus `--include`/`--exclude` filters; `import dir` also supports `--dry-run`.
+- Add `entropy-data apply --source <conn> --to <conn>` to copy portable organization state from one connection to another by exporting the source into a staged directory and importing it into the target. Supports `--prune`, `--dry-run` (per-resource create/update/prune counts), `--include`/`--exclude`, and `--keep <dir>` to retain the staged export.
+- Fix `import` silently dropping `sourcesystems/` from an organization export — source systems are now imported in their correct dependency position (after policies, before assets).
+- `export`/`import`/`apply` now also copy `certifications`, `classification-schemes`, `example-data`, and the `semantics` graph. All writes are idempotent PUT-by-id, team members are stripped on import, asset tag assignments are replayed, and audit fields are stripped before PUT.
+- The `semantics` graph is copied as one OSI-compliant ontology YAML document per namespace (`semantic-ontology/<namespace>.yaml`) via the app's `GET`/`PUT /api/semantics/experimental/namespaces/{ns}/ontology.yaml` endpoint, which imports it in the correct internal order (groups before members, concepts before relationships). Requires that endpoint on the target instance; the namespace row itself is copied first as a normal resource.
+- `export`/`import`/`apply` copy the organization's feature configuration as a singleton (`GET`/`PUT /api/organization/features`, artifact `organization-features/organization-features.yaml`), applied last so a restrictive policy it carries cannot reject earlier imports. Requires the app-side endpoint (entropy-data#1521); singletons are never pruned.
+
 ## [0.3.19]
 
 - Fix `entropy-data datacontracts yaml <id>` requesting `/api/datacontracts/{id}.yaml`, which does not exist on the server, resulting in a 404. It now requests the documented `/api/datacontracts/{id}/datacontract.yaml`.

README.md

@@ -111,9 +111,60 @@ entropy-data [--version] [--connection NAME] [--output table|json|yaml] [--debug
   search          query
   semantics       namespaces ... | concepts ... | relationships ... | search
   usage           list | submit | delete
-  import          zip
+  export          dir
+  import          zip | dir
+  apply           --to TARGET [--source SRC] [--prune] [--dry-run] [--include] [--exclude] [--keep DIR]
 ```
 
+## Copying organization state between instances
+
+`export`, `import` and `apply` move the portable declarative state of an organization
+(teams, tags, definitions, policies, source systems, certifications, classification
+schemes, assets, data contracts, data products, example data, access agreements, the
+semantics graph, and the organization feature configuration) between Entropy Data
+instances — for example to promote a test environment to prod.
+Only state reachable through the public `/api/**` API and portable across instances is
+copied (no secrets, telemetry, or environment-specific identity). Every write is an
+idempotent PUT-by-id, so runs converge and are safe to repeat.
+
+```bash
+# Export a source instance to a local YAML tree (one file per resource).
+entropy-data -c source export dir ./state
+
+# Import that tree into a target instance.
+entropy-data -c target import dir ./state
+
+# Copy directly from one connection to another (export -> stage -> import).
+entropy-data apply --source source --to target
+
+# Preview the changes without writing anything.
+entropy-data apply --source source --to target --dry-run
+
+# Mirror: also delete target resources that are absent from the source.
+entropy-data apply --source source --to target --prune
+```
+
+Useful options:
+
+- `--include a,b` / `--exclude a,b` — restrict the resource set (names as listed above).
+- `--prune` — after upserts, delete target resources absent from the source, in reverse
+  dependency order. Guarded by a confirmation prompt unless `--yes` is passed.
+- `--dry-run` — print per-resource create/update/(prune) counts; no writes.
+- `--keep DIR` (apply) — retain the staged export instead of using a temporary directory.
+
+Notes:
+
+- Team members are stripped on import (users are per-instance identities); the export
+  keeps them so the artifact is a faithful snapshot.
+- The semantics graph is copied as one OSI ontology YAML document per namespace
+  (`semantic-ontology/<namespace>.yaml`) via the app's `.../{ns}/ontology.yaml` endpoint,
+  which imports it in the correct internal dependency order. The namespace row is copied
+  first as a normal resource. Requires that endpoint on the target instance.
+- The organization feature configuration is an org-level singleton
+  (`organization-features/organization-features.yaml`), applied last and never pruned. It
+  requires the app's `GET`/`PUT /api/organization/features` endpoint (entropy-data#1521),
+  which must be merged and deployed to the target instance first.
+
 ## Connection Management
 
 Connections are stored in `~/.entropy-data/config.toml`:

src/entropy_data/cli.py

@@ -33,6 +33,16 @@ def get_client() -> EntropyDataClient:
     return EntropyDataClient(config)
 
 
+def client_for_connection(name: str) -> EntropyDataClient:
+    """Create an API client for a specific named connection (for `apply --to/--source`).
+
+    Resolves the connection by name only — the global --api-key/--host overrides
+    target the primary connection and must not bleed into a second endpoint.
+    """
+    config = resolve_connection(connection_name=name)
+    return EntropyDataClient(config)
+
+
 def get_output_format() -> OutputFormat:
     return _output_format
 
@@ -116,6 +126,7 @@ def main(
 # Register command groups
 from entropy_data.commands.access import access_app  # noqa: E402
 from entropy_data.commands.api_keys import api_keys_app  # noqa: E402
+from entropy_data.commands.apply import apply_command  # noqa: E402
 from entropy_data.commands.assets import assets_app  # noqa: E402
 from entropy_data.commands.certifications import certifications_app  # noqa: E402
 from entropy_data.commands.classifications import classifications_app  # noqa: E402
@@ -127,7 +138,7 @@ def main(
 from entropy_data.commands.definitions import definitions_app  # noqa: E402
 from entropy_data.commands.events import events_app  # noqa: E402
 from entropy_data.commands.example_data import example_data_app  # noqa: E402
-from entropy_data.commands.import_export import import_app  # noqa: E402
+from entropy_data.commands.import_export import export_app, import_app  # noqa: E402
 from entropy_data.commands.integrations import integrations_app  # noqa: E402
 from entropy_data.commands.lineage import lineage_app  # noqa: E402
 from entropy_data.commands.organization import organization_app  # noqa: E402
@@ -164,8 +175,14 @@ def main(
 app.add_typer(settings_app, name="settings", help="Manage organization settings.")
 app.add_typer(events_app, name="events", help="Poll events.")
 app.add_typer(lineage_app, name="lineage", help="Manage lineage (OpenLineage events).")
-app.add_typer(schemas_app, name="schemas", help="Get the JSON Schemas that data contracts (ODCS) and data products (ODPS) validate against.")
+app.add_typer(
+    schemas_app,
+    name="schemas",
+    help="Get the JSON Schemas that data contracts (ODCS) and data products (ODPS) validate against.",
+)
 app.add_typer(search_app, name="search", help="Search across resources.")
 app.add_typer(semantics_app, name="semantics", help="EXPERIMENTAL semantics API.")
 app.add_typer(usage_app, name="usage", help="Manage usage (OpenTelemetry traces).")
 app.add_typer(import_app, name="import", help="Import organization exports.")
+app.add_typer(export_app, name="export", help="Export organization state to a local YAML tree.")
+app.command(name="apply", help="Copy portable organization state from a source to a target connection.")(apply_command)

src/entropy_data/commands/apply.py

@@ -0,0 +1,86 @@
+"""`apply` — copy portable org state from a source instance to a target instance.
+
+Mechanism: export the source into a staged directory, then import that directory
+into the target. Both halves reuse the ``export dir`` / ``import dir`` engine, so
+the staged artifact is auditable and identical to a manual ``export`` + ``import``.
+"""
+
+import shutil
+import tempfile
+from pathlib import Path
+from typing import Annotated, Optional
+
+import typer
+
+from entropy_data.commands.import_export import _parse_csv, print_plan
+from entropy_data.output import console, error_console
+from entropy_data.resources import select_resources
+from entropy_data.sync import export_dir, import_dir, plan_import
+
+
+def apply_command(
+    to: Annotated[str, typer.Option("--to", help="Target named connection to apply state onto.")],
+    source: Annotated[
+        Optional[str],
+        typer.Option("--source", help="Source named connection (defaults to the global -c/--connection)."),
+    ] = None,
+    prune: Annotated[bool, typer.Option("--prune", help="Delete target resources absent from the source.")] = False,
+    yes: Annotated[bool, typer.Option("--yes", "-y", help="Skip the prune confirmation prompt.")] = False,
+    dry_run: Annotated[
+        bool, typer.Option("--dry-run", help="Print planned create/update/(prune) counts; write nothing to the target.")
+    ] = False,
+    keep: Annotated[
+        Optional[Path], typer.Option("--keep", help="Retain the staged export in this directory (else a temp dir).")
+    ] = None,
+    include: Annotated[Optional[str], typer.Option("--include", help="Comma-separated resources to include.")] = None,
+    exclude: Annotated[Optional[str], typer.Option("--exclude", help="Comma-separated resources to exclude.")] = None,
+) -> None:
+    """Copy portable organization state from a source connection to a target connection."""
+    from entropy_data.cli import client_for_connection, get_client, handle_error
+
+    try:
+        resources = select_resources(_parse_csv(include), _parse_csv(exclude))
+        source_client = client_for_connection(source) if source else get_client()
+        target_client = client_for_connection(to)
+    except Exception as e:
+        handle_error(e)
+        return
+
+    if keep is not None:
+        staged = keep
+        staged.mkdir(parents=True, exist_ok=True)
+        cleanup = False
+    else:
+        staged = Path(tempfile.mkdtemp(prefix="entropy-data-apply-"))
+        cleanup = True
+
+    try:
+        console.print(f"[bold]Exporting source into {staged}[/bold]")
+        export_result = export_dir(source_client, staged, resources)
+        console.print(f"\n[bold]Export:[/bold] {export_result.ok} exported, {export_result.fail} failed")
+
+        if dry_run:
+            print_plan(plan_import(target_client, staged, resources, prune=prune))
+            if export_result.fail > 0:
+                raise typer.Exit(1)
+            return
+
+        if prune and not yes:
+            console.print("[yellow]--prune will DELETE target resources absent from the source.[/yellow]")
+            if not typer.confirm("Proceed with prune?"):
+                console.print("Aborted.")
+                raise typer.Exit(1)
+
+        console.print("\n[bold]Applying to target[/bold]")
+        import_result = import_dir(target_client, staged, resources, prune=prune)
+        console.print(
+            f"\n[bold]Summary:[/bold] {import_result.ok} succeeded, {import_result.fail} failed "
+            f"({export_result.fail} export failures)"
+        )
+        if import_result.fail > 0 or export_result.fail > 0:
+            raise typer.Exit(1)
+    finally:
+        if cleanup:
+            shutil.rmtree(staged, ignore_errors=True)
+        elif not dry_run:
+            error_console.print(f"Staged export retained at {staged}")