Add workflow step catalog — community-installable step types

[Copilot]

The workflow engine shipped with a dynamic STEP_REGISTRY but no distribution mechanism for community-authored step types. This adds a full catalog system for discovering, installing, and managing custom step types, following the same patterns as the workflow/extension catalogs.

New classes (workflows/catalog.py)

• StepRegistry — persists installed custom steps in .specify/workflows/steps/step-registry.json; validates JSON shape on load and falls back to a clean default on corruption or unreadable files
• StepCatalog — multi-source catalog stack with SHA256-based caching; resolves SPECKIT_STEP_CATALOG_URL env var → .specify/step-catalogs.yml → ~/.specify/step-catalogs.yml → built-in defaults (step-catalog.json + step-catalog.community.json)
• StepCatalogError / StepValidationError / StepCatalogEntry supporting types

Dynamic step loading (workflows/__init__.py)

load_custom_steps(project_root) scans .specify/workflows/steps/, dynamically imports each package's __init__.py, finds the StepBase subclass matching the declared type_key, and registers it into STEP_REGISTRY. Broken packages are silently skipped. Dynamic imports are intentionally deferred to workflow execution time; CLI metadata commands (list, info) read from StepRegistry and step.yml only and never execute installed step code.

CLI surface (specify workflow step …)

specify workflow step list              # built-in + installed custom types
specify workflow step add <id>          # fetch step.yml + __init__.py, validate, install
specify workflow step remove <id>       # delete package dir + registry entry
specify workflow step search [query]    # search across all configured catalogs
specify workflow step info <id>         # detail view (built-in or custom)
specify workflow step catalog list      # show active catalog sources
specify workflow step catalog add <url> # add catalog to .specify/step-catalogs.yml
specify workflow step catalog remove <n># remove by index

add validates that the downloaded step.yml's type_key matches the catalog ID and that all fetches use HTTPS before writing anything to disk. Downloads happen in a temporary directory and are moved to the final location atomically only after all validation passes — a transient failure never corrupts or deletes a pre-existing step directory.

Catalog files

• workflows/step-catalog.json — official catalog (empty, ready for entries)
• workflows/step-catalog.community.json — community catalog (empty, ready for entries)

Robustness & security hardening

• Path traversal guard: step_id is validated against the resolved base directory in both add and remove before any filesystem operation.
• Collision guard: step add rejects IDs that collide with built-in or already-installed step types.
• Atomic install: step add downloads and validates in a temp directory, then moves to the final location only on success; pre-existing directories are never deleted on failure.
• URL validation: _validate_catalog_url() checks parsed.hostname (not just parsed.netloc) to correctly reject hostless URLs such as https://:80/path.
• Cache resilience: catalog cache writes are wrapped in try/except OSError so a read-only filesystem never aborts a successful HTTP fetch.
• Registry resilience: StepRegistry._load() validates the on-disk JSON shape and catches OSError/UnicodeError, falling back to a clean default on any read failure.
• Empty YAML tolerance: StepCatalog.add_catalog() treats an empty .specify/step-catalogs.yml as an empty mapping rather than a corruption error.

Tests

30+ tests across TestStepRegistryCustom, TestStepCatalog, and TestLoadCustomSteps covering CRUD, catalog resolution (env var / project / user / default), URL validation, search, dynamic loading edge cases, registry shape validation, unreadable-file fallback, and empty YAML initialization. The chmod-based unreadability test is skipped on Windows.

[Copilot]

Pull request overview

Adds a catalog/registry system for community-installable workflow step types, plus CLI commands to discover, install, and manage them alongside built-in steps.

Changes:

• Introduces StepRegistry and StepCatalog (multi-source resolution + SHA256 cache) for step type distribution/management.
• Adds dynamic filesystem-based loading of installed custom step packages into STEP_REGISTRY.
• Expands CLI with specify workflow step … and adds tests and initial (empty) catalog JSON files.

Show a summary per file

File	Description
workflows/step-catalog.json	Adds the built-in “official” step catalog scaffold (currently empty).
workflows/step-catalog.community.json	Adds the built-in “community” step catalog scaffold (currently empty).
src/specify_cli/workflows/catalog.py	Implements StepRegistry + StepCatalog with config resolution and caching.
src/specify_cli/workflows/init.py	Adds load_custom_steps(project_root) dynamic import/registration for installed step packages.
src/specify_cli/init.py	Adds Typer subcommands for listing/searching/installing/removing steps and managing step catalogs.
tests/test_workflows.py	Adds unit tests covering registry CRUD, catalog resolution/validation, search/info, and dynamic loading behavior.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

src/specify_cli/init.py:5555

• workflow_step_remove() builds step_dir from unvalidated step_id and then shutil.rmtree(step_dir). A malicious value like ../... could delete arbitrary directories outside .specify/workflows/steps. Add the same resolved-path relative_to() guard used by workflow_remove/workflow_add before performing deletions.

    step_dir = project_root / ".specify" / "workflows" / "steps" / step_id
    if step_dir.exists():
        import shutil
        shutil.rmtree(step_dir)

• Files reviewed: 6/6 changed files
• Comments generated: 4

[Copilot]

Copilot's findings

Comments suppressed due to low confidence (1)

src/specify_cli/workflows/catalog.py:1021

• StepCatalog.add_catalog()/remove_catalog() read YAML without handling yaml.YAMLError/OSError; those exceptions will bubble past the CLI handlers (which only catch StepValidationError) and produce a stack trace. Wrap YAML reads in try/except and re-raise as StepValidationError so the CLI consistently reports a user-facing error.

    def remove_catalog(self, index: int) -> str:
        """Remove a catalog source by index (0-based). Returns the removed name."""
        config_path = self.project_root / ".specify" / "step-catalogs.yml"
        if not config_path.exists():
            raise StepValidationError("No step catalog config file found.")

        data = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
        if not isinstance(data, dict):
            raise StepValidationError(
                "Catalog config file is corrupted (expected a mapping)."
            )

• Files reviewed: 6/6 changed files
• Comments generated: 2

[mnriem]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied all changes from the review thread in commits e311afb and df061e0:

• StepRegistry._load() — now validates the loaded JSON is a dict with a dict steps field (resets malformed steps to {}), and catches OSError/UnicodeError so unreadable files fall back to the default structure instead of crashing CLI commands.
• StepCatalog.add_catalog() — changed yaml.safe_load(...) to yaml.safe_load(...) or {} so an empty .specify/step-catalogs.yml is treated as an empty mapping and can be initialized by the command.

Three new tests cover these cases: malformed steps key, unreadable registry file (with functional-after-fallback assertion), and empty YAML config file.

[Copilot]

Copilot's findings

Comments suppressed due to low confidence (2)

src/specify_cli/workflows/init.py:113

• The module name passed to spec_from_file_location() is derived from type_key. If type_key contains characters like . it will create package-like names in sys.modules and can cause confusing import behavior/collisions. Safer approach: derive the module name from a hash of the path or type_key (and/or sanitize to [a-zA-Z0-9_]+).

            spec = _importlib_util.spec_from_file_location(
                f"_speckit_custom_step_{type_key}", init_py
            )

src/specify_cli/init.py:5664

• workflow step info also calls load_custom_steps() to populate STEP_REGISTRY, which executes installed step code as a side effect of a read-only command. Consider avoiding imports here (use StepRegistry + step.yml metadata), or require an explicit opt-in before executing third-party step code.

    # Load custom steps so registry is up to date
    from .workflows import load_custom_steps
    load_custom_steps(project_root)

• Files reviewed: 6/6 changed files
• Comments generated: 6

[mnriem]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied all changes from the review thread in commits 3e30520 and 1e5f5fd:

• No dynamic imports in list/info: Removed load_custom_steps() from workflow_step_list() and workflow_step_info(). Both commands now read only from StepRegistry (disk metadata) — no arbitrary step code is executed.
• Atomic install in step add: Downloads and validation now happen in a tempfile.mkdtemp() temp dir. The temp dir is moved to step_dir only after all validation passes. Pre-existing step_dir is removed before the move (to avoid shutil.move creating a nested subdirectory). finally ensures temp dir cleanup on any failure.
• _validate_catalog_url(): Changed parsed.netloc → parsed.hostname to correctly reject hostless URLs like https://:80/path.
• Cache writes: Wrapped cache_dir.mkdir() + file writes in try/except OSError in both WorkflowCatalog and StepCatalog so a read-only filesystem doesn't abort a successful HTTP fetch.
• test_registry_unreadable_file_resets: Added @pytest.mark.skipif(sys.platform == "win32", ...) to skip the chmod(0o000) test on Windows.