Add .tar.gz / .tgz archive support for extension, preset, and workflow installation

[Copilot]

The extension, preset, and workflow download pipelines only accepted ZIP archives, blocking use of npm registries and CI artifact stores that serve tarballs natively.

Core utilities (extensions.py)

• detect_archive_format(url, content_type="") — infers format from URL path extension (.zip, .tar.gz, .tgz) with Content-Type header fallback (application/gzip, application/x-gzip, application/x-tar+gzip)
• safe_extract_tarball(archive_path, dest_dir, error_class) — safe extraction with:
  • Pre-extraction validation of all members: rejects absolute paths, .. traversal, symlinks, hard links, devices, and FIFOs
  • PAX headers (XHDTYPE, XGLTYPE, SOLARIS_XHDTYPE) and GNU metadata-only entries (GNUTYPE_LONGNAME, GNUTYPE_LONGLINK) are silently skipped — they carry no extractable payload and are emitted by many common archiving tools; GNUTYPE_SPARSE is intentionally not skipped because sparse entries carry a real file payload and isreg() returns True for them
  • Python 3.11: passes pre-validated safe_members list to extractall()
  • Python 3.12+: uses tarfile.data_filter for additional OS-level protection
  • tarfile.TarError/OSError are caught and re-raised as the caller-supplied error_class for consistent error handling

Both helpers are public (no underscore prefix) and imported directly by presets.py and __init__.py.

Extensions & presets

• install_from_zip() on both managers now detects archive format from the file extension and dispatches to ZIP or tarball extraction accordingly — existing callers are unaffected
• download_extension() / download_pack() detect format from the download URL (or Content-Type fallback) and persist the archive with the correct extension (.zip or .tar.gz); unknown formats are rejected with a clear error rather than silently defaulting to ZIP
• Both functions capture response.geturl() as the canonical post-redirect URL, use it for Content-Type fallback format detection, and re-validate the final URL's scheme to guard against scheme-downgrade via redirects

__init__.py call sites

Path	Change
extension add --from	Detects format from URL/Content-Type before saving; unknown formats rejected; HTTPS re-checked on post-redirect URL
preset add --from	Same
extension update	Inline manifest peek handles both ZIP and tar.gz; cache filename sanitized via Path(extension).name to prevent path traversal
workflow add (URL)	Extracts workflow.yml from archive when URL points to one; temp-file paths initialized before write to avoid UnboundLocalError on disk-full
workflow add (local)	Accepts local .tar.gz/.tgz/.zip archive files (case-insensitive detection)
workflow add (catalog)	Same archive detection for catalog-sourced URLs

A shared _extract_workflow_yml(archive_path, fmt) helper handles root-level and single-nested-directory layouts for both formats, with tarfile.extractfile() handles properly closed via context managers.

Tests

30 new tests across test_extensions.py, test_presets.py, and test_workflows.py covering:

• test_extensions.py: format detection (URL + Content-Type), flat and nested tarball install, missing manifest errors, path traversal rejection, symlink rejection
• test_presets.py: flat and nested tarball install, missing manifest errors, path traversal rejection, symlink rejection
• test_workflows.py (TestWorkflowAddArchive, 9 CLI-level tests): local ZIP (flat/nested), local .tar.gz (flat/nested), .tgz alias, missing workflow.yml error cases, URL-based archive download for both ZIP and tar.gz formats

[Copilot]

Copilot's findings

• Files reviewed: 5/5 changed files
• Comments generated: 2

[mnriem]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied both changes in commit 0a02369:

• Public API (extensions.py): Renamed _detect_archive_format → detect_archive_format and _safe_extract_tarball → safe_extract_tarball (dropped the underscore prefix). All callers in extensions.py, presets.py, __init__.py, and tests/test_extensions.py have been updated accordingly.
• CLI-level tests (test_workflows.py): Added TestWorkflowAddArchive with 9 tests covering local ZIP (flat/nested), local .tar.gz (flat/nested), .tgz alias, missing workflow.yml error cases, and URL-based download for both ZIP and tar.gz formats.

[Copilot]

Copilot's findings

Comments suppressed due to low confidence (1)

src/specify_cli/init.py:3669

• archive_path.write_bytes(archive_data) can raise OSError/IOError (disk full, permission issues), but this block only handles URLError. That means specify extension add --from ... may crash with a traceback instead of emitting a controlled CLI error. Add except OSError (or broader I/O handling) around the write/install steps and surface a clear message before exiting.

                    suffix = ".tar.gz" if archive_fmt == "tar.gz" else ".zip"
                    safe_name = Path(extension).name or "extension"
                    archive_path = download_dir / f"{safe_name}-url-download{suffix}"
                    archive_path.write_bytes(archive_data)

• Files reviewed: 6/6 changed files
• Comments generated: 2

[Copilot]

Copilot's findings

• Files reviewed: 6/6 changed files
• Comments generated: 2

[mnriem]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied both changes in commit 05798a9:

• safe_extract_tarball() metadata members: PAX headers (XHDTYPE, XGLTYPE, SOLARIS_XHDTYPE) and GNU metadata entries (GNU_TYPES: longname/longlink/sparse) are now skipped instead of rejected. The _TAR_METADATA_TYPES tuple is defined once before the loop for efficiency. Symlinks, hard links, devices, and FIFOs are still rejected; only regular files and directories are extracted.
• Test imports: Replaced the __import__(...) inline pattern in all 9 TestWorkflowAddArchive tests with from unittest.mock import patch / patch.object(Path, "cwd", ...) — consistent with the rest of the file.

[Copilot]

Copilot's findings

Comments suppressed due to low confidence (2)

src/specify_cli/extensions.py:231

• In the Python 3.12+ branch, safe_extract_tarball() validates members into safe_members (and intentionally skips PAX/GNU metadata entries), but then calls tf.extractall(dest_dir, filter="data") without restricting extraction to safe_members. This makes extraction behavior inconsistent across Python versions and can reintroduce metadata entries (or other unexpected members) into the extraction step. Pass members=safe_members in the 3.12+ branch as well (and keep the filter="data" hardening), or extract only the validated members explicitly.

            # Extract — use the "data" filter on Python 3.12+ for extra hardening.
            # On older versions pass only the pre-validated members so that no
            # unvetted entry (added concurrently or via a race) slips through.
            if sys.version_info >= (3, 12):
                tf.extractall(dest_dir, filter="data")  # type: ignore[call-arg]
            else:
                tf.extractall(dest_dir, members=safe_members)  # noqa: S202 — validated above

src/specify_cli/init.py:3663

• extension add --from detects archive format from the original URL and optional Content-Type, but does not validate the final URL after redirects. An HTTPS URL can redirect to non-HTTPS, and format detection can be wrong if the redirect changes the effective filename/extension. Consider validating response.geturl() against the HTTPS/localhost policy and running detect_archive_format() against that final URL.

                    with urllib.request.urlopen(from_url, timeout=60) as response:
                        if not archive_fmt:
                            content_type = response.headers.get("Content-Type", "")
                            archive_fmt = detect_archive_format(from_url, content_type)
                        archive_data = response.read()

• Files reviewed: 6/6 changed files
• Comments generated: 4

[mnriem]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied all four changes in commit 1015ff2:

• _TAR_METADATA_TYPES: Replaced *tarfile.GNU_TYPES with explicit GNUTYPE_LONGNAME and GNUTYPE_LONGLINK only. GNUTYPE_SPARSE is intentionally excluded — sparse entries carry a real file payload and member.isreg() returns True for them (they're included in REGULAR_TYPES), so they pass the existing file-type check and are extracted correctly.
• download_extension() / download_pack(): response.geturl() is now captured as final_url inside the with block. The Content-Type fallback uses final_url (not the original URL), and after reading the response the final URL's scheme is re-validated to catch scheme-downgrade via redirects.
• preset add --from / extension add --from (__init__.py): Same redirect-safe pattern applied — final_url = response.geturl() captured inside the with block, used for format detection, with a post-redirect HTTPS scheme check.
• Three mock responses in test_extensions.py and test_presets.py were updated with mock_response.geturl.return_value = <download_url> to satisfy the new geturl() call.