Skip to content

Improve test environment for artifact tests #540

@MariusStorhaug

Description

Context

The artifact tests in tests/Artifacts.Tests.ps1 are used to verify that artifact retrieval and management functions work correctly. These tests run as part of the module's CI/CD pipeline in GitHub Actions.

Request

The current artifact tests depend on environment variables and real artifacts from the CI/CD pipeline:

$Owner = $env:GITHUB_REPOSITORY_OWNER
$Repository = $env:GITHUB_REPOSITORY_NAME
$WorkflowRunId = $env:GITHUB_RUN_ID
$ArtifactName = 'module'

This approach has several limitations:

  1. Dependency on real data — tests depend on artifacts actually existing from the current workflow run
  2. Brittle tests — if artifacts are deleted or workflow structure changes, tests may fail
  3. Limited test scenarios — can only test against whatever artifacts happen to exist
  4. CI/CD coupling — tests are tightly coupled to the pipeline state, making them difficult to run locally or in isolation

What is expected

A dedicated test environment for artifact tests that enables predictable, isolated testing with controlled test data.

Acceptance criteria

  • Artifact tests use predictable test data with known properties
  • Tests can cover edge cases and error scenarios (empty artifacts, large artifacts, missing artifacts)
  • Tests are not brittle due to external state changes
  • Test setup is documented

Technical decisions

Test data strategy: Use a dedicated test workflow or the BeforeAll.ps1 setup script to generate artifacts with predictable names and properties before the artifact tests run. This keeps the tests self-contained while still testing against real GitHub API behavior.

Integration vs unit tests: Keep tests as integration tests (calling real API endpoints) since the primary value is verifying the API interaction. Mock-based unit tests would not catch issues with artifact response parsing or pagination.

Alignment with #534: This issue aligns with the broader test consolidation effort in #534. The artifact tests should follow the same patterns established there (top-level BeforeAll, $env:GITHUB_RUN_ID as identifier, standard auth case structure).

Test approach: Update tests/Artifacts.Tests.ps1 to follow the standard test pattern from #534, using $env:GITHUB_REPOSITORY_* variables for the current repo (which is the unique pattern for artifact tests since artifacts are tied to the current workflow run).


Implementation plan

  • Update tests/Artifacts.Tests.ps1 to use the standard test pattern (top-level BeforeAll, auth cases, skip conditions)
  • Ensure the artifact test setup creates or identifies predictable test artifacts
  • Add tests for edge cases: non-existent artifact name, artifact from a specific workflow run
  • Document the test data setup and requirements in the test file or test README

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions