Skip to content

Run the integration suite in CI again - #13

Merged
wayneeseguin merged 3 commits into
developfrom
wire-integration-suite
Jul 29, 2026
Merged

Run the integration suite in CI again#13
wayneeseguin merged 3 commits into
developfrom
wire-integration-suite

Conversation

@wayneeseguin

Copy link
Copy Markdown
Contributor

The 3400-line end-to-end suite in ci/scripts/tests has not run in CI, and
both CI entry points fail before reaching it. Three independent breakages,
one consequence.

What broke

The suite stopped being invoked. 72188a2 (2023) wired the pipeline to
make test with a suite-runner target:

test: $(if $(wildcard $(SAFE_PATH)),use,build)
	$(TEST_PATH) $(SAFE_PATH) ${VAULT_VERSIONS}

test was later modernized to go test -race. ci/scripts/test-release-against-vault
still passes TEST_PATH, SAFE_PATH, and VAULT_VERSIONS, but make accepts
unknown variables silently, so all three were ignored and only unit tests ran.

Every test-vault-* job bailed at startup. The script guarded on
VAULT_VERSION; all five jobs pass VAULT_VERSIONS. Under set -u the guard
fired every time with "VAULT_VERSION must be set … Did you misconfigure
Concourse?"

Every pull-request build failed. test-pr runs ci/scripts/test, deleted
in that same 2023 commit while the job kept referencing it.

What this does

  • adds test-integration, which passes TEST_PATH/SAFE_PATH/VAULT_VERSIONS
    through to the suite — kept separate from test so check stays hermetic
  • guards and reports VAULT_VERSIONS, and calls the target that runs the suite
  • restores ci/scripts/test, running make check only, since the PR job has
    no Vault server

test-integration rebuilds first when driving the in-tree binary. I hit the
alternative during development: a stale ./safe sitting in the tree produced
a red run that had nothing to do with the code under test. An explicitly
overridden SAFE_PATH is still used as given, since CI supplies the release
artifact it means to test.

Verification

  • make check green; make test-integration VAULT_VERSIONS=1.13.2
    1725 passing, 0 failing, exit 0
  • the vault job script was run against a simulated Concourse resource layout:
    it bails with VAULT_VERSIONS unset, and with it set invokes
    make test-integration TEST_PATH=../git-ci/ci/scripts/tests SAFE_PATH=../build/safe-1.2.3-darwin-arm64 VAULT_VERSIONS=…,
    confirmed argument-by-argument so a multi-version value stays one argument
    rather than becoming extra make goals
  • the PR script was run from a simulated GOPATH layout: it bails without
    MODULE and reaches a green make check with it

Not verified: none of this ran on Concourse — I have no access to the
pipeline. The scripts and the make target are verified locally against
simulated resource layouts, which covers the wiring but not Concourse's own
load_var, image, or resource behavior. The first real pipeline run is still
the proof.

The suite in ci/scripts/tests is driven by TEST_PATH, SAFE_PATH, and
VAULT_VERSIONS. The make target that once passed those through became a
plain `go test` invocation, so the variables were accepted and ignored
and the suite stopped running.

Add a separate target rather than folding it back into `test`, which
`check` depends on and which must stay hermetic. Rebuild first when
driving the in-tree binary so the suite cannot run against a stale
build; an overridden SAFE_PATH is used as given.
The script guarded on VAULT_VERSION while every test-vault job passes
VAULT_VERSIONS, so each job bailed before running anything. Guard and
report the name the jobs actually set.

Call the target that runs the suite. The previous target ignored the
TEST_PATH, SAFE_PATH, and VAULT_VERSIONS it was given and ran only the
unit tests.
The job invokes ci/scripts/test, which was removed when the suite was
renamed. Every pull-request build has failed on the missing file since.

Run the hermetic checks only. The job has no Vault server, so the
end-to-end suite stays in the test-vault jobs that do.
@wayneeseguin
wayneeseguin merged commit 82ceff1 into develop Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant