Run the integration suite in CI again - #13
Merged
Conversation
The suite in ci/scripts/tests is driven by TEST_PATH, SAFE_PATH, and VAULT_VERSIONS. The make target that once passed those through became a plain `go test` invocation, so the variables were accepted and ignored and the suite stopped running. Add a separate target rather than folding it back into `test`, which `check` depends on and which must stay hermetic. Rebuild first when driving the in-tree binary so the suite cannot run against a stale build; an overridden SAFE_PATH is used as given.
The script guarded on VAULT_VERSION while every test-vault job passes VAULT_VERSIONS, so each job bailed before running anything. Guard and report the name the jobs actually set. Call the target that runs the suite. The previous target ignored the TEST_PATH, SAFE_PATH, and VAULT_VERSIONS it was given and ran only the unit tests.
The job invokes ci/scripts/test, which was removed when the suite was renamed. Every pull-request build has failed on the missing file since. Run the hermetic checks only. The job has no Vault server, so the end-to-end suite stays in the test-vault jobs that do.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 3400-line end-to-end suite in
ci/scripts/testshas not run in CI, andboth CI entry points fail before reaching it. Three independent breakages,
one consequence.
What broke
The suite stopped being invoked.
72188a2(2023) wired the pipeline tomake testwith a suite-runner target:testwas later modernized togo test -race.ci/scripts/test-release-against-vaultstill passes
TEST_PATH,SAFE_PATH, andVAULT_VERSIONS, but make acceptsunknown variables silently, so all three were ignored and only unit tests ran.
Every
test-vault-*job bailed at startup. The script guarded onVAULT_VERSION; all five jobs passVAULT_VERSIONS. Underset -uthe guardfired every time with "VAULT_VERSION must be set … Did you misconfigure
Concourse?"
Every pull-request build failed.
test-prrunsci/scripts/test, deletedin that same 2023 commit while the job kept referencing it.
What this does
test-integration, which passesTEST_PATH/SAFE_PATH/VAULT_VERSIONSthrough to the suite — kept separate from
testsocheckstays hermeticVAULT_VERSIONS, and calls the target that runs the suiteci/scripts/test, runningmake checkonly, since the PR job hasno Vault server
test-integrationrebuilds first when driving the in-tree binary. I hit thealternative during development: a stale
./safesitting in the tree produceda red run that had nothing to do with the code under test. An explicitly
overridden
SAFE_PATHis still used as given, since CI supplies the releaseartifact it means to test.
Verification
make checkgreen;make test-integration VAULT_VERSIONS=1.13.2→1725 passing, 0 failing, exit 0
it bails with
VAULT_VERSIONSunset, and with it set invokesmake test-integration TEST_PATH=../git-ci/ci/scripts/tests SAFE_PATH=../build/safe-1.2.3-darwin-arm64 VAULT_VERSIONS=…,confirmed argument-by-argument so a multi-version value stays one argument
rather than becoming extra make goals
GOPATHlayout: it bails withoutMODULEand reaches a greenmake checkwith itNot verified: none of this ran on Concourse — I have no access to the
pipeline. The scripts and the make target are verified locally against
simulated resource layouts, which covers the wiring but not Concourse's own
load_var, image, or resource behavior. The first real pipeline run is stillthe proof.