ci: reclaim OpenStack orphans at integration test start#808
Draft
cbartz wants to merge 9 commits into
Draft
Conversation
Add a script that deletes dangling servers, keypairs, images and test security groups left by integration tests, plus a 6-hourly workflow on the private-endpoint runner.
Drop the scheduled cleanup workflow. Force-cancelled runs leave resources behind; the next integration suite now deletes CI-named OpenStack leftovers older than 6h before creating new ones.
Public cleanup entry point first; private helpers below. Module-level imports in conftest instead of inline.
cbartz
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a start-of-integration-run “orphan reaper” for OpenStack resources created by CI, so resources left behind by force-cancelled or failed runs are reclaimed automatically the next time the integration suites connect to the tenant.
Changes:
- Added OpenStack orphan cleanup helpers that delete CI-named resources older than 6 hours (servers, images, keypairs, and test security groups).
- Hooked orphan cleanup into both the charm integration suite and the github-runner-manager integration suite at OpenStack connection time.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tests/integration/helpers/orphan_cleanup.py | New charm-suite OpenStack orphan cleanup helper, including server/image/keypair/SG deletion logic. |
| tests/integration/conftest.py | Invokes orphan cleanup at the start of the charm integration suite OpenStack connection fixture. |
| github-runner-manager/tests/integration/orphan_cleanup.py | New manager-suite OpenStack orphan cleanup helper (servers/keypairs/SGs). |
| github-runner-manager/tests/integration/conftest.py | Invokes orphan cleanup at the start of the manager integration suite OpenStack connection fixture. |
Keypairs often lack created_at; treating unknown age as stale risked deleting in-progress CI resources. Skip those instead.
Wait for server delete before SGs, keep OpenStack teardown inside the connection context, best-effort start-up reaping, and tighten test-id regexes to the 8-char CI ids we generate.
Align OpenStack teardown/orphan helpers with existing integration suite call conventions.
Keep suite call style consistent; openstacksdk accepts the name positionally as used elsewhere in this repository.
Drop wait= on image delete (not supported), materialize list_servers, and log orphan cleanup failures with exc_info.
--use-existing-app-suffix is for local runs that keep long-lived resources; do not reap them at suite start.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
At OpenStack integration-test connect, delete CI-named leftover resources older than 6h (servers, keypairs, images, test SGs). Covers charm and github-runner-manager suites. Force-cancelled prior runs are reclaimed the next time a suite starts.
Why we need it
Force-cancel / fixture failure leaves VMs and keypairs in the private-endpoint tenant. Start-of-run orphan cleanup reclaims them without a separate scheduled workflow.
Checklist
docs/changelog.mdwith user-relevant changesterraform fmtpasses andtflintreports no errorsgithub-runner-manager/pyproject.toml.