test(garm): refactor unit tests to the scenario framework#284
Merged
Conversation
The charm-behaviour tests faked `self` with a MagicMock and called unbound methods (`GarmCharm.restart(charm)`), which could only assert on the arguments the charm passed to its collaborators, never on what it actually did. Drive the real charm through Scenario instead and assert on the resulting State: the Juju secrets created, the files pushed, the Pebble plan left behind, and the status reported. Only GARM's HTTP surface is stubbed. ops-scenario 8.8.0 expands the go-framework extension itself, so the charm needs no packing for its container, peer relation and config options to resolve. Move the pure config-rendering and secret-generation helper tests to test_garm_toml.py, leaving test_charm.py purely Scenario-driven. charm.py statement coverage rises from 67% to 89%: Scenario exercises the relation-parsing and config-writing paths the mock-based tests never reached.
12 tasks
yhaliaw
reviewed
Jul 15, 2026
The seven helper tests moved out of test_charm.py predate the AAA docstring convention and carried single-line summaries across unchanged. State the why in the assert line where the requirement isn't obvious from the code, rather than restating the assertion.
The convention was only written down in STYLE.md, which is scoped to Go and spells it as comments, so Python tests had nothing to point at and the nearest neighbour is an unreliable guide: planner-operator and garm follow it, garm-configurator and webhook-gateway-operator do not. Also drop the paas_charm fixture details from the garm AGENTS.md. They described framework internals at a pinned version (the derived go_secret_key databag key, its literal status strings), which the AGENTS.md checker cannot validate and a dependency bump would silently invalidate. The test file already carries them as comments at the point of use; keep the pointer and the guidance that outlives the version.
main added the get-credentials action (#254) with two tests in the mock-self style this branch removes. Port them to Scenario rather than keep either side: run the action through the context and assert on the results and the ActionFailed, so they cover the action's real path to the secret instead of a MagicMock's return value.
cbartz
commented
Jul 15, 2026
florentianayuwono
approved these changes
Jul 15, 2026
florentianayuwono
left a comment
Collaborator
There was a problem hiding this comment.
thanks chris!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Refactors the GARM charm's unit tests in
test_charm.pyto the Scenario framework, aligningwith
garm-configurator. Each test now drives the real charm against aStateand asserts onthe resulting
State— secrets created, files pushed, Pebble plan, reported status — with onlyGARM's HTTP surface stubbed.
The pure config-rendering and secret-generation helpers move to
test_garm_toml.py, leavingtest_charm.pypurely Scenario-driven.src/charm.pystatement coverage rises from 67% to 90%.Also adds the arrange/act/assert docstring convention to the root
AGENTS.md. It was onlywritten down in
STYLE.md, which is scoped to Go and spells it as// Arrange:comments, soPython tests had nothing to point at — and it is followed unevenly (
planner-operatorandgarmdo,garm-configuratorandwebhook-gateway-operatordo not), which makes imitatingthe nearest neighbour an unreliable guide.
Why we need it
The charm-behaviour tests faked
selfwith aMagicMockand called unbound methods(
GarmCharm.restart(charm)). That could only assert on the arguments the charm passed to itscollaborators — never on what it actually did — and needed scaffolding like a sentinel
exception to stop execution before
super().restart(). Scenario removes the scaffolding andcovers the relation-parsing and config-writing paths the mock-based tests never reached.
ops-scenario8.8.0 expands thego-frameworkextension itself, so no packing is needed forthe charm's container, peer relation and config options to resolve.
Test plan
tox -c tox.toml -e fmt,lint,unit— 247 unit tests pass. Coverage measured withtox -c tox.toml -e unit,coverage-reporton this branch and onorigin/main(67% → 90% onsrc/charm.py).Integration tests (
tox -e garm-integration) have not been run — they need a live Jujumodel. This PR changes no production code, so the risk is contained to CI.
Test inventory: where each of main's 52 tests went
The diff is large and the test counts are misleading —
test_charm.pygoes 52 → 33 whilecovering more, because 18 tests moved out to
test_garm_toml.py, five groups merged, one wasdropped, and six are new. This table is the map. It is a one-time review aid, not a permanent
artifact.
Moved verbatim to
test_garm_toml.py(18) — names unchanged, so this part is mechanicallycheckable rather than a matter of trust:
build_provider_list_password_in_clouds_yaml,build_provider_list_returns_default_when_empty,generate_admin_password_meets_garm_policy,generate_admin_password_produces_unique_values,generate_garm_secrets_produces_unique_values,generate_garm_secrets_returns_jwt_and_passphrase,proxy_environment_edge_cases,proxy_environment_happy_path,render_clouds_yaml_quotes_special_chars,render_garm_toml_default_provider_applies_proxy_var_names,render_garm_toml_no_proxy_var_names_omits_key,render_garm_toml_passphrase_in_database_section,render_garm_toml_postgresql_backend,render_garm_toml_provider_section,render_garm_toml_section_fields,render_garm_toml_sslmode_propagated,render_garm_toml_with_configurator_providers,render_garm_toml_with_proxy_var_names.Seven of these carried single-line summaries rather than AAA docstrings — a gap that predates
this PR on
main. Since moving a test is the moment to fix it, they now have AAA docstrings;both test files are at 33/33 and 18/18.
Converted to Scenario (34 → 27, one dropped):
test_charm.pyensure_secrets_creates_secrets_on_first_runleader_creates_garm_secretsensure_secrets_skips_when_not_leadernon_leader_creates_no_secretsensure_secrets_skips_creation_when_secrets_existexisting_secrets_are_not_regeneratedrestart_ensures_secrets_before_readiness_gatesecrets_are_created_before_the_workload_is_readymaybe_first_run_skips_when_already_initializedfirst_run_skipped_when_garm_is_already_initialisedmaybe_first_run_skips_when_not_leaderfirst_run_skipped_when_not_leadermaybe_first_run_raises_on_connection_errorfirst_run_connection_error_errors_out_for_retryget_credentials_action_returns_credentials_when_availableget_credentials_action_returns_the_stored_credentialsget_credentials_action_fails_when_credentials_unavailableget_credentials_action_fails_before_the_credentials_existreconcile_calls_restartevery_observed_event_reconcilesreconcile_runners_reconciles_credentials_entities_then_scalesetsreconcilers_run_in_dependency_orderreconcile_runners_charmed_template_error_sets_waiting_statuscharmed_template_error_degrades_to_waitingreconcile_runners_success_refreshes_stale_app_statussuccessful_reconcile_refreshes_stale_app_statusbuild_desired_credentials_builds_credential_from_relationgithub_credential_is_built_from_relation_databuild_desired_credentials_dedupes_per_app_installationunits_sharing_a_github_app_yield_one_credentialrestart_proxy_vars_appear_in_pebble_layerproxy_vars_reach_the_workloadrestart_proxy_value_change_forces_layer_rewriteproxy_value_change_rewrites_the_workload_layerrestart_clearing_proxy_removes_it_from_layerclearing_the_proxy_removes_it_from_the_workload_layerrestart_unchanged_config_skips_replanunchanged_config_does_not_restart_the_workloadrestart_without_configurator_relation_still_prunes_orphaned_scalesetsmissing_configurator_relation_prunes_orphaned_scalesetsrestart_configurator_relation_present_but_unpopulated_does_not_pruneunpopulated_configurator_relation_does_not_prunerestart_missing_configurator_relation_refreshes_stale_app_statusmissing_configurator_relation_refreshes_stale_app_statusget_admin_credentials_returns_content_when_secret_existsmaybe_first_run_calls_first_run_when_not_initializedfirst_run_initialises_garm_with_the_stored_credentialsget_admin_credentials_returns_none_when_secret_not_foundreconcile_runners_skips_when_no_admin_credentialsreconcile_skipped_when_admin_credentials_are_unavailablemaybe_first_run_skips_on_missing_credential_keymaybe_first_run_skips_when_credentials_unavailablefirst_run_skipped_when_credentials_are_incompletebuild_desired_credentials_skips_incomplete_unitbuild_desired_credentials_skips_non_numeric_idsbuild_desired_credentials_skips_when_secret_unavailableincomplete_github_config_yields_no_credentialensure_controller_urls_sets_urls_from_base_urlensure_controller_urls_strips_trailing_slash_from_base_urlcontroller_urls_derive_from_the_ingress_urlrestart_no_proxy_hash_matches_rendered_configThe merges collapse tests that a Scenario
parametrizeexpresses as cases of one behaviour;no assertion was lost in any of them. The one dropped test recomputed the expected hash
using the same
if proxy_env:guard as the charm itself, so it could not fail —unchanged_config_does_not_restart_the_workloadcovers the real property instead.The two
get_credentials_action_*rows come frommain'sget-credentialsaction (#254),which landed after this branch opened and added its tests in the mock-
selfstyle this PRremoves. They are ported rather than dropped or kept as-is: the Scenario versions run the
action through the context and assert on
ctx.action_resultsand onActionFailed, so theycover the action's real path to the secret rather than a
MagicMock's return value.New tests with no ancestor on main (6) — coverage the mock-based style could not reach:
workload_is_configured_from_relation_dataworkload_is_not_configured_without_postgresql_datanon_leader_waits_for_the_leader_to_create_secretsreconcile_authenticates_against_the_local_garm_listenerscaleset_is_built_from_relation_datacontroller_urls_default_to_the_kubernetes_service_urlThe whole table is verified against the branch rather than by eye — with the 27 mapped names
in
mapped.txt:Review focus
Rewriting the tests surfaced two pre-existing bugs. Neither is fixed here — this PR is test
work only — and both are being addressed in a follow-up that builds on these tests:
restart()adds thegarm-commandlayer (service
app) beforesuper().restart(). paas_charm's_app_layer()snapshots theplan into
original-services.jsonon first call and re-emits those services into thecharmlayer, which is added later and wins. The
appservice is frozen at its first-run values, soproxy changes never reach the running service, and
_current_config_hash()reads the frozenhash — meaning the unchanged-config short-circuit never fires again after the first config
change. The old tests missed this by asserting on the dict passed to
add_layerrather thanthe resulting plan.
paas_charm.restart()ends with an unconditionalActiveStatus()after_reconcile_runners()has set
Waiting.Two tests changed meaning as a result, and are worth a look:
test_charmed_template_error_degrades_to_waitingnow exercises the steady-state(unchanged-config) path, the only path where the Waiting status survives (bug 2).
*_workload_layerproxy tests assert on thegarm-commandlayer the charm writes,matching what the previous tests checked; asserting the effective plan would fail on bug 1.
The follow-up renames them once they can assert the plan.
Also:
restart()'s"Waiting for postgresql relation"branch is unreachable — the framework'soptional: falsegate blocks withmissing integrations: postgresqlfirst — so that test nowasserts the real status.
New dependencies
ops-scenario==8.8.0added to theunitenv incharms/garm/tox.toml, matching the pinalready used by
garm-configurator. No runtime dependency change; no breaking changes.Checklist
CONTRIBUTING.mdhas been updated upon changes to the contribution/development process (e.g. changes to the way tests are run)docs/changelog.mdwith user-relevant changesterraform fmtpasses andtflintreports no errorsAGENTS.md.copilot-collections.yamlor.github/instructions/: I re-checked whether theAGENTS.md"12-factor divergences" guidance still matches the upstream copilot-collections guidance