Skip to content

USHIFT-7342: Replace hardcoded sleep with state check before restart #6996

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pacevedom wants to merge 2 commits into
base: main
Choose a base branch
from
+70 −7
Open

USHIFT-7342: Replace hardcoded sleep with state check before restart #6996

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f832770
Replace hardcoded sleep with state check before restart
pacevedom Jul 7, 2026
181cc5a
Fix transition check timeout, align waits, document rationale
pacevedom Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 70 additions & 7 deletions test/resources/microshift-process.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,23 +73,86 @@ MicroShift Service Is Active
Should Be Equal As Strings ${state} active

Wait For MicroShift Service
[Documentation] Wait for MicroShift service to be active
Wait Until Keyword Succeeds 50x 10s
[Documentation] Wait for MicroShift service to be active.
... Budget covers TimeoutStartSec=4m (240s) with margin.
Wait Until Keyword Succeeds 30x 10s
... MicroShift Service Is Active

Restart MicroShift
[Documentation] Restart the MicroShift service with a delay to avoid too
... many start requests.
Restart MicroShift # robocop: off=LEN01
[Documentation] Restart the MicroShift service safely, handling all systemd
... states including auto-restart loops after failures.
...
... Background: microshift.service has Restart=always. After a startup
... failure (e.g. invalid config), systemd auto-restarts the process
... repeatedly. Each attempt fails and increments the rate-limit counter.
... Once StartLimitBurst (default 5) is exceeded within
... StartLimitIntervalSec (default 10s), systemd triggers start-limit-hit
... and the service enters the "failed" state, refusing further starts.
...
... Step 1 — Wait for transition to settle (30x 10s = 300s).
... Poll ActiveState until it leaves activating/deactivating/reloading.
... This lets the auto-restart loop run its course and hit start-limit-hit
... before we proceed. Without this wait, reset-failed (step 2) clears the
... rate-limit counter, but pending auto-restarts keep firing and
... re-increment it — our restart in step 4 would be rejected.
... Budget covers TimeoutStartSec=4m (240s) for legitimate slow startups.
...
... Step 2 — Clear start-limit-hit via reset-failed.
... Safe to call now because the auto-restart loop has settled (step 1
... confirmed the service is no longer transitioning). Resets the
... rate-limit counter and clears the "failed" state so the next start
... is accepted.
...
... Step 3 — Capture current PIDs for both microshift and microshift-etcd.
... Taken after stability is confirmed (step 1) so the PIDs are stable
... and will not change before our restart in step 4.
...
... Step 4 — Issue systemctl restart (synchronous).
... The Systemctl keyword runs the command, checks RC=0, and waits for
... SubState=running. For Type=notify, this blocks until the new process
... sends READY=1 or TimeoutStartSec expires.
...
... Step 5 — Verify PID changes for microshift and microshift-etcd.
... Confirms both processes actually restarted as new PIDs. Passes
... instantly since step 4 already waited for the service to be running.
... Defense-in-depth against Restart=always auto-restart racing with our
... restart — ensures the running process is ours, not a stale
... auto-restarted instance that happened to reach SubState=running.
...
... Step 6 — Refresh kubeconfig and wait for readiness.
... Certificates may rotate across restarts, so kubeconfig must be
... re-fetched. Wait For MicroShift polls /readyz and /livez as a
... final sanity check that the API surface is fully available.
...
... All wait budgets are 30x 10s (300s), covering TimeoutStartSec=4m
... (240s) with margin.
#
# Step 1: wait for auto-restart loop to settle
Wait Until Keyword Succeeds 30x 10s
... MicroShift Service Is Not Transitioning
# Step 2: clear start-limit-hit
Execute Command systemctl reset-failed microshift.service sudo=True
Comment thread
ggiguash marked this conversation as resolved.
# Step 3: snapshot PIDs before restart
${ushift_pid}= MicroShift Process ID
${ushift_etcd_pid}= MicroShift Etcd Process ID
Sleep 10s
# Step 4: synchronous restart
Systemctl restart microshift.service
# Step 5: confirm PID changes
Wait Until MicroShift Process ID Changes ${ushift_pid}
Wait Until MicroShift Etcd Process ID Changes ${ushift_etcd_pid}
# Kubeconfig may change between restarts, ensure we are taking the latest one.
# Step 6: refresh kubeconfig and verify readiness
Setup Kubeconfig
Wait For MicroShift

MicroShift Service Is Not Transitioning
[Documentation] Verify microshift.service is not in a transitional ActiveState
... (activating, deactivating, reloading). This covers all transient
... sub-states including auto-restart and start-pre/start/stop phases.
${state}= Get Systemd Setting microshift.service ActiveState
Should Not Be Equal As Strings ${state} activating
Should Not Be Equal As Strings ${state} deactivating
Should Not Be Equal As Strings ${state} reloading

Stop MicroShift
[Documentation] Stop the MicroShift service
Systemctl stop microshift.service
Expand Down