From b8bdfb31f6841c15e09f31b1f6a292fac080b58f Mon Sep 17 00:00:00 2001 From: elkaix Date: Thu, 13 Aug 2026 00:06:26 -0400 Subject: [PATCH] feat: make lease and job-lock ownership generation-scoped The write lease and the companion job lock now claim a generation token and publish metadata atomically inside that claim, so acquisition, heartbeat, poison staging, release, and --clear-lease all verify they are acting on the generation they own instead of on whatever currently sits in the lock directory. Every ambiguous outcome fails closed and retains the lock rather than releasing a lease that may still cover a live writer. Also in this change: - A failed lease release downgrades an otherwise successful loop or watchdog run to BLOCKED/11 instead of reporting completion over a retained lock. - STUCK now persists a bounded tail of the attempts log to the plan file in a delimited MAESTRO ATTEMPT HISTORY block, so the evidence survives the run. - A post-launch companion, process, or result-channel fault reports IMPLEMENTER_STATE: COMPANION_FAILURE and ends BLOCKED without billing an implementation attempt. - --verify runs from the lease repository root, so root-relative verification no longer depends on the caller's shell directory. - Contention progress is emitted on the first wait tick and then at most every 30 seconds, and distinguishes the owner's lease age from the contender's wait window. - Companion status JSON is parsed from the status file rather than piped through stdin. --- README.md | 8 +- hooks/implementer-loop.sh | 150 +++++- hooks/implementer-watchdog.sh | 14 +- hooks/lib-companion.sh | 93 ++-- hooks/lib-job-lock.sh | 552 +++++++++++++++++++--- hooks/lib-process.sh | 15 +- hooks/lib-write-lease.sh | 659 ++++++++++++++++++--------- hooks/lib-write-turn.sh | 21 +- rules/orchestrator-implementer.md | 9 +- tests/bounded-calls.sh | 17 +- tests/detection.sh | 5 +- tests/fixtures/fake-companion.mjs | 25 +- tests/job-lock.sh | 631 +++++++++++++++++++++++++- tests/lease.sh | 730 ++++++++++++++++++++++++++++-- tests/liveness.sh | 222 +++++++-- tests/provenance-edge.sh | 58 ++- tests/stop-report.sh | 461 ++++++++++++++++++- 17 files changed, 3215 insertions(+), 455 deletions(-) diff --git a/README.md b/README.md index 814c4fd..0e3b7f8 100644 --- a/README.md +++ b/README.md @@ -74,14 +74,16 @@ The loop never ends in prose. Every run finishes on a machine-readable state: |:---:|---|---|---| | `0` | **VERIFIED_DONE** | Plan executed *and* the verify command passed locally | Review the diff — a claim is not proof | | `10` | **NEEDS_ANSWERS** | Codex hit real ambiguity and stopped instead of guessing | Answer the `QUESTIONS:` block, re-run | -| `11` | **BLOCKED** | Missing access, a destructive step, lease contention, or an unconfirmed cancelled writer | Surface it; never improvise around it | -| `12` | **STUCK** | Hit the iteration cap without verification | Read the attempts log, re-plan — don't just raise the cap | +| `11` | **BLOCKED** | Missing access, a destructive step, lease contention, an unconfirmed cancelled writer, or a post-launch companion/process/result failure with no structured implementer result | Surface it; inspect companion-failure evidence and the current diff before a fresh dispatch | +| `12` | **STUCK** | Hit the iteration cap without verification | Read the bounded attempt history appended to the plan, re-plan — don't just raise the cap | These codes describe `implementer-loop.sh`. The peer single-shot adapter uses rc `125` plus `MAESTRO_FINAL: WATCHDOG POISONED` for the same unconfirmed Write turn cancellation; `implementer-loop.sh` maps that outcome to `BLOCKED`/11. One Implementation run acquires one Lease interval across every Write turn and local Verification transaction, then releases or retains it once at the terminal state. The peer watchdog adapter acquires the same interval for its single Write turn. +The loop evaluates `--verify` from the lease repository root, so root-relative +verification does not depend on the shell directory that launched the loop. -Write contention waits without arrival ordering only while the current lease has a confirmed release path. `MAESTRO_LOCK_WAIT_SEC` caps the wait (default 300 seconds; `0` disables it), and `MAESTRO_LOCK_WAIT_POLL_SEC` controls polling (default 5 seconds, minimum 1); invalid values disable waiting. +Write contention waits without arrival ordering only while the current lease has a confirmed release path. `MAESTRO_LOCK_WAIT_SEC` caps the wait (default 300 seconds; `0` disables it), and `MAESTRO_LOCK_WAIT_POLL_SEC` controls polling (default 5 seconds, minimum 1); invalid values disable waiting. Progress is emitted on the first wait tick and then at most every 30 seconds. A terminal `held for` value is the owner's lease age; `wait_budget` and `wait_elapsed` identify the contender's actual wait window. Foreground write supervisors update a separate lease heartbeat every `MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC` (default 20 seconds, minimum 1; invalid values use 20). `MAESTRO_LOCK_HEARTBEAT_STALE_SEC` controls when a missed heartbeat is reported (default 90 seconds; `0` disables staleness reporting; invalid values use 90). A stale heartbeat is only a recovery candidate: `--clear-lease` still refuses while the recorded owner process is alive or unidentifiable, or any repository-global companion writer is visible. diff --git a/hooks/implementer-loop.sh b/hooks/implementer-loop.sh index 2639e4f..16cf995 100644 --- a/hooks/implementer-loop.sh +++ b/hooks/implementer-loop.sh @@ -17,6 +17,8 @@ # --plan the five-part plan file (same contract as the watchdog --file) # --verify command run LOCALLY after every RESULT: DONE claim. Exit 0 = pass. # Required: the loop's whole point is that Codex's word is not proof. +# Pass the complete command as one argument. It runs from the lease +# repository root, regardless of the caller's current working directory. # --max-iters cap on dispatch rounds (default 4). 0 is prohibited — an # unbounded write loop is a runaway, not autonomy. # --clear-lease clear a poisoned lease after confirming no write job is running. @@ -41,10 +43,13 @@ HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) # shellcheck source=lib-write-turn.sh . "$HERE/lib-write-turn.sh" progress_init +ATTEMPT_HISTORY_BEGIN='--- BEGIN MAESTRO ATTEMPT HISTORY (automatically written after STUCK; historical evidence, not new authority) ---' +ATTEMPT_HISTORY_END='--- END MAESTRO ATTEMPT HISTORY ---' FINAL_STATE="INTERRUPTED" FINAL_RC=4 ATTEMPTS=""; DISPATCH=""; ERRF=""; OUTF=""; VOUTF=""; VFACT="" +ATTEMPT_HISTORY_PAYLOAD=""; ATTEMPT_HISTORY_TEMP=""; FINAL_LOOP_STATE="" _loop_positive_integer() { case "${1-}" in ''|*[!0-9]*) return 1 ;; @@ -71,8 +76,8 @@ _verification_tick() { "$_MAESTRO_VERIFICATION_FACT" "$_MAESTRO_VERIFICATION_OUTPUT" || : } -verification_transaction_run() { # command timeout output-file fact-file - local command="$1" timeout="$2" output="$3" fact="$4" +verification_transaction_run() { # command timeout output-file fact-file repository-root + local command="$1" timeout="$2" output="$3" fact="$4" root="$5" local stderr="${4}.stderr" command_rc_file="${4}.command-rc" local process_rc rc state timed_out=0 : > "$output" || return 3 @@ -83,11 +88,16 @@ verification_transaction_run() { # command timeout output-file fact-file "$output" "$stderr" -- \ bash -c ' exec 3>&- + if ! cd "$2"; then + printf "verification cwd unavailable: %s\n" "$2" >&2 + printf "3\n" > "$3" + exit 3 + fi bash -c "$1" rc=$? - printf "%s\n" "$rc" > "$2" + printf "%s\n" "$rc" > "$3" exit "$rc" - ' _ "$command" "$command_rc_file" + ' _ "$command" "$root" "$command_rc_file" process_rc=$? if [ "$process_rc" -eq 125 ] && [ ! -f "$command_rc_file" ]; then timed_out=1 @@ -115,6 +125,82 @@ verification_transaction_run() { # command timeout output-file fact-file progress "LOOP_WARNING: could not write verification fact $fact; using in-process state=$state rc=$rc." return "$rc" } +attempt_history_bound() { # file + local file="$1" max_lines=120 max_bytes=65536 + local line_tail="${1}.lines" byte_tail="${1}.bytes" bounded="${1}.bounded" + local bytes marker marker_bytes keep wrapper_bytes payload_max + marker='MAESTRO_ATTEMPT_HISTORY_TRUNCATED: earlier bytes omitted; retained the most recent evidence within the 65536-byte limit' + wrapper_bytes=$(printf '\n\n%s\n\n%s\n' \ + "$ATTEMPT_HISTORY_BEGIN" "$ATTEMPT_HISTORY_END" | wc -c | tr -d ' ') + payload_max=$((max_bytes - wrapper_bytes - (2 * (max_lines + 1)))) + if ! tail -n "$max_lines" "$file" > "$line_tail"; then + rm -f "$line_tail" "$byte_tail" "$bounded" + return 1 + fi + bytes=$(wc -c < "$line_tail" | tr -d ' ') + if [ "$bytes" -le "$payload_max" ]; then + mv -f "$line_tail" "$file" + return $? + fi + marker_bytes=$(printf '%s\n' "$marker" | wc -c | tr -d ' ') + keep=$((payload_max - marker_bytes)) + if ! tail -c "$keep" "$line_tail" > "$byte_tail" || + ! { printf '%s\n' "$marker"; cat "$byte_tail"; } > "$bounded" || + ! mv -f "$bounded" "$file"; then + rm -f "$line_tail" "$byte_tail" "$bounded" + return 1 + fi + rm -f "$line_tail" "$byte_tail" +} + +attempt_history_resolve_plan() { # plan + local path="$1" link hops=0 dir + case "$path" in + /*) ;; + *) path="$(pwd -P)/$path" ;; + esac + while [ -L "$path" ]; do + hops=$((hops + 1)) + [ "$hops" -le 40 ] || return 1 + link=$(readlink "$path") || return 1 + case "$link" in + /*) path=$link ;; + *) path="${path%/*}/$link" ;; + esac + done + dir=$(cd "${path%/*}" 2>/dev/null && pwd -P) || return 1 + printf '%s/%s\n' "$dir" "${path##*/}" +} + +attempt_history_persist() { # plan attempts + local target dir + target=$(attempt_history_resolve_plan "$1") || return 1 + dir=${target%/*} + ATTEMPT_HISTORY_PAYLOAD=$(mktemp /tmp/maestro-attempt-history.XXXXXXXX) || + return 1 + ATTEMPT_HISTORY_TEMP=$(mktemp "$dir/.maestro-history.XXXXXXXX") || { + rm -f "$ATTEMPT_HISTORY_PAYLOAD" + ATTEMPT_HISTORY_PAYLOAD="" + return 1 + } + if ! sed 's/^/> /' "$2" > "$ATTEMPT_HISTORY_PAYLOAD" || + ! cp -p "$target" "$ATTEMPT_HISTORY_TEMP" || + ! { + printf '\n\n%s\n' "$ATTEMPT_HISTORY_BEGIN" && + cat "$ATTEMPT_HISTORY_PAYLOAD" && + printf '%s\n' "$ATTEMPT_HISTORY_END" + } >> "$ATTEMPT_HISTORY_TEMP" || + ! mv -f "$ATTEMPT_HISTORY_TEMP" "$target"; then + rm -f "$ATTEMPT_HISTORY_PAYLOAD" "$ATTEMPT_HISTORY_TEMP" + ATTEMPT_HISTORY_PAYLOAD="" + ATTEMPT_HISTORY_TEMP="" + return 1 + fi + rm -f "$ATTEMPT_HISTORY_PAYLOAD" + ATTEMPT_HISTORY_PAYLOAD="" + ATTEMPT_HISTORY_TEMP="" +} + maestro_finish() { FINAL_STATE="$1" @@ -135,21 +221,40 @@ maestro_interrupt() { write_turn_interrupt "$signal" "$result" "$evidence" rc=$? if [ "$rc" -eq 125 ]; then - progress "LOOP_STATE: BLOCKED — interrupted before writer quiescence could be confirmed; lease retained" + FINAL_LOOP_STATE="LOOP_STATE: BLOCKED — interrupted before writer quiescence could be confirmed; lease retained" maestro_finish BLOCKED 11 fi maestro_finish INTERRUPTED 4 } cleanup() { + local release_rc=0 trap - EXIT HUP INT TERM - [ -n "$ATTEMPTS" ] && rm -f "$ATTEMPTS" + write_lease_end "${ERRF:-/dev/null}" + release_rc=$? + if [ "$release_rc" -ne 0 ]; then + case "$FINAL_RC" in + 11|125) ;; + *) + FINAL_STATE=BLOCKED + FINAL_RC=11 + progress "LOOP_RELEASE: completion downgraded to BLOCKED because the write lease could not be released safely" + FINAL_LOOP_STATE="LOOP_STATE: BLOCKED — the write lease could not be released safely" + ;; + esac + fi + [ -n "$ATTEMPTS" ] && rm -f "$ATTEMPTS" "${ATTEMPTS}.lines" \ + "${ATTEMPTS}.bytes" "${ATTEMPTS}.bounded" + [ -n "$ATTEMPT_HISTORY_PAYLOAD" ] && + rm -f "$ATTEMPT_HISTORY_PAYLOAD" "${ATTEMPT_HISTORY_PAYLOAD}.lines" \ + "${ATTEMPT_HISTORY_PAYLOAD}.bytes" "${ATTEMPT_HISTORY_PAYLOAD}.bounded" + [ -n "$ATTEMPT_HISTORY_TEMP" ] && rm -f "$ATTEMPT_HISTORY_TEMP" [ -n "$DISPATCH" ] && rm -f "$DISPATCH" [ -n "$ERRF" ] && rm -f "$ERRF" [ -n "$OUTF" ] && rm -f "$OUTF" [ -n "$VOUTF" ] && rm -f "$VOUTF" [ -n "$VFACT" ] && rm -f "$VFACT" "${VFACT}.new" \ "${VFACT}.stderr" "${VFACT}.command-rc" - write_lease_end "${ERRF:-/dev/null}" || : + [ -z "$FINAL_LOOP_STATE" ] || progress "$FINAL_LOOP_STATE" progress "MAESTRO_FINAL: LOOP $FINAL_STATE rc=$FINAL_RC" exit "$FINAL_RC" } @@ -265,6 +370,9 @@ if [ "$lock_rc" -ne 0 ]; then [ "$lock_rc" -eq 11 ] && maestro_finish "BLOCKED" 11 maestro_finish "FAILED" "$lock_rc" fi +VERIFY_ROOT=$(write_lock_scope_root 2>/dev/null) || + VERIFY_ROOT=$(pwd -P) || + maestro_finish "FAILED" 3 ATTEMPTS=$(mktemp /tmp/maestro-attempts.XXXXXXXX) ERRF=$(mktemp /tmp/maestro-looperr.XXXXXXXX) @@ -303,7 +411,7 @@ while [ "$i" -lt "$MAX_ITERS" ]; do if [ "$rc" -eq 125 ]; then printf '%s\n' "$OUT" - progress "LOOP_STATE: BLOCKED after $i iteration(s) — write lease retained because turn quiescence was never confirmed; clear it with --clear-lease once no Codex job is writing." + FINAL_LOOP_STATE="LOOP_STATE: BLOCKED after $i iteration(s) — write lease retained because turn quiescence was never confirmed; clear it with --clear-lease once no Codex job is writing." maestro_finish "BLOCKED" 11 fi @@ -319,6 +427,7 @@ while [ "$i" -lt "$MAX_ITERS" ]; do kind="failed"; [ "$rc" -eq 124 ] && kind="hung" printf '\n## Attempt %s — job %s before producing a result\n%s\n' "$i" "$kind" \ "$(tail -n 40 "$ERRF")" >> "$ATTEMPTS" + attempt_history_bound "$ATTEMPTS" || maestro_finish "FAILED" 3 progress "LOOP: iteration $i $kind — re-dispatching with the evidence" continue fi @@ -334,15 +443,19 @@ while [ "$i" -lt "$MAX_ITERS" ]; do progress "LOOP_WARNING: could not append the stop report to plan file $PLAN; continuing with exit 10." fi printf '%s\n' "$OUT" - progress "LOOP_STATE: NEEDS_ANSWERS after $i iteration(s) — $persistence_note; relay the QUESTIONS verbatim to the user, answer the questions, and re-run this loop." + FINAL_LOOP_STATE="LOOP_STATE: NEEDS_ANSWERS after $i iteration(s) — $persistence_note; relay the QUESTIONS verbatim to the user, answer the questions, and re-run this loop." maestro_finish "NEEDS_ANSWERS" 10 ;; BLOCKED) printf '%s\n' "$OUT" - progress "LOOP_STATE: BLOCKED after $i iteration(s) — surface the blocker; never improvise around credentials or destructive steps." + if grep -q '^IMPLEMENTER_STATE: COMPANION_FAILURE$' "$ERRF" 2>/dev/null; then + FINAL_LOOP_STATE="LOOP_STATE: BLOCKED after $i iteration(s) — post-launch companion/process/result failure; inspect the evidence and current diff before a fresh dispatch. This did not consume another implementation attempt." + else + FINAL_LOOP_STATE="LOOP_STATE: BLOCKED after $i iteration(s) — surface the blocker; never improvise around credentials or destructive steps." + fi maestro_finish "BLOCKED" 11 ;; DONE) progress "LOOP: RESULT: DONE on iteration $i — verifying locally: $VERIFY" - verification_transaction_run "$VERIFY" "$VERIFY_TIMEOUT" "$VOUTF" "$VFACT" + verification_transaction_run "$VERIFY" "$VERIFY_TIMEOUT" "$VOUTF" "$VFACT" "$VERIFY_ROOT" vrc=$? vstate=$(verification_fact_value "$VFACT" state) if [ -z "$vstate" ]; then @@ -354,13 +467,13 @@ while [ "$i" -lt "$MAX_ITERS" ]; do esac fi if [ "$vstate" = lease-lost ]; then - progress "LOOP_STATE: BLOCKED — this loop no longer holds the write lease; stopping before re-dispatching" + FINAL_LOOP_STATE="LOOP_STATE: BLOCKED — this loop no longer holds the write lease; stopping before re-dispatching" maestro_finish "BLOCKED" 11 fi VOUT=$(cat "$VOUTF") if [ "$vstate" = passed ]; then printf '%s\n' "$OUT" - progress "LOOP_STATE: VERIFIED_DONE after $i iteration(s) — local verification passed." + FINAL_LOOP_STATE="LOOP_STATE: VERIFIED_DONE after $i iteration(s) — local verification passed." maestro_finish "VERIFIED_DONE" 0 fi if [ "$vstate" = timed-out ]; then @@ -383,9 +496,16 @@ while [ "$i" -lt "$MAX_ITERS" ]; do progress "LOOP: iteration $i RESULT: FAILED — re-dispatching with the evidence" ;; esac + attempt_history_bound "$ATTEMPTS" || maestro_finish "FAILED" 3 done +attempt_history_note="attempt history could not be appended to $PLAN" +if [ -w "$PLAN" ] && attempt_history_persist "$PLAN" "$ATTEMPTS"; then + attempt_history_note="bounded attempt history has been appended to $PLAN" +else + progress "LOOP_WARNING: could not append attempt history to plan file $PLAN; continuing with exit 12." +fi progress "LOOP_STUCK: $MAX_ITERS iteration(s) without verified completion. Attempts log:" -tail -n 120 "$ATTEMPTS" >&2 -progress "LOOP_STATE: STUCK — re-plan around the evidence above (a debugging discussion helps), or escalate to the user. Do not simply raise --max-iters." +cat "$ATTEMPTS" >&2 +FINAL_LOOP_STATE="LOOP_STATE: STUCK — $attempt_history_note; re-plan around the evidence above (a debugging discussion helps), or escalate to the user. Do not simply raise --max-iters." maestro_finish "STUCK" 12 diff --git a/hooks/implementer-watchdog.sh b/hooks/implementer-watchdog.sh index 38c86ed..a8786e6 100644 --- a/hooks/implementer-watchdog.sh +++ b/hooks/implementer-watchdog.sh @@ -36,8 +36,20 @@ maestro_interrupt() { } cleanup() { + local release_rc=0 trap - EXIT HUP INT TERM - write_lease_end "${EVIDENCE_FILE:-/dev/null}" || : + write_lease_end "${EVIDENCE_FILE:-/dev/null}" + release_rc=$? + if [ "$release_rc" -ne 0 ]; then + case "$FINAL_RC" in + 11|125) ;; + *) + FINAL_STATE=BLOCKED + FINAL_RC=11 + progress "WATCHDOG_RELEASE: completion downgraded to BLOCKED because the write lease could not be released safely" + ;; + esac + fi [ -z "$WORK" ] || rm -rf "$WORK" 2>/dev/null || : progress "MAESTRO_FINAL: WATCHDOG $FINAL_STATE rc=$FINAL_RC" exit "$FINAL_RC" diff --git a/hooks/lib-companion.sh b/hooks/lib-companion.sh index d3bd610..b7024fb 100644 --- a/hooks/lib-companion.sh +++ b/hooks/lib-companion.sh @@ -92,35 +92,30 @@ companion_call() { # stdout-file stderr-file [--tick fn] companion args... } companion_workspace_writers() { # companion stdout-file stderr-file - local C="$1" out="$2" err="$3" status parsed rc + local C="$1" out="$2" err="$3" parsed rc parsed="${out}.parsed" CODEX_COMPANION_SESSION_ID='' companion_call "$out" "$err" \ "$C" status --all --json rc=$? [ "$rc" -eq 0 ] || return 4 - status=$(cat "$out") - [ -n "${status//[[:space:]]/}" ] || return 4 - - printf '%s\n' "$status" | node -e ' - let input = ""; - process.stdin.setEncoding("utf8"); - process.stdin.on("data", (chunk) => { input += chunk; }); - process.stdin.on("end", () => { - try { - const value = JSON.parse(input); - if (!value || typeof value !== "object" || Array.isArray(value) || - !Array.isArray(value.running)) process.exit(4); - for (const job of value.running) { - if (!job || typeof job !== "object" || Array.isArray(job) || - typeof job.id !== "string" || !/^task-[a-z0-9][a-z0-9-]*[a-z0-9]$/.test(job.id) || - typeof job.write !== "boolean") process.exit(4); - process.stdout.write(`${job.id}\t${job.write}\n`); - } - } catch { - process.exit(4); + [ -s "$out" ] || return 4 + + node -e ' + const fs = require("node:fs"); + try { + const value = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); + if (!value || typeof value !== "object" || Array.isArray(value) || + !Array.isArray(value.running)) process.exit(4); + for (const job of value.running) { + if (!job || typeof job !== "object" || Array.isArray(job) || + typeof job.id !== "string" || !/^task-[a-z0-9][a-z0-9-]*[a-z0-9]$/.test(job.id) || + typeof job.write !== "boolean") process.exit(4); + process.stdout.write(`${job.id}\t${job.write}\n`); } - }); - ' > "$parsed" 2>/dev/null + } catch { + process.exit(4); + } + ' "$out" > "$parsed" 2>/dev/null rc=$? if [ "$rc" -ne 0 ]; then rm -f "$parsed" @@ -228,7 +223,11 @@ companion_start() { # companion prompt mode model effort lifecycle job-file stdo fi JOB=$(printf '%s' "$START" | grep -oE 'task-[a-z0-9][a-z0-9-]*[a-z0-9]' | head -1) if [ -z "$JOB" ]; then - printf 'could not start Codex job. Output: %s' "$START" >&2 + printf 'Codex task launch succeeded but returned no job id. Output: %s\n' "$START" >&2 + if [ "$mode" = write ]; then + "$lifecycle" cancel-begin unknown launch-response-unparseable "$err" || : + return 125 + fi return 3 fi printf '%s\n' "$JOB" > "$job_file" || return 3 @@ -564,9 +563,9 @@ companion_poll() { done } -companion_result() { # companion job lifecycle result-file stdout stderr +companion_result() { # companion job lifecycle result-file stdout stderr evidence local C="$1" JOB="$2" lifecycle="$3" result="$4" out="$5" err="$6" - local OUT rc + local evidence="$7" OUT rc companion_call "$out" "$err" --tick "$lifecycle" "$C" result "$JOB" rc=$? OUT=$(cat "$out") @@ -577,7 +576,16 @@ companion_result() { # companion job lifecycle result-file stdout stderr OUT=$(cat "$out") fi if [ "$rc" -ne 0 ] || [ -z "${OUT//[[:space:]]/}" ]; then - printf 'job %s returned an empty result twice' "$JOB" >&2 + { + printf 'MAESTRO_COMPANION_RESULT_TRANSPORT: job=%s rc=%s; final transport tails follow (8192 bytes per stream)\n' \ + "$JOB" "$rc" + printf '%s\n' '--- result transport stdout ---' + tail -c 8192 "$out" 2>/dev/null || : + printf '\n%s\n' '--- result transport stderr ---' + tail -c 8192 "$err" 2>/dev/null || : + printf '\n%s\n' '--- end result transport evidence ---' + } >> "$evidence" 2>/dev/null || : + printf 'job %s returned an empty or failed result twice\n' "$JOB" >&2 return 4 fi printf '%s\n' "$OUT" > "$result" || return 4 @@ -603,18 +611,19 @@ _companion_profile_write() { # file mode model effort job cancel-reason cancel-r } > "$tmp" && mv -f "$tmp" "$file" } -_companion_turn_cleanup() { # profile-file - local profile="$1" +_companion_turn_cleanup() { # profile-file [retain-job-lock] + local profile="$1" retain_job_lock="${2:-0}" rm -f "${profile}.job" "${profile}.transport.out" \ "${profile}.transport.err" "${profile}.cancel" \ "${profile}.cancel.new" 2>/dev/null || : + [ "$retain_job_lock" -eq 1 ] && return 0 job_lock_release } companion_turn() { # mode prompt-file max-idle poll result-file profile-file evidence-file lifecycle local mode="${1-}" prompt_file="${2-}" max_idle="${3-}" poll="${4-}" local result="${5-}" profile="${6-}" evidence="${7-}" lifecycle="${8-}" - local prompt C pin efforts model effort job rc derived + local prompt C pin efforts model effort job rc cleanup_rc derived local override_model="${MAESTRO_COMPANION_MODEL-}" local override_effort="${MAESTRO_COMPANION_EFFORT-}" local override_model_set=0 override_effort_set=0 @@ -647,6 +656,8 @@ companion_turn() { # mode prompt-file max-idle poll result-file profile-file evi : > "$result" || return 3 : > "$evidence" || return 3 _companion_turn_cleanup "$profile" + rc=$? + [ "$rc" -eq 0 ] || return "$rc" rm -f "$profile" "${profile}.new" || return 3 C=$(companion_resolve) || return 3 @@ -677,6 +688,8 @@ companion_turn() { # mode prompt-file max-idle poll result-file profile-file evi rc=$? if [ "$rc" -ne 0 ]; then _companion_turn_cleanup "$profile" + cleanup_rc=$? + [ "$cleanup_rc" -eq 0 ] || return "$cleanup_rc" return "$rc" fi started=$SECONDS @@ -684,7 +697,15 @@ companion_turn() { # mode prompt-file max-idle poll result-file profile-file evi "$lifecycle" "$job_file" "$call_out" "$call_err" rc=$? if [ "$rc" -ne 0 ]; then + if [ "$rc" -eq 125 ]; then + _companion_profile_write "$profile" "$mode" "$model" "$effort" \ + unknown launch-response-unparseable unconfirmed || : + _companion_turn_cleanup "$profile" 1 + return 125 + fi _companion_turn_cleanup "$profile" + cleanup_rc=$? + [ "$cleanup_rc" -eq 0 ] || return "$cleanup_rc" return "$rc" fi job=$(cat "$job_file") @@ -708,7 +729,9 @@ companion_turn() { # mode prompt-file max-idle poll result-file profile-file evi _companion_profile_write "$profile" "$mode" "$model" "$effort" \ "$job" "${reason:-unknown}" "${request:-unknown}" || : _companion_turn_cleanup "$profile" - [ "$mode" = write ] && return 125 + cleanup_rc=$? + if [ "$mode" = write ]; then return 125; fi + [ "$cleanup_rc" -eq 0 ] || return "$cleanup_rc" return 4 fi if [ "$mode" = read ]; then @@ -722,12 +745,14 @@ companion_turn() { # mode prompt-file max-idle poll result-file profile-file evi rc=$? if [ "$rc" -eq 0 ]; then if ! companion_result "$C" "$job" "$lifecycle" "$result" \ - "$call_out" "$call_err"; then + "$call_out" "$call_err" "$evidence"; then _companion_turn_cleanup "$profile" + cleanup_rc=$? + [ "$cleanup_rc" -eq 0 ] || return "$cleanup_rc" return 4 fi _companion_turn_cleanup "$profile" - return 0 + return $? fi if [ -f "$cancel_fact" ]; then reason=$(companion_cancel_fact_value "$cancel_fact" reason) @@ -736,6 +761,8 @@ companion_turn() { # mode prompt-file max-idle poll result-file profile-file evi _companion_profile_write "$profile" "$mode" "$model" "$effort" \ "$job" "${reason:-unknown}" "${request:-unknown}" || : _companion_turn_cleanup "$profile" + cleanup_rc=$? + [ "$rc" -eq 125 ] || [ "$cleanup_rc" -eq 0 ] || return "$cleanup_rc" return "$rc" } diff --git a/hooks/lib-job-lock.sh b/hooks/lib-job-lock.sh index de772f3..1ca4500 100644 --- a/hooks/lib-job-lock.sh +++ b/hooks/lib-job-lock.sh @@ -6,9 +6,10 @@ _MAESTRO_JOB_LOCK_LOADED=1 MAESTRO_JOB_LOCK_TOKEN="" MAESTRO_JOB_LOCK_DIR="" +MAESTRO_JOB_LOCK_IDENTITY="" MAESTRO_JOB_LOCK_ACQUIRED=0 export -n MAESTRO_JOB_LOCK_TOKEN MAESTRO_JOB_LOCK_DIR \ - MAESTRO_JOB_LOCK_ACQUIRED 2>/dev/null || : + MAESTRO_JOB_LOCK_IDENTITY MAESTRO_JOB_LOCK_ACQUIRED 2>/dev/null || : job_lock_path() { local workspace git_dir @@ -22,13 +23,290 @@ job_lock_path() { } job_lock_metadata_value() { # metadata field - sed -n "s/^${2}=//p" "$1" 2>/dev/null | head -1 + sed -n "s/^${2}=//p" "$1" 19>&- 2>/dev/null | head -1 19>&- } -job_lock_path_mtime_epoch() { # path - stat -f '%m' "$1" 2>/dev/null || stat -c '%Y' "$1" 2>/dev/null +lock_claim_random_id() { + local value + value=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n') || return 1 + lock_claim_generation_valid "$value" || return 1 + printf '%s' "$value" +} +lock_claim_generation_valid() { + [ "${#1}" -eq 32 ] || return 1 + case "$1" in *[!0-9a-f]*) return 1 ;; esac +} +lock_claim_inode_identity() { + stat -c '%d:%i' "$1" 19>&- 2>/dev/null || + stat -f '%d:%i' "$1" 19>&- 2>/dev/null +} +lock_claim_path_identity() { # lock-dir + local lock_dir="$1" before after generation marker + marker="$lock_dir/generation" + [ -f "$marker" ] && [ ! -L "$marker" ] || return 1 + before=$(lock_claim_inode_identity "$lock_dir") || return 1 + generation=$(cat "$marker" 19>&-) || return 1 + lock_claim_generation_valid "$generation" || return 1 + after=$(lock_claim_inode_identity "$lock_dir") || return 1 + [ "$before" = "$after" ] || return 1 + printf '%s:%s' "$before" "$generation" +} +lock_claim_identity_generation() { + local generation="${1##*:}" + lock_claim_generation_valid "$generation" || return 1 + printf '%s' "$generation" +} +lock_claim_metadata_value_once() { # metadata field + local metadata="$1" field="$2" count + count=$(sed -n "s/^${field}=//p" "$metadata" 19>&- 2>/dev/null | + wc -l 19>&- | tr -d '[:space:]' 19>&-) || return 1 + [ "$count" = 1 ] || return 1 + job_lock_metadata_value "$metadata" "$field" +} +lock_claim_record_matches_identity() { # metadata identity + local expected actual + expected=$(lock_claim_identity_generation "$2") || return 1 + actual=$(lock_claim_metadata_value_once "$1" generation) || return 1 + [ "$actual" = "$expected" ] +} +job_lock_path_identity() { + lock_claim_path_identity "$1" +} + +job_lock_path_mtime_epoch() { + stat -c '%Y' "$1" 19>&- 2>/dev/null || + stat -f '%m' "$1" 19>&- 2>/dev/null +} +# Shared generation fence for companion and write locks. `lockf`/`flock` +# ownership is released by the kernel when a claimant dies, including SIGKILL; +# the stable gate inode is never unlinked or replaced. +_MAESTRO_LOCK_CLAIM_DIR="" +_MAESTRO_LOCK_CLAIM_IDENTITY="" +_MAESTRO_LOCK_CLAIM_TOKEN="" +_MAESTRO_LOCK_CLAIM_NONCE="" +_MAESTRO_LOCK_CLAIM_GATE="" +_MAESTRO_LOCK_CLAIM_SAVED_FD="" +_MAESTRO_LOCK_CLAIM_MODE="" +case "$(uname -s 2>/dev/null)" in + Darwin|FreeBSD|NetBSD|OpenBSD) _MAESTRO_LOCK_CLAIM_BACKEND=lockf ;; + Linux) _MAESTRO_LOCK_CLAIM_BACKEND=flock ;; + *) _MAESTRO_LOCK_CLAIM_BACKEND=unavailable ;; +esac +export -n _MAESTRO_LOCK_CLAIM_DIR _MAESTRO_LOCK_CLAIM_IDENTITY \ + _MAESTRO_LOCK_CLAIM_TOKEN _MAESTRO_LOCK_CLAIM_NONCE \ + _MAESTRO_LOCK_CLAIM_GATE _MAESTRO_LOCK_CLAIM_SAVED_FD \ + _MAESTRO_LOCK_CLAIM_MODE _MAESTRO_LOCK_CLAIM_BACKEND 2>/dev/null || : + +lock_claim_current_matches() { # lock-dir identity token [present|initializing|absent|opaque] + local lock_dir="$1" identity="$2" token="$3" mode="${4:-present}" + local current_identity current_token metadata + case "$mode" in present|initializing|absent|opaque) ;; *) return 1 ;; esac + current_identity=$(lock_claim_path_identity "$lock_dir") || return 1 + [ "$current_identity" = "$identity" ] || return 1 + [ "$mode" != opaque ] || return 0 + metadata="$lock_dir/metadata" + if [ ! -e "$metadata" ] && [ ! -L "$metadata" ]; then + [ "$mode" = initializing ] || [ "$mode" = absent ] + return + fi + [ "$mode" != absent ] && [ -f "$metadata" ] && [ ! -L "$metadata" ] || + return 1 + current_token=$(lock_claim_metadata_value_once "$metadata" token) || return 1 + [ "$current_token" = "$token" ] || return 1 + lock_claim_record_matches_identity "$metadata" "$identity" +} + +lock_claim_gate_path() { # lock-dir + local lock_dir="$1" + case "$lock_dir" in + /*/*) printf '%s/maestro-generation-claim.lock' "${lock_dir%/*}" ;; + *) return 1 ;; + esac +} + +lock_claim_fd19_save() { + local fd + [ -z "$_MAESTRO_LOCK_CLAIM_SAVED_FD" ] || return 1 + if ! eval ': <&19' 2>/dev/null; then + _MAESTRO_LOCK_CLAIM_SAVED_FD=closed + return 0 + fi + fd=20 + while [ "$fd" -le 255 ]; do + if ! eval ": <&$fd" 2>/dev/null; then + eval "exec $fd<&19" || return 1 + _MAESTRO_LOCK_CLAIM_SAVED_FD=$fd + return 0 + fi + fd=$((fd + 1)) + done + return 1 +} + +lock_claim_fd19_restore() { + local saved="$_MAESTRO_LOCK_CLAIM_SAVED_FD" + [ -n "$saved" ] || return 1 + exec 19>&- || : + if [ "$saved" != closed ]; then + eval "exec 19<&$saved" || return 1 + eval "exec $saved>&-" || return 1 + fi + _MAESTRO_LOCK_CLAIM_SAVED_FD="" +} + +lock_claim_gate_acquire() { # gate-path + local gate="$1" rc=1 + (umask 077; : >> "$gate") || return 1 + lock_claim_fd19_save || return 1 + if ! exec 19>>"$gate"; then + lock_claim_fd19_restore || : + return 1 + fi + case "$_MAESTRO_LOCK_CLAIM_BACKEND" in + lockf) + command -v lockf >/dev/null 2>&1 && + lockf -s -t 0 19 >/dev/null 2>&1 + rc=$? + ;; + flock) + command -v flock >/dev/null 2>&1 && + flock -n 19 >/dev/null 2>&1 + rc=$? + ;; + esac + if [ "$rc" -ne 0 ]; then + lock_claim_fd19_restore || : + return 1 + fi } +lock_claim_publish_generation() { # lock-dir generation + local marker="$1/generation" + [ ! -e "$marker" ] && [ ! -L "$marker" ] || return 1 + (umask 077; printf '%s\n' "$2" > "$marker") 19>&- +} + +lock_claim_create() { # lock-dir output-identity; 0=created, 1=exists/clean-failure, 2=gate unavailable, 3=unsafe residue + local lock_dir="$1" output_var="${2-}" gate generation created_identity rc=1 + case "$output_var" in ''|[0-9]*|*[!A-Za-z0-9_]*) return 1 ;; esac + printf -v "$output_var" '%s' "" + [ -z "$_MAESTRO_LOCK_CLAIM_DIR" ] || return 2 + generation=$(lock_claim_random_id) || return 1 + gate=$(lock_claim_gate_path "$lock_dir") || return 2 + lock_claim_gate_acquire "$gate" || return 2 + if (umask 077; mkdir "$lock_dir") 19>&- 2>/dev/null; then + if ! lock_claim_publish_generation "$lock_dir" "$generation"; then + rm -f "$lock_dir/generation" 19>&- 2>/dev/null || : + if rmdir "$lock_dir" 19>&- 2>/dev/null || + { [ ! -e "$lock_dir" ] && [ ! -L "$lock_dir" ]; }; then + rc=1 + else + rc=3 + fi + elif created_identity=$(lock_claim_path_identity "$lock_dir"); then + printf -v "$output_var" '%s' "$created_identity" + rc=0 + else + rc=3 + fi + fi + lock_claim_fd19_restore || return 2 + return "$rc" +} + +lock_claim_forget() { + _MAESTRO_LOCK_CLAIM_DIR="" + _MAESTRO_LOCK_CLAIM_IDENTITY="" + _MAESTRO_LOCK_CLAIM_TOKEN="" + _MAESTRO_LOCK_CLAIM_NONCE="" + _MAESTRO_LOCK_CLAIM_GATE="" + _MAESTRO_LOCK_CLAIM_MODE="" +} + +lock_claim_unlock() { + local rc=0 + lock_claim_fd19_restore || rc=$? + lock_claim_forget + _MAESTRO_LOCK_CLAIM_SAVED_FD="" + return "$rc" +} +lock_claim_acquire() { # lock-dir identity token [present|initializing|absent|opaque] + local lock_dir="$1" identity="$2" token="$3" mode="${4:-present}" gate nonce + [ -n "$identity" ] && [ -n "$token" ] || return 1 + case "$mode" in present|initializing|absent|opaque) ;; *) return 1 ;; esac + [ -z "$_MAESTRO_LOCK_CLAIM_DIR" ] || return 1 + lock_claim_current_matches "$lock_dir" "$identity" "$token" "$mode" || + return 1 + gate=$(lock_claim_gate_path "$lock_dir") || return 1 + nonce=$(lock_claim_random_id) || return 1 + lock_claim_gate_acquire "$gate" || return 1 + _MAESTRO_LOCK_CLAIM_DIR=$lock_dir + _MAESTRO_LOCK_CLAIM_IDENTITY=$identity + _MAESTRO_LOCK_CLAIM_TOKEN=$token + _MAESTRO_LOCK_CLAIM_NONCE=$nonce + _MAESTRO_LOCK_CLAIM_GATE=$gate + _MAESTRO_LOCK_CLAIM_MODE=$mode + if ! lock_claim_current_matches "$lock_dir" "$identity" "$token" "$mode"; then + lock_claim_unlock + return 1 + fi +} + +lock_claim_path() { + [ "$_MAESTRO_LOCK_CLAIM_DIR" = "$1" ] || return 1 + printf '%s' "$1" +} + +lock_claim_current_path_matches() { # identity token + [ -n "$_MAESTRO_LOCK_CLAIM_DIR" ] && + lock_claim_current_matches "$_MAESTRO_LOCK_CLAIM_DIR" "$1" "$2" \ + "$_MAESTRO_LOCK_CLAIM_MODE" +} + +lock_claim_release() { # lock-dir identity token + local lock_dir="$1" identity="$2" token="$3" rc=1 + [ "$_MAESTRO_LOCK_CLAIM_DIR" = "$lock_dir" ] || return 1 + [ "$_MAESTRO_LOCK_CLAIM_IDENTITY" = "$identity" ] || return 1 + [ "$_MAESTRO_LOCK_CLAIM_TOKEN" = "$token" ] || return 1 + lock_claim_current_matches "$lock_dir" "$identity" "$token" \ + "$_MAESTRO_LOCK_CLAIM_MODE" && rc=0 + lock_claim_unlock || return 1 + return "$rc" +} + +lock_claim_move() { # lock-dir identity token destination + local lock_dir="$1" identity="$2" token="$3" destination="$4" + [ "$_MAESTRO_LOCK_CLAIM_DIR" = "$lock_dir" ] || return 1 + [ "$_MAESTRO_LOCK_CLAIM_IDENTITY" = "$identity" ] || return 1 + [ "$_MAESTRO_LOCK_CLAIM_TOKEN" = "$token" ] || return 1 + if ! lock_claim_current_path_matches "$identity" "$token" || + [ -e "$destination" ] || + ! mv "$lock_dir" "$destination" 19>&- 2>/dev/null; then + lock_claim_release "$lock_dir" "$identity" "$token" || : + return 1 + fi + if ! lock_claim_unlock; then + progress "MAESTRO_LOCK_CLEANUP: canonical lock retired; generation gate cleanup failed" + fi + return 0 +} + +lock_claim_discard() { # lock-dir identity token + local lock_dir="$1" identity="$2" token="$3" destination + destination="${lock_dir}.reclaim.${_MAESTRO_LOCK_CLAIM_NONCE:-$$}" + lock_claim_move "$lock_dir" "$identity" "$token" "$destination" || return 1 + if ! rm -rf "$destination" 2>/dev/null; then + progress "MAESTRO_LOCK_CLEANUP: canonical lock released; cleanup of retired generation failed: $destination" + fi + return 0 +} + +lock_claim_release_active() { + local lock_dir="$_MAESTRO_LOCK_CLAIM_DIR" + [ -n "$lock_dir" ] || return 0 + lock_claim_release "$lock_dir" "$_MAESTRO_LOCK_CLAIM_IDENTITY" \ + "$_MAESTRO_LOCK_CLAIM_TOKEN" || : +} job_lock_session_id() { local session_id="${MAESTRO_SESSION_ID:-}" case "$session_id" in @@ -52,15 +330,17 @@ job_lock_job_valid() { } job_lock_metadata_valid() { # metadata require-job - local metadata="$1" require_job="${2:-0}" token pid session class start job - [ -f "$metadata" ] || return 1 - token=$(job_lock_metadata_value "$metadata" token) + local metadata="$1" require_job="${2:-0}" token generation pid session class start job + [ -f "$metadata" ] && [ ! -L "$metadata" ] || return 1 + token=$(lock_claim_metadata_value_once "$metadata" token) || return 1 + generation=$(lock_claim_metadata_value_once "$metadata" generation) || return 1 pid=$(job_lock_metadata_value "$metadata" pid) session=$(job_lock_metadata_value "$metadata" session) class=$(job_lock_metadata_value "$metadata" class) start=$(job_lock_metadata_value "$metadata" start) job=$(job_lock_metadata_value "$metadata" job) [ -n "$token" ] || return 1 + lock_claim_generation_valid "$generation" || return 1 case "$token" in *[!A-Za-z0-9_-]*) return 1 ;; esac case "$pid" in ''|*[!0-9]*) return 1 ;; esac case "$session" in @@ -77,29 +357,46 @@ job_lock_metadata_valid() { # metadata require-job fi } -job_lock_publish_metadata() { # lock-dir token record - local lock_dir="$1" token="$2" record="$3" metadata temp recorded_token +job_lock_publish_metadata() { # lock-dir identity token record + local lock_dir="$1" identity="$2" token="$3" record="$4" + local metadata temp recorded_token metadata="$lock_dir/metadata" temp="$lock_dir/metadata.tmp.$token" - if [ -f "$metadata" ]; then - recorded_token=$(job_lock_metadata_value "$metadata" token) - [ "$recorded_token" = "$token" ] || return 1 - fi - (umask 077; printf '%s\n' "$record" > "$temp") || return 1 + lock_claim_acquire "$lock_dir" "$identity" "$token" initializing || return 1 if [ -f "$metadata" ]; then recorded_token=$(job_lock_metadata_value "$metadata" token) if [ "$recorded_token" != "$token" ]; then - rm -f "$temp" 2>/dev/null || : + lock_claim_release "$lock_dir" "$identity" "$token" || : return 1 fi fi - if mv -f "$temp" "$metadata"; then - return 0 + if ! (umask 077; printf '%s\n' "$record" > "$temp") 19>&- || + ! mv -f "$temp" "$metadata" 19>&-; then + rm -f "$temp" 19>&- 2>/dev/null || : + lock_claim_release "$lock_dir" "$identity" "$token" || : + return 1 fi - rm -f "$temp" 2>/dev/null || : - return 1 + if ! recorded_token=$(lock_claim_metadata_value_once "$metadata" token); then + lock_claim_release "$lock_dir" "$identity" "$token" || : + return 1 + fi + lock_claim_release "$lock_dir" "$identity" "$token" || return 1 + [ "$recorded_token" = "$token" ] } +job_lock_cleanup_failed_publication() { # lock-dir identity token + local lock_dir="$1" identity="$2" token="$3" current_identity current_token + [ -d "$lock_dir" ] || return 0 + lock_claim_acquire "$lock_dir" "$identity" "$token" initializing || return 1 + current_identity=$(job_lock_path_identity "$lock_dir") || current_identity="" + current_token=$(job_lock_metadata_value "$lock_dir/metadata" token) + if [ "$current_identity" != "$identity" ] || + { [ -n "$current_token" ] && [ "$current_token" != "$token" ]; }; then + lock_claim_release "$lock_dir" "$identity" "$token" || : + return 1 + fi + lock_claim_discard "$lock_dir" "$identity" "$token" +} job_lock_job_state() { # job output-variable local job="$1" output_var="$2" C scratch out err parsed rc observed="" case "$output_var" in ''|[0-9]*|*[!A-Za-z0-9_]*) return 4 ;; esac @@ -118,10 +415,12 @@ job_lock_job_state() { # job output-variable const [file, expected] = process.argv.slice(1); try { const value = JSON.parse(fs.readFileSync(file, "utf8")); + const job = value?.job; if (!value || typeof value !== "object" || Array.isArray(value) || - value.id !== expected || typeof value.status !== "string" || - !/^[A-Za-z][A-Za-z_-]*$/.test(value.status)) process.exit(4); - process.stdout.write(`${value.status.toLowerCase()}\n`); + !job || typeof job !== "object" || Array.isArray(job) || + job.id !== expected || typeof job.status !== "string" || + !/^[A-Za-z][A-Za-z_-]*$/.test(job.status)) process.exit(4); + process.stdout.write(`${job.status.toLowerCase()}\n`); } catch { process.exit(4); } ' "$out" "$job" > "$parsed" 2>/dev/null rc=$? @@ -176,13 +475,13 @@ job_lock_wait_tick() { # deadline class job session lock-dir job_lock_recovery_message() { # detail lock-dir progress "MAESTRO_JOB_LOCK: $1 (lock: $2). Recover with: bash ~/.claude/hooks/implementer-loop.sh --clear-job-lock" } - job_lock_acquire() { # read|write local class="${1-}" metadata token session now record wait_cap wait_poll deadline local recorded_token owner_pid owner_session owner_class owner_job state reclaim - local age mtime + local age mtime generation_identity current_identity create_rc case "$class" in read|write) ;; *) return 3 ;; esac MAESTRO_JOB_LOCK_TOKEN="" + MAESTRO_JOB_LOCK_IDENTITY="" MAESTRO_JOB_LOCK_ACQUIRED=0 MAESTRO_JOB_LOCK_DIR=$(job_lock_path) || return 3 metadata="$MAESTRO_JOB_LOCK_DIR/metadata" @@ -208,24 +507,52 @@ job_lock_acquire() { # read|write else deadline=0 fi + token=$(lock_claim_random_id) || { + progress "MAESTRO_JOB_LOCK: lock token generation failed; no lock was created" + return 3 + } + while :; do - if (umask 077; mkdir "$MAESTRO_JOB_LOCK_DIR") 2>/dev/null; then - token=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n') + create_rc=0 + generation_identity="" + lock_claim_create "$MAESTRO_JOB_LOCK_DIR" generation_identity || create_rc=$? + if [ "$create_rc" -eq 2 ]; then + if job_lock_wait_tick "$deadline" unknown unknown unknown \ + "$MAESTRO_JOB_LOCK_DIR"; then + continue + fi + progress "MAESTRO_JOB_LOCK: companion dispatch blocked by a competing generation claimant (lock: $MAESTRO_JOB_LOCK_DIR)" + return 11 + fi + if [ "$create_rc" -eq 3 ]; then + progress "MAESTRO_JOB_LOCK: lock generation initialization failed and the canonical generation could not be retired; retaining fail-closed lock (lock: $MAESTRO_JOB_LOCK_DIR)" + return 11 + fi + if [ "$create_rc" -eq 0 ]; then session=$(job_lock_session_id) now=$(date +%s) - record=$(printf 'token=%s\npid=%s\nsession=%s\nclass=%s\nstart=%s' \ - "$token" "$$" "${session:-unknown}" "$class" "$now") - if ! job_lock_publish_metadata "$MAESTRO_JOB_LOCK_DIR" "$token" "$record"; then - rm -f "$metadata" "$MAESTRO_JOB_LOCK_DIR"/metadata.tmp.* 2>/dev/null || : - rmdir "$MAESTRO_JOB_LOCK_DIR" 2>/dev/null || : + generation=$(lock_claim_identity_generation "$generation_identity") || return 11 + record=$(printf 'token=%s\ngeneration=%s\npid=%s\nsession=%s\nclass=%s\nstart=%s' \ + "$token" "$generation" "$$" "${session:-unknown}" "$class" "$now") + if ! job_lock_publish_metadata "$MAESTRO_JOB_LOCK_DIR" "$generation_identity" "$token" "$record"; then + if ! job_lock_cleanup_failed_publication "$MAESTRO_JOB_LOCK_DIR" \ + "$generation_identity" "$token"; then + progress "MAESTRO_JOB_LOCK: metadata publication failed and the canonical generation could not be retired; retaining fail-closed lock (lock: $MAESTRO_JOB_LOCK_DIR)" + return 11 + fi return 3 fi MAESTRO_JOB_LOCK_TOKEN=$token + MAESTRO_JOB_LOCK_IDENTITY=$generation_identity MAESTRO_JOB_LOCK_ACQUIRED=1 return 0 fi [ -d "$MAESTRO_JOB_LOCK_DIR" ] || return 3 + generation_identity=$(job_lock_path_identity "$MAESTRO_JOB_LOCK_DIR") || { + job_lock_recovery_message "lock generation identity is unavailable; failing closed" "$MAESTRO_JOB_LOCK_DIR" + return 11 + } if [ ! -f "$metadata" ]; then mtime=$(job_lock_path_mtime_epoch "$MAESTRO_JOB_LOCK_DIR") || mtime="" @@ -245,8 +572,9 @@ job_lock_acquire() { # read|write return 11 fi - if ! job_lock_metadata_valid "$metadata" 0; then - job_lock_recovery_message "metadata is malformed; owner cannot be identified; failing closed" "$MAESTRO_JOB_LOCK_DIR" + if ! job_lock_metadata_valid "$metadata" 0 || + ! lock_claim_record_matches_identity "$metadata" "$generation_identity"; then + job_lock_recovery_message "metadata is malformed or belongs to another generation; owner cannot be identified; failing closed" "$MAESTRO_JOB_LOCK_DIR" return 11 fi recorded_token=$(job_lock_metadata_value "$metadata" token) @@ -278,19 +606,29 @@ job_lock_acquire() { # read|write progress "MAESTRO_JOB_LOCK: companion dispatch blocked; class=$owner_class job=$owner_job session=$owner_session remains $state (lock: $MAESTRO_JOB_LOCK_DIR)" return 11 fi - - [ "$(job_lock_metadata_value "$metadata" token)" = "$recorded_token" ] || continue - reclaim="${MAESTRO_JOB_LOCK_DIR}.reclaim.${recorded_token}.$$" - [ ! -e "$reclaim" ] || return 11 - if ! mv "$MAESTRO_JOB_LOCK_DIR" "$reclaim" 2>/dev/null; then + if ! lock_claim_acquire "$MAESTRO_JOB_LOCK_DIR" \ + "$generation_identity" "$recorded_token"; then + progress "MAESTRO_JOB_LOCK: companion dispatch blocked by a competing generation claimant (lock: $MAESTRO_JOB_LOCK_DIR)" + return 11 + fi + reclaim=$(lock_claim_path "$MAESTRO_JOB_LOCK_DIR") || { + lock_claim_release "$MAESTRO_JOB_LOCK_DIR" \ + "$generation_identity" "$recorded_token" || : + return 11 + } + if ! lock_claim_current_path_matches "$generation_identity" "$recorded_token"; then + lock_claim_release "$MAESTRO_JOB_LOCK_DIR" \ + "$generation_identity" "$recorded_token" || : continue fi - rm -rf "$reclaim" 2>/dev/null || return 11 + lock_claim_discard "$MAESTRO_JOB_LOCK_DIR" \ + "$generation_identity" "$recorded_token" || return 11 done } job_lock_publish_job() { # job-id - local job="${1-}" metadata recorded_token pid session class start record + local job="${1-}" metadata recorded_token generation pid session class start record + local expected_identity current_identity job_lock_job_valid "$job" || return 3 [ "${MAESTRO_JOB_LOCK_ACQUIRED:-0}" -eq 1 ] || return 3 metadata="${MAESTRO_JOB_LOCK_DIR:-}/metadata" @@ -300,36 +638,73 @@ job_lock_publish_job() { # job-id progress "MAESTRO_JOB_LOCK: lock generation changed while publishing job=$job" return 3 fi + expected_identity=${MAESTRO_JOB_LOCK_IDENTITY:-} + current_identity=$(job_lock_path_identity "$MAESTRO_JOB_LOCK_DIR") || current_identity="" + if [ -z "$expected_identity" ] || [ "$current_identity" != "$expected_identity" ]; then + progress "MAESTRO_JOB_LOCK: lock directory identity changed while publishing job=$job" + return 3 + fi + lock_claim_record_matches_identity "$metadata" "$expected_identity" || { + progress "MAESTRO_JOB_LOCK: lock metadata belongs to another generation while publishing job=$job" + return 3 + } pid=$(job_lock_metadata_value "$metadata" pid) session=$(job_lock_metadata_value "$metadata" session) class=$(job_lock_metadata_value "$metadata" class) start=$(job_lock_metadata_value "$metadata" start) - record=$(printf 'token=%s\npid=%s\nsession=%s\nclass=%s\nstart=%s\njob=%s' \ - "$recorded_token" "$pid" "$session" "$class" "$start" "$job") - job_lock_publish_metadata "$MAESTRO_JOB_LOCK_DIR" "$recorded_token" "$record" + generation=$(lock_claim_identity_generation "$expected_identity") || return 3 + record=$(printf 'token=%s\ngeneration=%s\npid=%s\nsession=%s\nclass=%s\nstart=%s\njob=%s' \ + "$recorded_token" "$generation" "$pid" "$session" "$class" "$start" "$job") + job_lock_publish_metadata "$MAESTRO_JOB_LOCK_DIR" "$expected_identity" "$recorded_token" "$record" } job_lock_release() { - local metadata recorded_token release_dir + local metadata recorded_token observed_identity [ "${MAESTRO_JOB_LOCK_ACQUIRED:-0}" -eq 1 ] || return 0 metadata="${MAESTRO_JOB_LOCK_DIR:-}/metadata" - [ -f "$metadata" ] || return 0 - recorded_token=$(job_lock_metadata_value "$metadata" token) - [ "$recorded_token" = "${MAESTRO_JOB_LOCK_TOKEN:-}" ] || return 0 - release_dir="${MAESTRO_JOB_LOCK_DIR}.release.${recorded_token}.$$" - [ ! -e "$release_dir" ] || return 0 - [ "$(job_lock_metadata_value "$metadata" token)" = "$recorded_token" ] || return 0 - if mv "$MAESTRO_JOB_LOCK_DIR" "$release_dir" 2>/dev/null; then - rm -rf "$release_dir" 2>/dev/null || : - MAESTRO_JOB_LOCK_ACQUIRED=0 - MAESTRO_JOB_LOCK_TOKEN="" + if [ -z "${MAESTRO_JOB_LOCK_DIR:-}" ] || [ ! -f "$metadata" ]; then + progress "MAESTRO_JOB_LOCK: acquired lock metadata is unavailable during release; retaining local ownership" + return 11 + fi + observed_identity=${MAESTRO_JOB_LOCK_IDENTITY:-} + if [ -z "$observed_identity" ]; then + progress "MAESTRO_JOB_LOCK: acquired lock identity is unavailable during release; retaining the lock" + return 11 + fi + recorded_token=$(lock_claim_metadata_value_once "$metadata" token) || recorded_token="" + if [ "$recorded_token" != "${MAESTRO_JOB_LOCK_TOKEN:-}" ]; then + progress "MAESTRO_JOB_LOCK: lock token changed before release; retaining the lock" + return 11 fi + if ! lock_claim_record_matches_identity "$metadata" "$observed_identity"; then + progress "MAESTRO_JOB_LOCK: lock metadata belongs to another generation before release; retaining the lock" + return 11 + fi + if ! lock_claim_acquire "$MAESTRO_JOB_LOCK_DIR" \ + "$observed_identity" "$recorded_token"; then + progress "MAESTRO_JOB_LOCK: generation claim is unavailable during release; retaining the lock" + return 11 + fi + if ! lock_claim_current_path_matches "$observed_identity" "$recorded_token"; then + lock_claim_release "$MAESTRO_JOB_LOCK_DIR" \ + "$observed_identity" "$recorded_token" || : + progress "MAESTRO_JOB_LOCK: generation changed during release; releasing nothing" + return 11 + fi + if ! lock_claim_discard "$MAESTRO_JOB_LOCK_DIR" \ + "$observed_identity" "$recorded_token"; then + progress "MAESTRO_JOB_LOCK: generation could not be retired during release; retaining the lock" + return 11 + fi + MAESTRO_JOB_LOCK_ACQUIRED=0 + MAESTRO_JOB_LOCK_TOKEN="" + MAESTRO_JOB_LOCK_IDENTITY="" return 0 } - job_lock_clear() { # result-file evidence-file local result="${1-}" evidence="${2-}" lock_dir metadata age_path mtime now age local valid=0 token pid session class job state jobs reclaim current_token + local observed_identity current_identity claim_token claim_mode=present [ -n "$result" ] && [ -n "$evidence" ] && [ "$result" != "$evidence" ] || return 3 : > "$evidence" || return 3 printf 'state=blocked\n' > "$result" || return 3 @@ -339,6 +714,10 @@ job_lock_clear() { # result-file evidence-file printf 'state=absent\n' > "$result" return 0 fi + observed_identity=$(job_lock_path_identity "$lock_dir") || { + progress "MAESTRO_JOB_LOCK: refusing to clear — lock generation identity is unconfirmed (lock: $lock_dir)" + return 11 + } metadata="$lock_dir/metadata" token=$(job_lock_metadata_value "$metadata" token) pid=$(job_lock_metadata_value "$metadata" pid) @@ -356,18 +735,23 @@ job_lock_clear() { # result-file evidence-file ;; esac - if job_lock_metadata_valid "$metadata" 1; then valid=1; fi + if job_lock_metadata_valid "$metadata" 1 && + lock_claim_record_matches_identity "$metadata" "$observed_identity"; then + valid=1 + fi if [ "$valid" -eq 1 ]; then if ! job_lock_job_state "$job" state; then jobs="" - if job_lock_workspace_jobs jobs && [ -n "$jobs" ]; then + if ! job_lock_workspace_jobs jobs; then + progress "MAESTRO_JOB_LOCK: refusing to clear — companion status for holder class=$class job=$job session=$session and repository-global status are unavailable or malformed" + return 11 + fi + if [ -n "$jobs" ]; then progress "MAESTRO_JOB_LOCK: refusing to clear — companion status for holder class=$class job=$job session=$session is unavailable while repository-global job $(printf '%s\n' "$jobs" | awk 'NF { print $1; exit }') is running" - else - progress "MAESTRO_JOB_LOCK: refusing to clear — companion status for holder class=$class job=$job session=$session is unavailable or malformed" + return 11 fi - return 11 - fi - if ! job_lock_state_terminal "$state"; then + progress "MAESTRO_JOB_LOCK: holder pid=$pid is dead and repository-global status confirms no running jobs; clearing class=$class job=$job session=$session" + elif ! job_lock_state_terminal "$state"; then progress "MAESTRO_JOB_LOCK: refusing to clear — holder class=$class job=$job session=$session remains $state" return 11 fi @@ -396,26 +780,44 @@ job_lock_clear() { # result-file evidence-file fi if [ "$valid" -eq 1 ]; then - current_token=$(job_lock_metadata_value "$metadata" token) - [ "$current_token" = "$token" ] || { - progress "MAESTRO_JOB_LOCK: refusing to clear — lock generation changed during recovery" - return 11 - } - reclaim="${lock_dir}.reclaim.${token}.$$" + claim_token=$token + claim_mode=present else + claim_token="untrusted-$observed_identity" + claim_mode=opaque + fi + if ! lock_claim_acquire "$lock_dir" "$observed_identity" "$claim_token" \ + "$claim_mode"; then + progress "MAESTRO_JOB_LOCK: refusing to clear — generation claim is unavailable" + return 11 + fi + reclaim=$(lock_claim_path "$lock_dir") || { + lock_claim_release "$lock_dir" "$observed_identity" "$claim_token" || : + return 11 + } + current_identity=$(job_lock_path_identity "$reclaim") || current_identity="" + if [ "$current_identity" != "$observed_identity" ]; then + lock_claim_release "$lock_dir" "$observed_identity" "$claim_token" || : + progress "MAESTRO_JOB_LOCK: refusing to clear — lock generation changed during recovery" + return 11 + fi + metadata="$reclaim/metadata" + if [ "$valid" -eq 0 ]; then if job_lock_metadata_valid "$metadata" 0; then + lock_claim_release "$lock_dir" "$observed_identity" "$claim_token" || : progress "MAESTRO_JOB_LOCK: refusing to clear — metadata became valid during recovery" return 11 fi - token=$(od -An -N8 -tx1 /dev/urandom | tr -d ' \n') - reclaim="${lock_dir}.reclaim.clear-${token}.$$" + else + current_token=$(job_lock_metadata_value "$metadata" token) + if [ "$current_token" != "$token" ]; then + lock_claim_release "$lock_dir" "$observed_identity" "$claim_token" || : + progress "MAESTRO_JOB_LOCK: refusing to clear — lock generation changed during recovery" + return 11 + fi fi - [ ! -e "$reclaim" ] || return 11 - if ! mv "$lock_dir" "$reclaim" 2>/dev/null; then - progress "MAESTRO_JOB_LOCK: refusing to clear — lock generation changed during recovery" + lock_claim_discard "$lock_dir" "$observed_identity" "$claim_token" || return 11 - fi - rm -rf "$reclaim" 2>/dev/null || return 11 if [ -d "$lock_dir" ]; then progress "MAESTRO_JOB_LOCK: cleared the requested generation, but a successor already holds the lock" return 11 diff --git a/hooks/lib-process.sh b/hooks/lib-process.sh index 6b2d1ac..bde7d25 100755 --- a/hooks/lib-process.sh +++ b/hooks/lib-process.sh @@ -10,6 +10,7 @@ _MAESTRO_PROCESS_LOADED=1 # The module owns the spawned process group; callers own output and evidence files. _MAESTRO_PROCESS_PID="" +_MAESTRO_PROCESS_COMPLETION="" progress_init() { if ! { true >&3; } 2>/dev/null; then exec 3>&1; fi @@ -32,7 +33,7 @@ _process_group_stop() { # pid process_run_bounded() { # timeout label tick stdout stderr -- command [args...] local timeout="${1-}" label="${2-}" tick="${3-}" local stdout_file="${4-}" stderr_file="${5-}" - local pid start elapsed timed_out=0 ticks=0 rc + local pid start elapsed timed_out=0 ticks=0 rc completion shift 5 2>/dev/null || return 3 [ "${1-}" = "--" ] || return 3 shift @@ -48,17 +49,23 @@ process_run_bounded() { # timeout label tick stdout stderr -- command [args...] [ "$#" -gt 0 ] || return 3 : > "$stdout_file" || return 3 : > "$stderr_file" || return 3 + completion=$(mktemp "${TMPDIR:-/tmp}/maestro-process-completion.XXXXXX") || + return 3 + _MAESTRO_PROCESS_COMPLETION=$completion set -m ( "$@" > "$stdout_file" 2> "$stderr_file" + rc=$? + printf '%s\n' "$rc" > "$completion" + exit "$rc" ) & pid=$! _MAESTRO_PROCESS_PID=$pid set +m start=$SECONDS - while kill -0 "$pid" 2>/dev/null; do + while [ ! -s "$completion" ]; do elapsed=$((SECONDS - start)) if [ "$elapsed" -ge "$timeout" ]; then timed_out=1 @@ -83,6 +90,8 @@ process_run_bounded() { # timeout label tick stdout stderr -- command [args...] if kill -0 -"$pid" 2>/dev/null; then _process_group_stop "$pid" fi + rm -f "$completion" + _MAESTRO_PROCESS_COMPLETION="" _MAESTRO_PROCESS_PID="" if [ "$timed_out" -eq 1 ]; then @@ -98,6 +107,8 @@ process_interrupt() { # HUP|INT|TERM evidence-file [ -n "$pid" ] || return 0 _process_group_stop "$pid" wait "$pid" 2>/dev/null || : + rm -f "${_MAESTRO_PROCESS_COMPLETION:-}" + _MAESTRO_PROCESS_COMPLETION="" _MAESTRO_PROCESS_PID="" if [ -n "$evidence" ]; then printf 'MAESTRO_PROCESS: interrupted active process group with %s\n' "$signal" >> "$evidence" 2>/dev/null || : diff --git a/hooks/lib-write-lease.sh b/hooks/lib-write-lease.sh index 493e5cf..0f46364 100755 --- a/hooks/lib-write-lease.sh +++ b/hooks/lib-write-lease.sh @@ -20,9 +20,11 @@ _MAESTRO_WRITE_LEASE_HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) # proof that this shell owns the repository write interval. MAESTRO_LOCK_TOKEN="" MAESTRO_LOCK_DIR="" +MAESTRO_LOCK_IDENTITY="" MAESTRO_LOCK_ACQUIRED=0 _MAESTRO_WRITE_LEASE_RETAIN=0 -export -n MAESTRO_LOCK_TOKEN MAESTRO_LOCK_DIR MAESTRO_LOCK_ACQUIRED 2>/dev/null || : +export -n MAESTRO_LOCK_TOKEN MAESTRO_LOCK_DIR MAESTRO_LOCK_IDENTITY \ + MAESTRO_LOCK_ACQUIRED 2>/dev/null || : repo_digest() { local inside worktrees roots root_list digest material tracked untracked paths entries local regular_paths hashes link_output link_rc path type mode contents worktree nested_list candidate nested_top @@ -306,7 +308,7 @@ provenance_log_append() { # path record write_lock_metadata_value() { local metadata="$1" field="$2" - sed -n "s/^${field}=//p" "$metadata" 2>/dev/null | head -1 + sed -n "s/^${field}=//p" "$metadata" 19>&- 2>/dev/null | head -1 19>&- } write_lock_remove_publication_temps() { # lock-dir @@ -314,36 +316,64 @@ write_lock_remove_publication_temps() { # lock-dir rm -f "$lock_dir"/metadata.tmp.* "$lock_dir"/heartbeat.tmp.* 2>/dev/null } -write_lock_publish_metadata() { # lock_dir token record - local lock_dir="$1" token="$2" record="$3" metadata temp recorded_token +write_lock_unknown_entry() { # lock-dir output-variable + local lock_dir="$1" output_var="$2" entry base + printf -v "$output_var" '%s' "" + for entry in "$lock_dir"/* "$lock_dir"/.[!.]* "$lock_dir"/..?*; do + [ -e "$entry" ] || [ -L "$entry" ] || continue + base=${entry##*/} + case "$base" in + generation|metadata|metadata.new|heartbeat|metadata.tmp.*|heartbeat.tmp.*) continue ;; + esac + printf -v "$output_var" '%s' "$entry" + return 0 + done + return 1 +} + +write_lock_publish_metadata() { # lock_dir identity token record + local lock_dir="$1" identity="$2" token="$3" record="$4" + local metadata temp recorded_token metadata="$lock_dir/metadata" temp="$lock_dir/metadata.tmp.$token" - [ ! -d "$lock_dir/.reclaim" ] || return 1 - if [ -f "$metadata" ]; then - recorded_token=$(write_lock_metadata_value "$metadata" token) - [ "$recorded_token" = "$token" ] || return 1 - fi - printf '%s\n' "$record" > "$temp" || return 1 - if [ -d "$lock_dir/.reclaim" ]; then - rm -f "$temp" 2>/dev/null || : - return 1 - fi + lock_claim_acquire "$lock_dir" "$identity" "$token" initializing || return 1 if [ -f "$metadata" ]; then recorded_token=$(write_lock_metadata_value "$metadata" token) if [ "$recorded_token" != "$token" ]; then - rm -f "$temp" 2>/dev/null || : + lock_claim_release "$lock_dir" "$identity" "$token" || : return 1 fi fi - if mv -f "$temp" "$metadata"; then - return 0 + if ! (umask 077; printf '%s\n' "$record" > "$temp") 19>&- || + ! mv -f "$temp" "$metadata" 19>&-; then + rm -f "$temp" 19>&- 2>/dev/null || : + lock_claim_release "$lock_dir" "$identity" "$token" || : + return 1 fi - rm -f "$temp" 2>/dev/null || : - return 1 + if ! recorded_token=$(lock_claim_metadata_value_once "$metadata" token); then + lock_claim_release "$lock_dir" "$identity" "$token" || : + return 1 + fi + lock_claim_release "$lock_dir" "$identity" "$token" || return 1 + [ "$recorded_token" = "$token" ] +} +write_lock_cleanup_failed_publication() { # lock-dir identity token + local lock_dir="$1" identity="$2" token="$3" current_identity current_token + [ -d "$lock_dir" ] || return 0 + lock_claim_acquire "$lock_dir" "$identity" "$token" initializing || return 1 + current_identity=$(write_lock_path_identity "$lock_dir") || current_identity="" + current_token=$(write_lock_metadata_value "$lock_dir/metadata" token) + if [ "$current_identity" != "$identity" ] || + { [ -n "$current_token" ] && [ "$current_token" != "$token" ]; }; then + lock_claim_release "$lock_dir" "$identity" "$token" || : + return 1 + fi + lock_claim_discard "$lock_dir" "$identity" "$token" } + write_lock_heartbeat_write() { - local interval invalid now last lock_dir heartbeat temp + local interval invalid now last lock_dir heartbeat temp identity token generation write_lock_is_owner || return 0 interval=${MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC:-20} invalid=0 @@ -366,31 +396,37 @@ write_lock_heartbeat_write() { return 0 fi lock_dir="${MAESTRO_LOCK_DIR:-$(write_lock_path)}" + identity=${MAESTRO_LOCK_IDENTITY:-} + token=${MAESTRO_LOCK_TOKEN:-} + lock_claim_acquire "$lock_dir" "$identity" "$token" || return 0 + generation=$(lock_claim_identity_generation "$identity") || { + lock_claim_release "$lock_dir" "$identity" "$token" || : + return 1 + } heartbeat="$lock_dir/heartbeat" - temp="$lock_dir/heartbeat.tmp.$MAESTRO_LOCK_TOKEN" - if ! printf 'token=%s\nepoch=%s\n' "$MAESTRO_LOCK_TOKEN" "$now" > "$temp"; then - rm -f "$temp" 2>/dev/null || : + temp="$lock_dir/heartbeat.tmp.$token" + if ! printf 'token=%s\ngeneration=%s\nepoch=%s\n' \ + "$token" "$generation" "$now" > "$temp" 19>&- || + ! mv -f "$temp" "$heartbeat" 19>&-; then + rm -f "$temp" 19>&- 2>/dev/null || : + lock_claim_release "$lock_dir" "$identity" "$token" || : return 1 fi - if ! write_lock_is_owner; then - rm -f "$temp" 2>/dev/null || : - return 0 - fi - if mv -f "$temp" "$heartbeat"; then - MAESTRO_LOCK_HEARTBEAT_LAST_WRITE_EPOCH=$now - MAESTRO_LOCK_HEARTBEAT_LAST_TOKEN=$MAESTRO_LOCK_TOKEN - return 0 + if ! lock_claim_release "$lock_dir" "$identity" "$token"; then + return 1 fi - rm -f "$temp" 2>/dev/null || : - return 1 + MAESTRO_LOCK_HEARTBEAT_LAST_WRITE_EPOCH=$now + MAESTRO_LOCK_HEARTBEAT_LAST_TOKEN=$token } -write_lock_heartbeat_epoch() { # lock_dir token - local lock_dir="$1" token="$2" heartbeat recorded_token +write_lock_heartbeat_epoch() { # lock_dir token identity + local lock_dir="$1" token="$2" identity="${3-}" heartbeat recorded_token heartbeat="$lock_dir/heartbeat" - [ -f "$heartbeat" ] || return 0 - recorded_token=$(write_lock_metadata_value "$heartbeat" token) + [ -f "$heartbeat" ] && [ ! -L "$heartbeat" ] || return 0 + recorded_token=$(lock_claim_metadata_value_once "$heartbeat" token) || return 0 [ "$recorded_token" = "$token" ] || return 0 + [ -z "$identity" ] || + lock_claim_record_matches_identity "$heartbeat" "$identity" || return 0 write_lock_metadata_value "$heartbeat" epoch } @@ -417,8 +453,13 @@ write_lock_process_start() { # pid } write_lock_path_mtime_epoch() { # path - stat -f '%m' "$1" 2>/dev/null || stat -c '%Y' "$1" 2>/dev/null + stat -c '%Y' "$1" 19>&- 2>/dev/null || + stat -f '%m' "$1" 19>&- 2>/dev/null } +write_lock_path_identity() { # path + lock_claim_path_identity "$1" +} + write_lock_session_id() { # prints a validated session id, or unknown local session_id="${MAESTRO_SESSION_ID:-}" @@ -439,18 +480,40 @@ write_lock_session_id() { # prints a validated session id, or unknown # ponytail: no ticket files, so no arrival order and nothing to reap — the cap # bounds starvation. Add FIFO only if starvation is actually reproduced. write_lock_wait_tick() { # deadline job session lock_dir - local deadline="$1" job="$2" session="$3" lock_dir="$4" now remaining sleep_for + local deadline="$1" job="$2" session="$3" lock_dir="$4" + local now remaining sleep_for last_log log_interval=30 [ "$deadline" -gt 0 ] 2>/dev/null || return 1 now=$(date +%s) [ "$now" -lt "$deadline" ] || return 1 remaining=$((deadline - now)) sleep_for=$MAESTRO_LOCK_WAIT_POLL_SEC [ "$sleep_for" -le "$remaining" ] || sleep_for=$remaining - progress "MAESTRO_LOCK: waiting for the write lease held by job=$job session=$session — ${remaining}s left before blocking (lock: $lock_dir); arrival order is not guaranteed" + last_log=${MAESTRO_LOCK_WAIT_LAST_LOG_EPOCH:-0} + if [ "$last_log" -eq 0 ] || [ "$((now - last_log))" -ge "$log_interval" ]; then + progress "MAESTRO_LOCK: waiting for the write lease held by job=$job session=$session — ${remaining}s left before blocking (lock: $lock_dir); arrival order is not guaranteed" + MAESTRO_LOCK_WAIT_LAST_LOG_EPOCH=$now + fi sleep "$sleep_for" return 0 } +write_lock_wait_summary() { # wait-start wait-budget + local started="$1" budget="$2" now elapsed + case "$started" in + ''|*[!0-9]*) + printf 'wait_budget=%ss wait_elapsed=unknown' "$budget" + return 0 + ;; + esac + now=$(date +%s) || { + printf 'wait_budget=%ss wait_elapsed=unknown' "$budget" + return 0 + } + elapsed=$((now - started)) + [ "$elapsed" -ge 0 ] || elapsed=0 + printf 'wait_budget=%ss wait_elapsed=%ss' "$budget" "$elapsed" +} + write_lock_poison_gate() { # lock_dir metadata → 11 when poisoned (after printing), else 0 local lock_dir="$1" metadata="$2" local poison_metadata quiescence unconfirmed_job unconfirmed_reason owner_session @@ -477,17 +540,20 @@ write_lock_effective_poison() { # lock-dir metadata output-path output-quiescenc } write_lock_is_owner() { - local lock_dir metadata recorded_token + local lock_dir metadata recorded_token current_identity [ "${MAESTRO_LOCK_ACQUIRED:-0}" -eq 1 ] || { [ -n "${MAESTRO_LOCK_TOKEN:-}" ] || return 1 } + [ -n "${MAESTRO_LOCK_IDENTITY:-}" ] || return 1 lock_dir="${MAESTRO_LOCK_DIR:-$(write_lock_path)}" - [ ! -d "$lock_dir/.reclaim" ] || return 1 + current_identity=$(write_lock_path_identity "$lock_dir") || return 1 + [ "$current_identity" = "$MAESTRO_LOCK_IDENTITY" ] || return 1 metadata="$lock_dir/metadata" - [ -f "$metadata" ] || return 1 - recorded_token=$(write_lock_metadata_value "$metadata" token) + [ -f "$metadata" ] && [ ! -L "$metadata" ] || return 1 + recorded_token=$(lock_claim_metadata_value_once "$metadata" token) || return 1 [ -n "${MAESTRO_LOCK_TOKEN:-}" ] && - [ "$recorded_token" = "$MAESTRO_LOCK_TOKEN" ] + [ "$recorded_token" = "$MAESTRO_LOCK_TOKEN" ] && + lock_claim_record_matches_identity "$metadata" "$MAESTRO_LOCK_IDENTITY" } @@ -518,10 +584,12 @@ write_lock_acquire() { local identity_note owner_session malformed_metadata local writers writers_rc digest_before log_path last prior_job prior_after observed_at local stale_digest_before stale_digest_after stale_released_at session_id started_at - local metadata_record reclaim_dir current_token - local wait_cap wait_poll wait_deadline initializing_grace - local heartbeat_stale heartbeat_epoch heartbeat_effective heartbeat_age heartbeat_note + local metadata_record reclaim_dir current_token generation_identity current_identity generation + local wait_cap wait_poll wait_deadline wait_started_epoch initializing_grace remaining sleep_for + local heartbeat_stale heartbeat_epoch heartbeat_effective heartbeat_age heartbeat_note create_rc MAESTRO_LOCK_ACQUIRED=0 + MAESTRO_LOCK_IDENTITY="" + MAESTRO_LOCK_WAIT_LAST_LOG_EPOCH=0 MAESTRO_LOCK_DIR=$(write_lock_path) || return 3 metadata="$MAESTRO_LOCK_DIR/metadata" @@ -558,12 +626,45 @@ write_lock_acquire() { initializing_grace=0 - if [ "$wait_cap" -gt 0 ]; then wait_deadline=$(( $(date +%s) + wait_cap )); else wait_deadline=0; fi + wait_started_epoch=$(date +%s 2>/dev/null) || wait_started_epoch="" + case "$wait_started_epoch" in + ''|*[!0-9]*) + progress "MAESTRO_LOCK: wait clock unavailable; waiting disabled" + wait_cap=0 + wait_deadline=0 + ;; + *) + if [ "$wait_cap" -gt 0 ]; then + wait_deadline=$((wait_started_epoch + wait_cap)) + else + wait_deadline=0 + fi + ;; + esac + token=$(lock_claim_random_id) || { + progress "MAESTRO_LOCK: lock token generation failed; no lease was created" + return 3 + } + attempt=0 while [ "$attempt" -lt 2 ]; do write_lock_poison_gate "$MAESTRO_LOCK_DIR" "$metadata" || return 11 - if mkdir "$MAESTRO_LOCK_DIR" 2>/dev/null; then - token=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n') + create_rc=0 + generation_identity="" + lock_claim_create "$MAESTRO_LOCK_DIR" generation_identity || create_rc=$? + if [ "$create_rc" -eq 2 ]; then + if write_lock_wait_tick "$wait_deadline" unknown unknown \ + "$MAESTRO_LOCK_DIR"; then + continue + fi + progress "MAESTRO_LOCK: write dispatch blocked by a competing generation claimant ($(write_lock_wait_summary "$wait_started_epoch" "$wait_cap"); lock: $MAESTRO_LOCK_DIR)" + return 11 + fi + if [ "$create_rc" -eq 3 ]; then + progress "MAESTRO_LOCK: lease generation initialization failed and the canonical generation could not be retired; retaining fail-closed lock (lock: $MAESTRO_LOCK_DIR)" + return 11 + fi + if [ "$create_rc" -eq 0 ]; then process_start=$(write_lock_process_start "$$") # ponytail: mkdir gives mutual exclusion; ps only sharpens stale-owner recovery. # Without it, record the gap and let the contention path fail closed instead. @@ -574,23 +675,27 @@ write_lock_acquire() { session_id=$(write_lock_session_id) now=$(date +%s) started_at=$(date -u '+%Y-%m-%dT%H:%M:%SZ') - metadata_record=$(printf 'token=%s\npid=%s\nprocess_start=%s\njob_id=%s\nsession_id=%s\nstarted_at=%s\nstarted_epoch=%s\ndigest_before=unavailable' \ - "$token" "$$" "$process_start" "$requested_job" "${session_id:-unknown}" \ + generation=$(lock_claim_identity_generation "$generation_identity") || return 11 + metadata_record=$(printf 'token=%s\ngeneration=%s\npid=%s\nprocess_start=%s\njob_id=%s\nsession_id=%s\nstarted_at=%s\nstarted_epoch=%s\ndigest_before=unavailable' \ + "$token" "$generation" "$$" "$process_start" "$requested_job" "${session_id:-unknown}" \ "$started_at" "$now") - if ! write_lock_publish_metadata "$MAESTRO_LOCK_DIR" "$token" "$metadata_record"; then - rm -f "$metadata" "$MAESTRO_LOCK_DIR/heartbeat" 2>/dev/null || : - write_lock_remove_publication_temps "$MAESTRO_LOCK_DIR" || : - rmdir "$MAESTRO_LOCK_DIR" 2>/dev/null || : + if ! write_lock_publish_metadata "$MAESTRO_LOCK_DIR" "$generation_identity" "$token" "$metadata_record"; then + if ! write_lock_cleanup_failed_publication "$MAESTRO_LOCK_DIR" \ + "$generation_identity" "$token"; then + progress "MAESTRO_LOCK: metadata publication failed and the canonical generation could not be retired; retaining fail-closed lock (lock: $MAESTRO_LOCK_DIR)" + return 11 + fi return 3 fi MAESTRO_LOCK_TOKEN="$token" + MAESTRO_LOCK_IDENTITY="$generation_identity" MAESTRO_LOCK_ACQUIRED=1 digest_before=$(repo_digest_bounded 2>/dev/null) || digest_before=unavailable - metadata_record=$(printf 'token=%s\npid=%s\nprocess_start=%s\njob_id=%s\nsession_id=%s\nstarted_at=%s\nstarted_epoch=%s\ndigest_before=%s' \ - "$token" "$$" "$process_start" "$requested_job" "${session_id:-unknown}" \ + metadata_record=$(printf 'token=%s\ngeneration=%s\npid=%s\nprocess_start=%s\njob_id=%s\nsession_id=%s\nstarted_at=%s\nstarted_epoch=%s\ndigest_before=%s' \ + "$token" "$generation" "$$" "$process_start" "$requested_job" "${session_id:-unknown}" \ "$started_at" "$now" "$digest_before") - if ! write_lock_publish_metadata "$MAESTRO_LOCK_DIR" "$token" "$metadata_record"; then + if ! write_lock_publish_metadata "$MAESTRO_LOCK_DIR" "$generation_identity" "$token" "$metadata_record"; then progress "MAESTRO_LOCK: repository digest could not be recorded; lease retains digest_before=unavailable (lock: $MAESTRO_LOCK_DIR)" fi @@ -619,14 +724,33 @@ write_lock_acquire() { progress "MAESTRO_LOCK: could not create write lock at $MAESTRO_LOCK_DIR" return 3 fi + generation_identity=$(write_lock_path_identity "$MAESTRO_LOCK_DIR") || { + progress "MAESTRO_LOCK: write lease generation identity is unavailable; failing closed (lock: $MAESTRO_LOCK_DIR)" + return 11 + } if [ ! -f "$metadata" ]; then if [ "$wait_cap" -gt 0 ] && [ "$initializing_grace" -lt 3 ]; then - initializing_grace=$((initializing_grace + 1)) - sleep 1 - continue + now=$(date +%s 2>/dev/null) || now="" + case "$now" in + ''|*[!0-9]*) wait_cap=0 ;; + *) + remaining=$((wait_deadline - now)) + if [ "$remaining" -gt 0 ]; then + sleep_for=1 + [ "$sleep_for" -le "$remaining" ] || sleep_for=$remaining + initializing_grace=$((initializing_grace + 1)) + sleep "$sleep_for" + continue + fi + ;; + esac fi - progress "MAESTRO_LOCK: write dispatch blocked by an initializing owner (job=unknown session=unknown pid=unknown held=unknown, lock: $MAESTRO_LOCK_DIR)" + progress "MAESTRO_LOCK: write dispatch blocked by an initializing owner (job=unknown session=unknown pid=unknown held=unknown; $(write_lock_wait_summary "$wait_started_epoch" "$wait_cap"); lock: $MAESTRO_LOCK_DIR)" + return 11 + fi + if ! lock_claim_record_matches_identity "$metadata" "$generation_identity"; then + progress "MAESTRO_LOCK: write lease metadata is malformed or belongs to another generation; owner cannot be identified; failing closed (lock: $MAESTRO_LOCK_DIR)" return 11 fi @@ -672,7 +796,8 @@ write_lock_acquire() { held="${held}s" if [ "$heartbeat_stale" -ne 0 ]; then heartbeat_effective=$started_epoch - heartbeat_epoch=$(write_lock_heartbeat_epoch "$MAESTRO_LOCK_DIR" "$recorded_token") + heartbeat_epoch=$(write_lock_heartbeat_epoch "$MAESTRO_LOCK_DIR" \ + "$recorded_token" "$generation_identity") case "$heartbeat_epoch" in ''|*[!0-9]*) ;; *) @@ -695,7 +820,7 @@ write_lock_acquire() { continue fi fi - progress "MAESTRO_LOCK: write dispatch blocked; held by job=$owner_job session=${owner_session:-unknown} pid=${owner_pid:-unknown} for $held (lock: $MAESTRO_LOCK_DIR)${identity_note}${heartbeat_note}" + progress "MAESTRO_LOCK: write dispatch blocked; held by job=$owner_job session=${owner_session:-unknown} pid=${owner_pid:-unknown} for $held (lease_age=$held; $(write_lock_wait_summary "$wait_started_epoch" "$wait_cap"); lock: $MAESTRO_LOCK_DIR)${identity_note}${heartbeat_note}" return 11 fi @@ -712,14 +837,14 @@ write_lock_acquire() { if write_lock_wait_tick "$wait_deadline" "$owner_job" "${owner_session:-unknown}" "$MAESTRO_LOCK_DIR"; then continue fi - progress "MAESTRO_LOCK: write dispatch blocked; lease retained because orphaned job=$owner_job session=${owner_session:-unknown} is still running (lock: $MAESTRO_LOCK_DIR)" + progress "MAESTRO_LOCK: write dispatch blocked; lease retained because orphaned job=$owner_job session=${owner_session:-unknown} is still running ($(write_lock_wait_summary "$wait_started_epoch" "$wait_cap"); lock: $MAESTRO_LOCK_DIR)" return 11 fi elif printf '%s\n' "$writers" | awk '$2 == "true" { found = 1 } END { exit !found }'; then if write_lock_wait_tick "$wait_deadline" "$owner_job" "${owner_session:-unknown}" "$MAESTRO_LOCK_DIR"; then continue fi - progress "MAESTRO_LOCK: write dispatch blocked; lease retained because an unidentified write-capable job is still running (session=${owner_session:-unknown}, lock: $MAESTRO_LOCK_DIR)" + progress "MAESTRO_LOCK: write dispatch blocked; lease retained because an unidentified write-capable job is still running (session=${owner_session:-unknown}; $(write_lock_wait_summary "$wait_started_epoch" "$wait_cap"); lock: $MAESTRO_LOCK_DIR)" return 11 fi @@ -727,60 +852,55 @@ write_lock_acquire() { # Recheck after liveness so that transition cannot be erased as stale. write_lock_poison_gate "$MAESTRO_LOCK_DIR" "$metadata" || return 11 - # Claim this exact generation before deletion. The second token check closes - # the path-reuse ABA: a delayed reclaimer may observe a successor at the same - # pathname, but it cannot delete that successor using the stale classification. + # Serialize classification and atomically retire only this generation. + # The kernel-backed claim is released automatically if this shell dies. current_token=$(write_lock_metadata_value "$metadata" token) if [ "$current_token" != "$recorded_token" ]; then attempt=$((attempt + 1)) continue fi - reclaim_dir="$MAESTRO_LOCK_DIR/.reclaim" - if ! mkdir "$reclaim_dir" 2>/dev/null; then - progress "MAESTRO_LOCK: write dispatch blocked by a competing stale-lease reclaimer (lock: $MAESTRO_LOCK_DIR)" + if ! lock_claim_acquire "$MAESTRO_LOCK_DIR" \ + "$generation_identity" "$recorded_token"; then + progress "MAESTRO_LOCK: write dispatch blocked by a competing generation claimant ($(write_lock_wait_summary "$wait_started_epoch" "$wait_cap"); lock: $MAESTRO_LOCK_DIR)" return 11 fi - current_token=$(write_lock_metadata_value "$metadata" token) - if [ "$current_token" != "$recorded_token" ]; then - rmdir "$reclaim_dir" 2>/dev/null || : + reclaim_dir=$(lock_claim_path "$MAESTRO_LOCK_DIR") || { + lock_claim_release "$MAESTRO_LOCK_DIR" \ + "$generation_identity" "$recorded_token" || : + return 11 + } + metadata="$reclaim_dir/metadata" + if ! lock_claim_current_path_matches "$generation_identity" "$recorded_token"; then + lock_claim_release "$MAESTRO_LOCK_DIR" \ + "$generation_identity" "$recorded_token" || : attempt=$((attempt + 1)) continue fi - if ! write_lock_poison_gate "$MAESTRO_LOCK_DIR" "$metadata"; then - rmdir "$reclaim_dir" 2>/dev/null || : + if ! write_lock_poison_gate "$reclaim_dir" "$metadata"; then + lock_claim_release "$MAESTRO_LOCK_DIR" \ + "$generation_identity" "$recorded_token" || : return 11 fi - # Observe and publish the adopted baseline while this generation claim still - # blocks a successor. Removing the lock first lets a successor publish and - # release before this older record, reversing baseline order. - stale_digest_after=$(repo_digest_bounded 2>/dev/null) || stale_digest_after=unavailable - stale_released_at=$(date -u '+%Y-%m-%dT%H:%M:%SZ') - if ! rm -f "$metadata" "$MAESTRO_LOCK_DIR/metadata.new" \ - "$MAESTRO_LOCK_DIR/heartbeat" 2>/dev/null; then - rmdir "$reclaim_dir" 2>/dev/null || : - return 11 - fi + stale_digest_after=$(repo_digest_bounded 19>&- 2>/dev/null) || + stale_digest_after=unavailable + stale_released_at=$(date -u '+%Y-%m-%dT%H:%M:%SZ' 19>&-) if repo_digest_is_observed "$stale_digest_before" && repo_digest_is_observed "$stale_digest_after" && [ "$stale_digest_before" != "$stale_digest_after" ]; then progress "PROVENANCE: ADOPTED UNOBSERVED INTERVAL — the tree changed while an orphaned lease was held (job=$owner_job, expected=$stale_digest_before, observed=$stale_digest_after); the interval was not observed and the author is unknown" fi - if log_path=$(provenance_log_path 2>/dev/null); then - # The interval between the orphan's last write and this steal was never observed, - # so the after value is adopted, not witnessed. + if log_path=$(provenance_log_path 19>&- 2>/dev/null); then provenance_log_append "$log_path" "$(printf '%s type=orphan-adopted job=%s session=%s before=%s after=%s' \ "$stale_released_at" "$owner_job" "${owner_session:-unknown}" \ - "$stale_digest_before" "$stale_digest_after")" || : + "$stale_digest_before" "$stale_digest_after")" 19>&- || : fi - write_lock_remove_publication_temps "$MAESTRO_LOCK_DIR" || : - if rmdir "$reclaim_dir" 2>/dev/null && - rmdir "$MAESTRO_LOCK_DIR" 2>/dev/null; then + if lock_claim_discard "$MAESTRO_LOCK_DIR" \ + "$generation_identity" "$recorded_token"; then progress "MAESTRO_LOCK: broke stale write lock held by job=$owner_job session=${owner_session:-unknown} pid=${owner_pid:-unknown}" attempt=$((attempt + 1)) continue fi - rmdir "$reclaim_dir" 2>/dev/null || : case "$started_epoch" in ''|*[!0-9]*) held="unknown" ;; @@ -791,24 +911,26 @@ write_lock_acquire() { held="${held}s" ;; esac - progress "MAESTRO_LOCK: write dispatch blocked; held by job=$owner_job session=${owner_session:-unknown} pid=${owner_pid:-unknown} for $held (lock: $MAESTRO_LOCK_DIR)" + progress "MAESTRO_LOCK: write dispatch blocked; held by job=$owner_job session=${owner_session:-unknown} pid=${owner_pid:-unknown} for $held (lease_age=$held; $(write_lock_wait_summary "$wait_started_epoch" "$wait_cap"); lock: $MAESTRO_LOCK_DIR)" return 11 done - progress "MAESTRO_LOCK: write dispatch blocked by a competing acquirer (job=unknown session=unknown pid=unknown held=unknown, lock: $MAESTRO_LOCK_DIR)" + progress "MAESTRO_LOCK: write dispatch blocked by a competing acquirer (job=unknown session=unknown pid=unknown held=unknown; $(write_lock_wait_summary "$wait_started_epoch" "$wait_cap"); lock: $MAESTRO_LOCK_DIR)" return 11 } write_lock_set_job() { - local job="$1" metadata next_metadata recorded_token owner_pid owner_start started_at started_epoch - local digest_before session_id - [ "${MAESTRO_LOCK_ACQUIRED:-0}" -eq 1 ] || { - [ -n "${MAESTRO_LOCK_TOKEN:-}" ] || return 0 + local job="$1" metadata recorded_token owner_pid owner_start started_at started_epoch + local digest_before session_id identity token generation metadata_record + write_lock_is_owner || { + progress "MAESTRO_LOCK: this lease is no longer held by this process; job update skipped" + return 0 } + identity=${MAESTRO_LOCK_IDENTITY:-} + token=${MAESTRO_LOCK_TOKEN:-} metadata="${MAESTRO_LOCK_DIR:-$(write_lock_path)}/metadata" - [ -f "$metadata" ] || return 0 recorded_token=$(write_lock_metadata_value "$metadata" token) - [ "$recorded_token" = "${MAESTRO_LOCK_TOKEN:-}" ] || { + [ "$recorded_token" = "$token" ] || { progress "MAESTRO_LOCK: this lease is no longer held by this process; job update skipped" return 0 } @@ -820,15 +942,12 @@ write_lock_set_job() { digest_before=${digest_before:-unavailable} session_id=$(write_lock_metadata_value "$metadata" session_id) session_id=$(MAESTRO_SESSION_ID="${session_id:-}" write_lock_session_id) - next_metadata="${MAESTRO_LOCK_DIR}/metadata.new" - if ! printf 'token=%s\npid=%s\nprocess_start=%s\njob_id=%s\nsession_id=%s\nstarted_at=%s\nstarted_epoch=%s\ndigest_before=%s\n' \ - "$recorded_token" "$owner_pid" "$owner_start" "$job" "${session_id:-unknown}" \ - "$started_at" "$started_epoch" "$digest_before" > "$next_metadata"; then - rm -f "$next_metadata" 2>/dev/null || : - return 3 - fi - if ! write_lock_is_owner || ! mv -f "$next_metadata" "$metadata"; then - rm -f "$next_metadata" 2>/dev/null || : + generation=$(lock_claim_identity_generation "$identity") || return 3 + metadata_record=$(printf 'token=%s\ngeneration=%s\npid=%s\nprocess_start=%s\njob_id=%s\nsession_id=%s\nstarted_at=%s\nstarted_epoch=%s\ndigest_before=%s' \ + "$token" "$generation" "$owner_pid" "$owner_start" "$job" "${session_id:-unknown}" \ + "$started_at" "$started_epoch" "$digest_before") + if ! write_lock_publish_metadata "$MAESTRO_LOCK_DIR" "$identity" "$token" \ + "$metadata_record"; then progress "MAESTRO_LOCK: this lease changed while publishing job=$job; job update rejected" return 3 fi @@ -836,18 +955,24 @@ write_lock_set_job() { write_lock_poison() { local job="$1" reason="$2" metadata next_metadata recorded_token owner_pid owner_start - local owner_job started_at started_epoch digest_before session_id + local owner_job started_at started_epoch digest_before session_id identity token generation write_lock_is_owner || { progress "MAESTRO_LOCK: this lease is no longer held by this process; poison not staged" return 3 } - metadata="${MAESTRO_LOCK_DIR:-$(write_lock_path)}/metadata" - [ -f "$metadata" ] || return 3 + identity=${MAESTRO_LOCK_IDENTITY:-} + token=${MAESTRO_LOCK_TOKEN:-} + lock_claim_acquire "$MAESTRO_LOCK_DIR" "$identity" "$token" || { + progress "MAESTRO_LOCK: this lease changed before poison could be staged" + return 3 + } + metadata="$MAESTRO_LOCK_DIR/metadata" recorded_token=$(write_lock_metadata_value "$metadata" token) - [ "$recorded_token" = "${MAESTRO_LOCK_TOKEN:-}" ] || { + if [ "$recorded_token" != "$token" ]; then + lock_claim_release "$MAESTRO_LOCK_DIR" "$identity" "$token" || : progress "MAESTRO_LOCK: this lease is no longer held by this process; poison not staged" return 3 - } + fi owner_pid=$(write_lock_metadata_value "$metadata" pid) owner_start=$(write_lock_metadata_value "$metadata" process_start) owner_job=$(write_lock_metadata_value "$metadata" job_id) @@ -857,28 +982,76 @@ write_lock_poison() { digest_before=${digest_before:-unavailable} session_id=$(write_lock_metadata_value "$metadata" session_id) session_id=$(MAESTRO_SESSION_ID="${session_id:-}" write_lock_session_id) - next_metadata="${MAESTRO_LOCK_DIR}/metadata.new" - if printf 'token=%s\npid=%s\nprocess_start=%s\njob_id=%s\nsession_id=%s\nstarted_at=%s\nstarted_epoch=%s\ndigest_before=%s\nquiescence=unconfirmed\nunconfirmed_job=%s\nunconfirmed_reason=%s\n' \ - "$recorded_token" "$owner_pid" "$owner_start" "$owner_job" "${session_id:-unknown}" \ - "$started_at" "$started_epoch" "$digest_before" "$job" "$reason" > "$next_metadata"; then - return 0 - else - rm -f "$next_metadata" 2>/dev/null || : + generation=$(lock_claim_identity_generation "$identity") || { + lock_claim_release "$MAESTRO_LOCK_DIR" "$identity" "$token" || : + return 3 + } + next_metadata="$MAESTRO_LOCK_DIR/metadata.new" + if ! printf 'token=%s\ngeneration=%s\npid=%s\nprocess_start=%s\njob_id=%s\nsession_id=%s\nstarted_at=%s\nstarted_epoch=%s\ndigest_before=%s\nquiescence=unconfirmed\nunconfirmed_job=%s\nunconfirmed_reason=%s\n' \ + "$token" "$generation" "$owner_pid" "$owner_start" "$owner_job" "${session_id:-unknown}" \ + "$started_at" "$started_epoch" "$digest_before" "$job" "$reason" \ + > "$next_metadata" 19>&-; then + rm -f "$next_metadata" 19>&- 2>/dev/null || : + lock_claim_release "$MAESTRO_LOCK_DIR" "$identity" "$token" || : + return 3 + fi + if ! lock_claim_release "$MAESTRO_LOCK_DIR" "$identity" "$token"; then + progress "MAESTRO_LOCK: lease changed after poison staging; retaining fail-closed state" return 3 fi } +write_lock_finalize_poison() { + local lock_dir identity token metadata next_metadata staged_token quiescence + write_lock_is_owner || return 1 + lock_dir=${MAESTRO_LOCK_DIR:-} + identity=${MAESTRO_LOCK_IDENTITY:-} + token=${MAESTRO_LOCK_TOKEN:-} + lock_claim_acquire "$lock_dir" "$identity" "$token" || return 1 + metadata="$lock_dir/metadata" + next_metadata="$lock_dir/metadata.new" + staged_token=$(write_lock_metadata_value "$next_metadata" token) + quiescence=$(write_lock_metadata_value "$next_metadata" quiescence) + if [ "$staged_token" != "$token" ] || [ "$quiescence" != unconfirmed ] || + ! lock_claim_record_matches_identity "$next_metadata" "$identity" || + ! mv -f "$next_metadata" "$metadata" 19>&-; then + lock_claim_release "$lock_dir" "$identity" "$token" || : + return 1 + fi + lock_claim_release "$lock_dir" "$identity" "$token" +} + write_lock_release() { local metadata recorded_token owner_job owner_session writers writers_rc quiescence - local unconfirmed_job unconfirmed_reason poison_metadata - local digest_before digest_after log_path released_at reclaim_dir current_token + local unconfirmed_job unconfirmed_reason poison_metadata claimed current_token + local digest_before digest_after log_path released_at observed_identity current_identity unknown_entry [ "${MAESTRO_LOCK_ACQUIRED:-0}" -eq 1 ] || return 0 metadata="${MAESTRO_LOCK_DIR:-}/metadata" - [ -n "${MAESTRO_LOCK_DIR:-}" ] && [ -f "$metadata" ] || return 0 + if [ -z "${MAESTRO_LOCK_DIR:-}" ] || [ ! -f "$metadata" ]; then + progress "MAESTRO_LOCK: acquired lease metadata is unavailable during release; retaining fail-closed state" + return 11 + fi + observed_identity=${MAESTRO_LOCK_IDENTITY:-} + [ -n "$observed_identity" ] || { + progress "MAESTRO_LOCK: acquired lease identity is unavailable during release; retaining the lock" + return 11 + } + current_identity=$(write_lock_path_identity "$MAESTRO_LOCK_DIR") || { + progress "MAESTRO_LOCK: lease generation identity is unavailable during release; retaining the lock" + return 11 + } + [ "$current_identity" = "$observed_identity" ] || { + progress "MAESTRO_LOCK: lease generation changed before release; releasing nothing" + return 11 + } + lock_claim_record_matches_identity "$metadata" "$observed_identity" || { + progress "MAESTRO_LOCK: lease metadata belongs to another generation during release; retaining the lock" + return 11 + } recorded_token=$(write_lock_metadata_value "$metadata" token) [ "$recorded_token" = "${MAESTRO_LOCK_TOKEN:-}" ] || { progress "MAESTRO_LOCK: this lease is no longer held by this process; releasing nothing" - return 0 + return 11 } owner_job=$(write_lock_metadata_value "$metadata" job_id) owner_job=${owner_job:-unknown} @@ -886,17 +1059,20 @@ write_lock_release() { owner_session=$(MAESTRO_SESSION_ID="${owner_session:-}" write_lock_session_id) write_lock_effective_poison "$MAESTRO_LOCK_DIR" "$metadata" \ poison_metadata quiescence + if [ "$poison_metadata" != "$metadata" ] && + ! lock_claim_record_matches_identity "$poison_metadata" "$observed_identity"; then + progress "MAESTRO_LOCK: staged poison belongs to another generation; retaining the lock" + return 11 + fi if [ "$quiescence" = "unconfirmed" ]; then unconfirmed_job=$(write_lock_metadata_value "$poison_metadata" unconfirmed_job) unconfirmed_reason=$(write_lock_metadata_value "$poison_metadata" unconfirmed_reason) - owner_session=$(write_lock_metadata_value "$poison_metadata" session_id) - owner_session=$(MAESTRO_SESSION_ID="${owner_session:-}" write_lock_session_id) progress "MAESTRO_LOCK: write lease retained because quiescence was never confirmed (job=${unconfirmed_job:-$owner_job} session=${owner_session:-unknown} reason=${unconfirmed_reason:-unknown}, lock: $MAESTRO_LOCK_DIR)" - return 0 + return 11 fi if [ "$_MAESTRO_WRITE_LEASE_RETAIN" -eq 1 ]; then progress "MAESTRO_LOCK: write lease retained because cancellation poison could not be persisted (job=$owner_job session=${owner_session:-unknown}, lock: $MAESTRO_LOCK_DIR)" - return 0 + return 11 fi writers="" @@ -904,72 +1080,88 @@ write_lock_release() { writers_rc=$? if [ "$writers_rc" -eq 4 ]; then progress "MAESTRO_LOCK: job liveness could not be determined; write lease retained (job=$owner_job session=${owner_session:-unknown}, lock: $MAESTRO_LOCK_DIR)" - return 0 + return 11 fi if [ "$owner_job" != "unknown" ]; then if printf '%s\n' "$writers" | awk -v job="$owner_job" '$1 == job { found = 1 } END { exit !found }'; then progress "MAESTRO_LOCK: write lease retained because job $owner_job session=${owner_session:-unknown} is still running; a later dispatch will resolve it (lock: $MAESTRO_LOCK_DIR)" - return 0 + return 11 fi elif printf '%s\n' "$writers" | awk '$2 == "true" { found = 1 } END { exit !found }'; then progress "MAESTRO_LOCK: write lease retained because an unidentified write-capable job is still running; a later dispatch will resolve it (session=${owner_session:-unknown}, lock: $MAESTRO_LOCK_DIR)" - return 0 + return 11 fi digest_before=$(write_lock_metadata_value "$metadata" digest_before) digest_before=${digest_before:-unavailable} digest_after=$(repo_digest_bounded 2>/dev/null) || digest_after=unavailable released_at=$(date -u '+%Y-%m-%dT%H:%M:%SZ') - # Cancellation can poison while this process checks job liveness. write_lock_effective_poison "$MAESTRO_LOCK_DIR" "$metadata" \ poison_metadata quiescence + if [ "$poison_metadata" != "$metadata" ] && + ! lock_claim_record_matches_identity "$poison_metadata" "$observed_identity"; then + progress "MAESTRO_LOCK: staged poison generation changed during release; retaining the lock" + return 11 + fi if [ "$quiescence" = "unconfirmed" ]; then - unconfirmed_job=$(write_lock_metadata_value "$poison_metadata" unconfirmed_job) - unconfirmed_reason=$(write_lock_metadata_value "$poison_metadata" unconfirmed_reason) - owner_session=$(write_lock_metadata_value "$poison_metadata" session_id) - owner_session=$(MAESTRO_SESSION_ID="${owner_session:-}" write_lock_session_id) - progress "MAESTRO_LOCK: write lease retained because quiescence was never confirmed (job=${unconfirmed_job:-$owner_job} session=${owner_session:-unknown} reason=${unconfirmed_reason:-unknown}, lock: $MAESTRO_LOCK_DIR)" - return 0 + progress "MAESTRO_LOCK: write lease retained because cancellation poison arrived during release (lock: $MAESTRO_LOCK_DIR)" + return 11 fi current_token=$(write_lock_metadata_value "$metadata" token) - [ "$current_token" = "${MAESTRO_LOCK_TOKEN:-}" ] || return 0 - reclaim_dir="$MAESTRO_LOCK_DIR/.reclaim" - if ! mkdir "$reclaim_dir" 2>/dev/null; then + if [ "$current_token" != "${MAESTRO_LOCK_TOKEN:-}" ]; then + progress "MAESTRO_LOCK: lease token changed during release; retaining the lock" + return 11 + fi + if ! lock_claim_acquire "$MAESTRO_LOCK_DIR" \ + "$observed_identity" "${MAESTRO_LOCK_TOKEN:-}"; then progress "MAESTRO_LOCK: lease release lost the generation claim; retaining the lock" - return 0 + return 11 fi - current_token=$(write_lock_metadata_value "$metadata" token) - if [ "$current_token" != "${MAESTRO_LOCK_TOKEN:-}" ]; then - rmdir "$reclaim_dir" 2>/dev/null || : + claimed=$(lock_claim_path "$MAESTRO_LOCK_DIR") || { + lock_claim_release "$MAESTRO_LOCK_DIR" \ + "$observed_identity" "${MAESTRO_LOCK_TOKEN:-}" || : + progress "MAESTRO_LOCK: lease release lost its claimed path; retaining the lock" + return 11 + } + metadata="$claimed/metadata" + if ! lock_claim_current_path_matches "$observed_identity" \ + "${MAESTRO_LOCK_TOKEN:-}"; then + lock_claim_release "$MAESTRO_LOCK_DIR" \ + "$observed_identity" "${MAESTRO_LOCK_TOKEN:-}" || : progress "MAESTRO_LOCK: lease generation changed during release; releasing nothing" - return 0 + return 11 + fi + write_lock_effective_poison "$claimed" "$metadata" poison_metadata quiescence 19>&- + if [ "$poison_metadata" != "$metadata" ] && + ! lock_claim_record_matches_identity "$poison_metadata" "$observed_identity"; then + lock_claim_release "$MAESTRO_LOCK_DIR" \ + "$observed_identity" "${MAESTRO_LOCK_TOKEN:-}" || : + progress "MAESTRO_LOCK: staged poison generation changed during release; retaining the lock" + return 11 fi - write_lock_effective_poison "$MAESTRO_LOCK_DIR" "$metadata" \ - poison_metadata quiescence if [ "$quiescence" = "unconfirmed" ]; then - rmdir "$reclaim_dir" 2>/dev/null || : + lock_claim_release "$MAESTRO_LOCK_DIR" \ + "$observed_identity" "${MAESTRO_LOCK_TOKEN:-}" || : progress "MAESTRO_LOCK: write lease retained because cancellation poison arrived during release (lock: $MAESTRO_LOCK_DIR)" - return 0 + return 11 + fi + if write_lock_unknown_entry "$claimed" unknown_entry; then + lock_claim_release "$MAESTRO_LOCK_DIR" \ + "$observed_identity" "${MAESTRO_LOCK_TOKEN:-}" || : + progress "MAESTRO_LOCK: write lease retained because its directory contains an unrecognized entry ($unknown_entry)" + return 11 fi - # Publish the completed baseline while the generation claim still excludes a - # successor. Otherwise the next acquirer can compare against the older record - # and report this authorized dispatch as a false baseline gap. - if log_path=$(provenance_log_path 2>/dev/null); then - # Best-effort diagnostic only, not an enforcement boundary: repository writers can rewrite this log. - # A differing before/after pair is a dispatch-window change with an unknown author: - # lease metadata delimits an interval; it never identifies which process wrote. + if log_path=$(provenance_log_path 19>&- 2>/dev/null); then provenance_log_append "$log_path" "$(printf '%s type=dispatch job=%s session=%s before=%s after=%s' \ - "$released_at" "$owner_job" "${owner_session:-unknown}" "$digest_before" "$digest_after")" || : + "$released_at" "$owner_job" "${owner_session:-unknown}" "$digest_before" "$digest_after")" 19>&- || : fi - if ! rm -f "$metadata" "$MAESTRO_LOCK_DIR/metadata.new" \ - "$MAESTRO_LOCK_DIR/heartbeat" 2>/dev/null || - ! write_lock_remove_publication_temps "$MAESTRO_LOCK_DIR" || - ! rmdir "$reclaim_dir" 2>/dev/null || - ! rmdir "$MAESTRO_LOCK_DIR" 2>/dev/null; then - rmdir "$reclaim_dir" 2>/dev/null || : - return 0 + if ! lock_claim_discard "$MAESTRO_LOCK_DIR" \ + "$observed_identity" "${MAESTRO_LOCK_TOKEN:-}"; then + progress "MAESTRO_LOCK: lease generation could not be removed after release; retaining fail-closed state" + return 11 fi MAESTRO_LOCK_ACQUIRED=0 + MAESTRO_LOCK_IDENTITY="" } # Operator-facing manual check; do not wire this into cleanup. @@ -1016,14 +1208,14 @@ provenance_check() { write_lease_clear() { # result-file evidence-file local result="${1-}" evidence="${2-}" - local lock_path metadata staged_metadata heartbeat orphan stale_heartbeat + local lock_path metadata staged_metadata heartbeat orphan stale_heartbeat heartbeat_missing local lock_mtime now lock_age poisoned_job poisoned_reason poisoned_session local poison_metadata quiescence owner_token owner_job owner_session owner_pid local owner_start started_epoch malformed heartbeat_stale heartbeat_note local heartbeat_effective heartbeat_epoch heartbeat_age current_start - local writers writers_rc running_job clear_token reclaim_dir - local current_poison current_quiescence current_token - local recovery_record recovery_pid recovery_start + local writers writers_rc running_job clear_token reclaim_dir claim_mode + local current_poison current_quiescence current_token generation_claim_token + local recovery_record recovery_pid recovery_start observed_identity current_identity [ -n "$result" ] && [ -n "$evidence" ] && [ "$result" != "$evidence" ] || return 3 : > "$evidence" || return 3 @@ -1038,12 +1230,17 @@ write_lease_clear() { # result-file evidence-file printf 'state=absent\n' > "$result" return 0 fi + observed_identity=$(write_lock_path_identity "$lock_path") || { + progress "MAESTRO_LOCK: refusing to clear — Lease interval identity is unconfirmed (lock: $lock_path)" + return 11 + } metadata="$lock_path/metadata" staged_metadata="$lock_path/metadata.new" heartbeat="$lock_path/heartbeat" orphan=0 stale_heartbeat=0 + heartbeat_missing=0 if [ ! -e "$metadata" ] && [ ! -e "$staged_metadata" ]; then lock_mtime=$(write_lock_path_mtime_epoch "$lock_path") || { progress "MAESTRO_LOCK: refusing to clear — metadata is absent and lock age is unconfirmed (lock: $lock_path)" @@ -1063,6 +1260,17 @@ write_lease_clear() { # result-file evidence-file else write_lock_effective_poison "$lock_path" "$metadata" \ poison_metadata quiescence + if [ ! -f "$metadata" ] || [ -L "$metadata" ] || + ! lock_claim_record_matches_identity "$metadata" "$observed_identity"; then + progress "MAESTRO_LOCK: refusing to clear — Lease interval metadata belongs to another generation (lock: $lock_path)" + return 11 + fi + if [ "$poison_metadata" != "$metadata" ] && + { [ ! -f "$poison_metadata" ] || [ -L "$poison_metadata" ] || + ! lock_claim_record_matches_identity "$poison_metadata" "$observed_identity"; }; then + progress "MAESTRO_LOCK: refusing to clear — staged Lease interval metadata is malformed or belongs to another generation (lock: $lock_path)" + return 11 + fi if [ "$quiescence" != "unconfirmed" ]; then owner_token=$(write_lock_metadata_value "$metadata" token) owner_job=$(write_lock_metadata_value "$metadata" job_id) @@ -1088,9 +1296,12 @@ write_lease_clear() { # result-file evidence-file if [ "$heartbeat_stale" -ne 0 ]; then now=$(date +%s) heartbeat_effective=$started_epoch - heartbeat_epoch=$(write_lock_heartbeat_epoch "$lock_path" "$owner_token") + heartbeat_epoch=$(write_lock_heartbeat_epoch "$lock_path" \ + "$owner_token" "$observed_identity") case "$heartbeat_epoch" in - ''|*[!0-9]*) ;; + ''|*[!0-9]*) + heartbeat_missing=1 + ;; *) [ "$heartbeat_epoch" -gt "$heartbeat_effective" ] && heartbeat_effective=$heartbeat_epoch @@ -1100,6 +1311,8 @@ write_lease_clear() { # result-file evidence-file [ "$heartbeat_age" -lt 0 ] && heartbeat_age=0 if [ "$heartbeat_age" -gt "$heartbeat_stale" ]; then stale_heartbeat=1 + elif [ "$heartbeat_missing" -eq 1 ]; then + heartbeat_note="; metadata-only lease (heartbeat file missing; lease start ${heartbeat_age}s old)" else heartbeat_note="; heartbeat ${heartbeat_age}s old" fi @@ -1170,65 +1383,84 @@ write_lease_clear() { # result-file evidence-file progress "MAESTRO_LOCK: refusing to clear — a write-capable job is still running (${running_job:-unknown}) session=${poisoned_session:-unknown}" return 11 fi - reclaim_dir="$lock_path/.reclaim" - if ! mkdir "$reclaim_dir" 2>/dev/null; then - progress "MAESTRO_LOCK: refusing to clear — Lease interval generation claim is unavailable (lock: $lock_path)" + generation_claim_token=${clear_token:-} + claim_mode=present + if [ -z "$generation_claim_token" ]; then + generation_claim_token="orphan-$observed_identity" + claim_mode=absent + fi + if ! lock_claim_acquire "$lock_path" "$observed_identity" \ + "$generation_claim_token" "$claim_mode"; then + progress "MAESTRO_LOCK: refusing to clear — generation claim is unavailable (lock: $lock_path)" return 11 fi + reclaim_dir=$(lock_claim_path "$lock_path") || { + lock_claim_release "$lock_path" "$observed_identity" \ + "$generation_claim_token" || : + return 11 + } + metadata="$reclaim_dir/metadata" + staged_metadata="$reclaim_dir/metadata.new" if [ "$orphan" -eq 1 ]; then if [ -e "$metadata" ] || [ -e "$staged_metadata" ]; then - rmdir "$reclaim_dir" 2>/dev/null || : + lock_claim_release "$lock_path" "$observed_identity" \ + "$generation_claim_token" || : progress "MAESTRO_LOCK: refusing to clear — metadata appeared while claiming the orphan Lease interval" return 11 fi + current_identity=$(write_lock_path_identity "$reclaim_dir") || current_identity="" + if [ "$current_identity" != "$observed_identity" ]; then + lock_claim_release "$lock_path" "$observed_identity" \ + "$generation_claim_token" || : + progress "MAESTRO_LOCK: refusing to clear — Lease interval generation changed during recovery" + return 11 + fi else - write_lock_effective_poison "$lock_path" "$metadata" \ - current_poison current_quiescence + write_lock_effective_poison "$reclaim_dir" "$metadata" \ + current_poison current_quiescence 19>&- + if [ "$current_poison" != "$metadata" ] && + { [ ! -f "$current_poison" ] || [ -L "$current_poison" ] || + ! lock_claim_record_matches_identity "$current_poison" "$observed_identity"; }; then + lock_claim_release "$lock_path" "$observed_identity" \ + "$generation_claim_token" || : + progress "MAESTRO_LOCK: refusing to clear — Lease interval poison belongs to another generation" + return 11 + fi + current_identity=$(write_lock_path_identity "$reclaim_dir") || current_identity="" current_token=$(write_lock_metadata_value "$current_poison" token) if [ -z "$current_token" ] && [ "$current_poison" != "$metadata" ]; then current_token=$(write_lock_metadata_value "$metadata" token) fi - if [ "$current_token" != "$clear_token" ] || + if [ "$current_identity" != "$observed_identity" ] || + [ "$current_token" != "$clear_token" ] || { [ "$stale_heartbeat" -eq 0 ] && [ "$current_quiescence" != unconfirmed ]; }; then - rmdir "$reclaim_dir" 2>/dev/null || : + lock_claim_release "$lock_path" "$observed_identity" \ + "$generation_claim_token" || : progress "MAESTRO_LOCK: refusing to clear — Lease interval generation changed during recovery" return 11 fi fi if [ "$orphan" -eq 1 ]; then - progress "MAESTRO_LOCK: clearing structurally invalid orphan write lease (lock: $lock_path, removing: every entry under $lock_path, then $lock_path)" - if ! find "$lock_path" -mindepth 1 -maxdepth 1 \ - ! -name .reclaim -exec rm -rf {} + 2>/dev/null || - ! rmdir "$reclaim_dir" 2>/dev/null || - ! write_lock_remove_publication_temps "$lock_path" || - ! rmdir "$lock_path" 2>/dev/null; then - rmdir "$reclaim_dir" 2>/dev/null || : - progress "MAESTRO_LOCK: failed to clear structurally invalid orphan write lease at $lock_path" - return 11 - fi + progress "MAESTRO_LOCK: clearing structurally invalid orphan write lease (lock: $lock_path, removing atomically: every entry in the claimed generation)" + elif [ "$stale_heartbeat" -eq 1 ] && [ "$heartbeat_missing" -eq 1 ]; then + progress "MAESTRO_LOCK: clearing metadata-only write lease after its missing heartbeat exceeded the stale threshold (job=$owner_job session=${owner_session:-unknown} lease_start_age=${heartbeat_age}s, lock: $lock_path)" elif [ "$stale_heartbeat" -eq 1 ]; then progress "MAESTRO_LOCK: clearing a write lease whose heartbeat went stale (job=$owner_job session=${owner_session:-unknown} last_heartbeat=${heartbeat_age}s ago, lock: $lock_path)" - elif [ -e "$staged_metadata" ]; then - progress "MAESTRO_LOCK: clearing poisoned write lease (job=${poisoned_job:-unknown} session=${poisoned_session:-unknown} reason=${poisoned_reason:-unknown}, lock: $lock_path, removing: $staged_metadata)" + elif [ -e "$reclaim_dir/metadata.new" ]; then + progress "MAESTRO_LOCK: clearing poisoned write lease (job=${poisoned_job:-unknown} session=${poisoned_session:-unknown} reason=${poisoned_reason:-unknown}, lock: $lock_path, removing staged entry atomically: $lock_path/metadata.new)" else progress "MAESTRO_LOCK: clearing poisoned write lease (job=${poisoned_job:-unknown} session=${poisoned_session:-unknown} reason=${poisoned_reason:-unknown}, lock: $lock_path)" fi - if [ "$orphan" -eq 0 ]; then - if ! rm -f "$metadata" "$heartbeat" 2>/dev/null || - ! rm -rf "$staged_metadata" 2>/dev/null || - ! write_lock_remove_publication_temps "$lock_path" || - ! rmdir "$reclaim_dir" 2>/dev/null || - ! rmdir "$lock_path" 2>/dev/null; then - rmdir "$reclaim_dir" 2>/dev/null || : - if [ "$stale_heartbeat" -eq 1 ]; then - progress "MAESTRO_LOCK: failed to clear stale-heartbeat write lease at $lock_path session=${owner_session:-unknown}" - else - progress "MAESTRO_LOCK: failed to clear poisoned write lease at $lock_path session=${poisoned_session:-unknown}" - fi - return 11 - fi + if ! lock_claim_discard "$lock_path" "$observed_identity" \ + "$generation_claim_token"; then + progress "MAESTRO_LOCK: failed to atomically remove the claimed write lease generation at $lock_path" + return 11 + fi + if [ -d "$lock_path" ]; then + progress "MAESTRO_LOCK: cleared the requested generation, but a successor already holds the write lease (lock: $lock_path)" + return 11 fi printf 'state=cleared\n' > "$result" return 0 @@ -1268,8 +1500,7 @@ _write_lease_turn_event() { # event job reason result-file evidence-file ;; cancel-end) _MAESTRO_WRITE_LEASE_RETAIN=1 - if [ ! -e "${MAESTRO_LOCK_DIR:-}/metadata.new" ] || - ! mv -f "${MAESTRO_LOCK_DIR}/metadata.new" "${MAESTRO_LOCK_DIR}/metadata" 2>/dev/null; then + if ! write_lock_finalize_poison; then printf 'MAESTRO_LOCK: cancellation poison could not be finalized (job=%s reason=%s)\n' \ "$job" "$reason" >> "$evidence" 2>/dev/null || : return 11 @@ -1292,6 +1523,7 @@ write_lease_begin() { # evidence-file [ -n "$evidence" ] || return 3 MAESTRO_LOCK_TOKEN="" MAESTRO_LOCK_DIR="" + MAESTRO_LOCK_IDENTITY="" MAESTRO_LOCK_ACQUIRED=0 _MAESTRO_WRITE_LEASE_RETAIN=0 write_lock_acquire unknown @@ -1304,6 +1536,7 @@ write_lease_end() { # evidence-file if [ -z "$lock_path" ] || [ ! -d "$lock_path" ]; then MAESTRO_LOCK_TOKEN="" MAESTRO_LOCK_DIR="" + MAESTRO_LOCK_IDENTITY="" MAESTRO_LOCK_ACQUIRED=0 _MAESTRO_WRITE_LEASE_RETAIN=0 fi diff --git a/hooks/lib-write-turn.sh b/hooks/lib-write-turn.sh index 7ec813a..c4d8e78 100755 --- a/hooks/lib-write-turn.sh +++ b/hooks/lib-write-turn.sh @@ -192,14 +192,20 @@ write_turn_run() { # plan-file max-idle poll result-file evidence-file BLOCKED) return 11 ;; FAILED) return 4 ;; *) - printf '%s\n' "IMPLEMENTER_STATE: MISSING (no RESULT line — treat as FAILED and re-dispatch with the output)" >> "$evidence" - return 4 + printf 'IMPLEMENTER_STATE: COMPANION_FAILURE\n' >> "$evidence" + printf '%s\n' "WATCHDOG_BLOCKED: job ${_MAESTRO_WRITE_TURN_JOB:-unknown} returned no valid full-line RESULT record; inspect the result and current diff before a fresh dispatch." >> "$evidence" + return 11 ;; esac ;; 125) reason=$(sed -n 's/^cancel_reason=//p' "$profile" 2>/dev/null | head -1) request=$(sed -n 's/^cancel_request=//p' "$profile" 2>/dev/null | head -1) + if [ "$reason" = launch-response-unparseable ]; then + printf 'IMPLEMENTER_STATE: COMPANION_FAILURE\n' >> "$evidence" + printf 'WATCHDOG_BLOCKED: task launch succeeded without a parseable job id; an untracked writer may still be running. Retained the write lease and companion job lock until repository-global status confirms quiescence.\n' >> "$evidence" + progress "WATCHDOG_POISONED: task launch returned no job id; after confirming no Codex job is running, clear both retained locks with --clear-job-lock and --clear-lease." + fi if [ -z "$request" ] || [ "$request" = unconfirmed ] || [ "$request" = not-attempted ]; then progress "WATCHDOG_POISONED: job ${_MAESTRO_WRITE_TURN_JOB:-unknown} was not confirmed cancelled and may still be running; the write lease is retained and this run is over." @@ -221,10 +227,11 @@ write_turn_run() { # plan-file max-idle poll result-file evidence-file return 3 ;; *) - printf 'WATCHDOG_FAILED: job %s ended failed. Re-dispatch with the failure evidence, or ask the user.\n' \ + printf 'IMPLEMENTER_STATE: COMPANION_FAILURE\n' >> "$evidence" + printf 'WATCHDOG_BLOCKED: job %s ended without a structured implementer result; inspect the companion/process/result evidence before a fresh dispatch.\n' \ "${_MAESTRO_WRITE_TURN_JOB:-unknown}" >> "$evidence" rm -f "$profile" "${profile}.new" - return 4 + return 11 ;; esac } @@ -232,9 +239,9 @@ write_turn_run() { # plan-file max-idle poll result-file evidence-file write_turn_interrupt() { # HUP|INT|TERM result-file evidence-file local signal="${1-}" result="${2-}" evidence="${3-}" rc reason job case "$signal" in - HUP) reason=signal-hup ;; - INT) reason=signal-int ;; - TERM) reason=signal-term ;; + HUP) reason="signal-hup" ;; + INT) reason="signal-int" ;; + TERM) reason="signal-term" ;; *) return 4 ;; esac process_interrupt "$signal" "$evidence" || : diff --git a/rules/orchestrator-implementer.md b/rules/orchestrator-implementer.md index a2ff836..ac09658 100644 --- a/rules/orchestrator-implementer.md +++ b/rules/orchestrator-implementer.md @@ -24,6 +24,7 @@ bash ~/.claude/hooks/implementer-watchdog.sh --file ``` These are the only ways to invoke the implementer. Always run the outer Bash call in the background: foreground Bash stdout/stderr is invisible to the user, while background-task output streams live. This is observed Claude Code behavior, not a preference. Both append the implementer contract (execute-the-plan discipline, structured RESULT lines, no web-search) to every dispatch — never retype it. Always use files: a plan carries exact steps and verification commands, and inlining that into shell quotes corrupts it. Write the plan to `/tmp` (the gate exempts `/tmp`). +`--verify` consumes exactly one argument. Quote the complete shell command (or pass it as one array element); after Bash splits the invocation, the hook cannot distinguish a lost quote from a legitimate option such as `--max-iters 4` inside the verifier. **Prefer the loop.** It is autonomous between iterations: dispatch → parse RESULT → on a DONE claim, re-run your verify command *locally* (a claim is not proof) → on any failure, append the actual failing output to the next dispatch so Codex never repeats an approach blind. It only stops for the four states below — do not babysit it between rounds. @@ -89,7 +90,7 @@ A plan you can't finish writing means the decision isn't formed yet — form it, Every write dispatch is killed at the `MAESTRO_MAX_DISPATCH_SEC` hard ceiling (2400s when unset; read-only defaults to 1200s), and a write-mode kill retains and poisons the lease. An over-budget verification list therefore still costs a recovery round; the larger healthy-write budget is resilience, not permission to duplicate comprehensive verification inside the dispatch. -**Name only fast leaf checks. The loop's `--verify` owns the comprehensive suite.** It runs *after* the dispatch, locally, on your side of the deadline — so a full suite listed in-dispatch is paid for twice, and the second payment is the one that cancels. +**Name only fast leaf checks. The loop's `--verify` owns the comprehensive suite.** It runs *after* the dispatch, locally, from the lease repository root — the caller's shell directory does not affect root-relative paths — on your side of the deadline. A full suite listed in-dispatch is paid for twice, and the second payment is the one that cancels. **Banning a slow suite by name does not ban it.** Suites call suites, and a ban on the wrapper leaves every caller reachable. Measured in this repo: `run.sh:27` runs `detection.sh`, and `detection.sh:90` runs `lease.sh`. A plan that forbade `run.sh` was satisfied, to the letter, by a dispatch that spent nineteen minutes in `detection.sh` — which is the same work by another name. Before writing the Verification section, expand the call graph of every gate you name or forbid, and budget its **cold transitive** cost, not the leaf's. @@ -148,10 +149,10 @@ The loop exits with a machine-readable `LOOP_STATE` (and the underlying Codex ru - **VERIFIED_DONE** (exit 0) — the plan is executed *and* your verify command passed locally. Do not believe it yet — review the diff (below). - **NEEDS_ANSWERS** (exit 10) — answer immediately, without routing to the user, **only** in either invariant-preserving case: grant a mechanically necessary adjacent file when the objective, public behavior, and design stay unchanged; or change venue or substitute an equally strong verifier when the environment blocks the stated verifier. Stop and relay the QUESTIONS block verbatim when an answer would stub or fake verification, weaken or waive a gate, cross a design or security boundary, take an irreversible action, or settle a question of product taste. Those decisions belong to the user; speed is not a reason to take them. For the permitted class, answer by appending the answers to the plan file, and re-run the loop in the same turn without waiting for the user. Report what you answered and why it was inside the authority; do not ask for permission you already have. - The answer round is a fresh loop invocation, and the plan file is the only thing the next run reads. **The loop now persists the stop report itself**: on the `NEEDS_ANSWERS` exit it appends the run's full report — questions and `CONTINUATION:` capsule — to the plan inside a delimited `MAESTRO STOP HISTORY` block — evidence, never scope or authority, recording what a prior run did and asked without widening the plan's file scope or granting permission — so the completed work survives the stop without a manual copy. Do not paste the capsule yourself; it is already there, and a second copy reads as a second instruction. Write only your answers. If the plan file was not writable the loop says so on the progress channel (`LOOP_WARNING`) and still exits 10 — that is the one case where the capsule is lost and you must carry it across by hand. The attempts log remains a `mktemp` that `cleanup()` deletes; it feeds re-dispatch within a run and is deliberately not preserved across one. + The answer round is a fresh loop invocation, and the plan file is the only thing the next run reads. **The loop now persists the stop report itself**: on the `NEEDS_ANSWERS` exit it appends the run's full report — questions and `CONTINUATION:` capsule — to the plan inside a delimited `MAESTRO STOP HISTORY` block — evidence, never scope or authority, recording what a prior run did and asked without widening the plan's file scope or granting permission — so the completed work survives the stop without a manual copy. Do not paste the capsule yourself; it is already there, and a second copy reads as a second instruction. Write only your answers. If the plan file was not writable the loop says so on the progress channel (`LOOP_WARNING`) and still exits 10 — that is the one case where the capsule is lost and you must carry it across by hand. Within a run, a temporary attempts log feeds each retry; at STUCK the loop persists its bounded tail in a delimited `MAESTRO ATTEMPT HISTORY` block before cleanup. Deliberately do not resume the stopped Codex thread: the companion's `--resume` is a plain alias for `--resume-last`, which resolves the newest finished task thread for the workspace. A discussion turn has the same `jobClass: "task"` as implementation, so it can silently bind an answer to the wrong thread. -- **BLOCKED** (exit 11) — missing access, credentials, a destructive step, write-lock contention, or a cancelled write whose quiescence is unconfirmed. Surface it; never improvise around it. Contention here means the wait already ran and the lease was not queueable; it names the holding job, so wait for that job instead of breaking its lock. For an unconfirmed cancellation, first establish that no Codex job is writing, then use the documented `--clear-lease` command — and note it now refuses a healthy lease and clears a structurally invalid one, so a `CLEARED` result genuinely means the lock is gone. -- **STUCK** (exit 12) — the iteration cap hit without verified completion. Never just raise `--max-iters`: read the attempts log, and if the root cause is not obvious, take the evidence to a **debugging discussion** first (hypothesis + actual output; let Codex try to break it) — a duel beats a blind re-plan. Then re-plan around the actual failing output and run the loop again. +- **BLOCKED** (exit 11) — missing access, credentials, a destructive step, write-lock contention, an unconfirmed cancelled writer, or a post-launch companion/process/result-channel failure that produced no structured implementer result. Surface it; never improvise around it. Contention here means the wait already ran and the lease was not queueable; it names the holding job, so wait for that job instead of breaking its lock. For an unconfirmed cancellation, first establish that no Codex job is writing, then use the documented `--clear-lease` command. For `IMPLEMENTER_STATE: COMPANION_FAILURE`, inspect the evidence and current diff before a fresh dispatch; Maestro deliberately does not bill an automatic implementation retry for an infrastructure fault. +- **STUCK** (exit 12) — the iteration cap hit without verified completion. Never just raise `--max-iters`: read the attempts log, which the loop appends to the plan as bounded historical evidence, not scope or authority. If the root cause is not obvious, take the evidence to a **debugging discussion** first (hypothesis + actual output; let Codex try to break it). Then re-plan around the actual failing output and run the loop again. Single-shot watchdog runs must end with one full-line RESULT record. Maestro accepts only defined full-line tokens and, defensively, uses the last valid record rather than a prefix or quoted example. A watchdog cancellation is the exception: it emits `RESULT: BLOCKED`, then `MAESTRO_FINAL: WATCHDOG POISONED rc=125`; treat rc 125 as unconfirmed quiescence, not ordinary RESULT: BLOCKED/11, and clear the retained lease only after proving no writer remains. Handle terminal Codex records as follows: diff --git a/tests/bounded-calls.sh b/tests/bounded-calls.sh index ac13a9c..125ad14 100755 --- a/tests/bounded-calls.sh +++ b/tests/bounded-calls.sh @@ -395,7 +395,7 @@ t14_nonzero_status_and_result_are_lost() ( local output="$TEST_ROOT/nonzero-status.out" calls="$TEST_ROOT/nonzero-status.calls" local call_out="$TEST_ROOT/nonzero-status.stdout" local call_err="$TEST_ROOT/nonzero-status.stderr" - local result="$TEST_ROOT/nonzero-result" rc statuses + local result="$TEST_ROOT/nonzero-result" evidence="$TEST_ROOT/nonzero-result.evidence" rc statuses sleep() { command sleep 0.05; } : > "$calls" MAESTRO_TEST_CALL_LOG="$calls" \ @@ -412,7 +412,7 @@ t14_nonzero_status_and_result_are_lost() ( MAESTRO_TEST_RESULT='RESULT: DONE' MAESTRO_TEST_RESULT_EXIT=1 \ companion_result "$FIXTURE" task-result-exit-aaaaaa : "$result" \ - "$call_out" "$call_err" >/dev/null 2>&1 + "$call_out" "$call_err" "$evidence" >/dev/null 2>&1 rc=$? [ "$rc" -eq 4 ] || { echo "nonzero result rc=$rc want 4"; return 1; } ) @@ -478,6 +478,18 @@ t16_three_segment_job_id_round_trips() ( done ) +t17_repeated_instant_exit_is_reaped() ( + local out="$TEST_ROOT/instant-exit.out" err="$TEST_ROOT/instant-exit.err" + local i rc + for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 \ + 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40; do + process_run_bounded 1 TEST_INSTANT : "$out" "$err" -- bash -c 'exit 0' + rc=$? + [ "$rc" -eq 0 ] || + { echo "instant exit $i rc=$rc want 0"; return 1; } + done +) + check() { local fn="$1" label="$2" detail if detail=$("$fn" 2>&1); then @@ -508,5 +520,6 @@ check t13_cancellation_writes_one_terminal_fact "cancellation produces one calle check t14_nonzero_status_and_result_are_lost "nonzero status and result calls fail closed despite parseable output" check t15_capitalized_terminal_status_completes "capitalized terminal status is normalized once" check t16_three_segment_job_id_round_trips "three-segment job ids round-trip without truncation" +check t17_repeated_instant_exit_is_reaped "repeated instant exits cannot wedge bounded polling" printf '\n=== %d passed, %d failed ===\n' "$PASS" "$FAIL" [ "$FAIL" -eq 0 ] diff --git a/tests/detection.sh b/tests/detection.sh index f9a207d..65534ec 100755 --- a/tests/detection.sh +++ b/tests/detection.sh @@ -44,7 +44,7 @@ mkrepo() { && printf 'ORIGINAL untracked\n' > u.sh \ && git add s.sh && git commit -q -m init ) } -dispatch() { ( cd "$1" && bash "$LOOP" --plan "$D/plan.md" --verify true --max-iters 1 ) 2>&1; } +dispatch() { ( cd "$1" && bash "$LOOP" --plan "$D/plan.md" --verify true --max-iters 1 --max-idle 2 --poll 1 ) 2>&1; } # ---- 1. tracked mutation between two dispatches is reported at the SECOND acquisition mkrepo "$D/r1" @@ -93,8 +93,5 @@ for O in "$O2" "$O3" "$O4"; do [ "$TRAIL" = "0" ] || { echo "VERIFY FAIL(6): $TRAIL lines after sentinel"; exit 1; } done -# ---- 7. regressions -bash "$ROOT/tests/lease.sh" 2>&1 | tail -1 | grep -qE '[1-9][0-9]* passed, 0 failed' || { echo "VERIFY FAIL(7): lease suite"; exit 1; } -bash "$ROOT/tests/shared-git-dir.sh" 2>&1 | tail -1 | grep -q 'VERIFY PASS' || { echo "VERIFY FAIL(7): planJ"; exit 1; } echo "VERIFY PASS: gap detected at acquisition (tracked + untracked content), reported before dispatch, no tautological clean, no false gap, ignored out of scope, sentinel intact" diff --git a/tests/fixtures/fake-companion.mjs b/tests/fixtures/fake-companion.mjs index 47c0b85..ec63226 100644 --- a/tests/fixtures/fake-companion.mjs +++ b/tests/fixtures/fake-companion.mjs @@ -72,7 +72,7 @@ if (command === "task") { `${Date.now()}\t${taskId}\n` ); } - console.log(`Started ${taskId}`); + console.log(process.env.MAESTRO_TEST_TASK_RESPONSE_RAW ?? `Started ${taskId}`); const delay = Number(process.env.MAESTRO_TEST_TASK_DELAY ?? "0"); if (delay > 0) { await new Promise((resolve) => setTimeout(resolve, delay * 1000)); @@ -130,7 +130,7 @@ if (command === "status" && args.at(-1) === "--json") { process.env.MAESTRO_TEST_JOB_PHASE_FILE, process.env.MAESTRO_TEST_JOB_PHASE ?? "completed" ); - const value = { + const job = { id: args[0], status, phase: status === "running" ? "running" : "done", @@ -139,15 +139,30 @@ if (command === "status" && args.at(-1) === "--json") { request: { model: "gpt-5.6-sol", effort: "high" } }; if (process.env.MAESTRO_TEST_LOGFILE) { - value.logFile = process.env.MAESTRO_TEST_LOGFILE; + job.logFile = process.env.MAESTRO_TEST_LOGFILE; } - console.log(JSON.stringify(value)); + console.log(JSON.stringify({ workspaceRoot: process.cwd(), job })); process.exit(Number(process.env.MAESTRO_TEST_JOB_STATUS_EXIT ?? "0")); } if (command === "result") { + if (process.env.MAESTRO_TEST_MUTATE_JOB_LOCK_METADATA) { + const metadata = process.env.MAESTRO_TEST_MUTATE_JOB_LOCK_METADATA; + const value = fs.readFileSync(metadata, "utf8"); + fs.writeFileSync( + metadata, + value.replace(/^token=.*$/m, "token=foreign-release"), + "utf8" + ); + } appendCall(`result ${args.join(" ")}`); - console.log(process.env.MAESTRO_TEST_RESULT ?? "RESULT: DONE"); + const result = process.env.MAESTRO_TEST_RESULT_FILE + ? fs.readFileSync(process.env.MAESTRO_TEST_RESULT_FILE, "utf8") + : process.env.MAESTRO_TEST_RESULT ?? "RESULT: DONE"; + if (process.env.MAESTRO_TEST_RESULT_STDERR) { + console.error(process.env.MAESTRO_TEST_RESULT_STDERR); + } + console.log(result); process.exit(Number(process.env.MAESTRO_TEST_RESULT_EXIT ?? "0")); } diff --git a/tests/job-lock.sh b/tests/job-lock.sh index 4a0bb26..f292ec3 100644 --- a/tests/job-lock.sh +++ b/tests/job-lock.sh @@ -48,10 +48,45 @@ new_repo() { lock_path() { printf '%s/.git/maestro-job-lock' "$1"; } status_empty() { printf '{"running":[],"latestFinished":null}\n' > "$1"; } + +prepare_generation() { # lock [generation] + local lock="$1" generation="${2-}" + mkdir -p "$lock" || return 1 + if [ -z "$generation" ]; then + [ ! -f "$lock/generation" ] || return 0 + generation=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n') || return 1 + fi + printf '%s\n' "$generation" > "$lock/generation" +} + +sync_metadata_generation() { # lock + local lock="$1" generation temp + generation=$(cat "$lock/generation") || return 1 + temp="$lock/metadata.generation" + awk -v generation="$generation" ' + BEGIN { written = 0 } + /^generation=/ { + if (!written) print "generation=" generation + written = 1 + next + } + { print } + END { if (!written) print "generation=" generation } + ' "$lock/metadata" > "$temp" || return 1 + command mv "$temp" "$lock/metadata" +} status_running() { printf '{"running":[{"id":"%s","write":%s}],"latestFinished":null}\n' \ "$2" "$3" > "$1" } +no_job_lock_generation_siblings() { # lock + local lock="$1" candidate + for candidate in "$lock".reclaim.* "$lock".release.*; do + [ ! -e "$candidate" ] || + { echo "companion job lock left a generation sibling: $candidate"; return 1; } + done +} + wait_for() { # seconds command... local limit="$1" i=0 @@ -139,10 +174,13 @@ run_clear() { # repo output phase status-exit write_metadata() { # repo token pid session class [job] local lock repo="$1" token="$2" pid="$3" session="$4" class="$5" job="${6-}" + local generation lock=$(lock_path "$repo") - mkdir -p "$lock" + prepare_generation "$lock" || return 1 + generation=$(cat "$lock/generation") || return 1 { printf 'token=%s\n' "$token" + printf 'generation=%s\n' "$generation" printf 'pid=%s\n' "$pid" printf 'session=%s\n' "$session" printf 'class=%s\n' "$class" @@ -201,7 +239,7 @@ t3_malformed_metadata_fails_closed() ( new_repo "$repo" status_empty "$repo/status.json" lock=$(lock_path "$repo") - mkdir -p "$lock" + prepare_generation "$lock" || return 1 : > "$lock/metadata" ids="$repo/ids"; flag="$repo/terminal"; starts="$repo/starts"; output="$repo/out" printf 'task-malform-aaaaaa\n' > "$ids"; : > "$flag"; : > "$starts" @@ -219,7 +257,7 @@ t4_publication_window_is_not_clearable() ( new_repo "$repo" status_empty "$repo/status.json" lock=$(lock_path "$repo") - mkdir -p "$lock" + prepare_generation "$lock" || return 1 ( cd "$repo" || exit 1 exec env HOME="$HOME_DIR" PATH="$TEST_PATH" \ @@ -233,7 +271,8 @@ t4_publication_window_is_not_clearable() ( ) > "$repo/waiter.out" 2>&1 3>&1 & waiter=$!; PIDS="$PIDS $waiter" sleep 0.5 - kill -0 "$waiter" 2>/dev/null || { echo "young-lock contender did not wait"; return 1; } + kill -0 "$waiter" 2>/dev/null || + { echo "young-lock contender did not wait: $(tr '\n' ' ' < "$repo/waiter.out")"; return 1; } clear_out="$repo/clear.out" run_clear "$repo" "$clear_out" completed 0; clear_rc=$? [ "$clear_rc" -eq 11 ] || { echo "clear rc=$clear_rc want 11: $(tr '\n' ' ' < "$clear_out")"; return 1; } @@ -262,7 +301,7 @@ t5_operator_recovery_is_fail_closed() ( { echo "terminal clear rc=$rc lock=$(test -d "$(lock_path "$repo")" && echo present || echo absent): $(tr '\n' ' ' < "$output")"; return 1; } repo="$TEST_ROOT/t5-malformed"; new_repo "$repo"; status_empty "$repo/status.json" - lock=$(lock_path "$repo"); mkdir -p "$lock"; : > "$lock/metadata" + lock=$(lock_path "$repo"); prepare_generation "$lock" || return 1; : > "$lock/metadata" touch -t 202001010000 "$lock/metadata" output="$repo/clear.out" run_clear "$repo" "$output" completed 0; rc=$? @@ -314,6 +353,572 @@ t6_read_and_write_turns_serialize() ( [ ! -d "$(lock_path "$repo")" ] || { echo "lock survived serialized turns"; return 1; } ) +t7_dead_holder_uses_global_quiescence() ( + local repo="$TEST_ROOT/t7" lock output pid + new_repo "$repo" + lock=$(lock_path "$repo") + "$REAL_NODE" -e ' + const fs = require("node:fs"); + fs.writeFileSync(process.argv[1], JSON.stringify({ + running: [], + latestFinished: null, + padding: " ".repeat(256 * 1024) + })); + ' "$repo/status.json" || return 1 + write_metadata "$repo" dead-holder 999999 session-dead-holder write task-dead-holder + output="$repo/clear.out" + set -m + ( + cd "$repo" || exit 1 + exec env HOME="$HOME_DIR" PATH="$TEST_PATH" \ + MAESTRO_TEST_JOB_PHASE=running \ + MAESTRO_TEST_JOB_STATUS_EXIT=1 \ + MAESTRO_TEST_STATUS="$repo/status.json" \ + bash "$LOOP" --clear-job-lock + ) > "$output" 2>&1 3>&1 & + pid=$! + PIDS="$PIDS $pid" + set +m + wait_turn "$pid" 4 + [ "$TURN_RC" -eq 0 ] || + { echo "dead-holder clear rc=$TURN_RC want 0: $(tr '\n' ' ' < "$output")"; return 1; } + [ ! -d "$lock" ] || + { echo "dead-holder lock survived global quiescence"; return 1; } + grep -q '^MAESTRO_FINAL: LOOP CLEARED rc=0$' "$output" || + { echo "dead-holder clear final missing: $(tr '\n' ' ' < "$output")"; return 1; } +) + +t8_generation_claim_serializes_reclaimers() ( + local repo="$TEST_ROOT/t8" lock state shim real_mv a_out b_out a_pid a_rc b_rc token + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + state="$repo/state" + shim="$repo/shim" + mkdir -p "$state" "$shim" + status_empty "$repo/status.json" + write_metadata "$repo" old-generation 999999 session-old write task-old0000-aaaaaa + real_mv=$(command -v mv) + cat > "$shim/mv" < "$state/before-move" + while [ ! -e "$state/allow-move" ]; do sleep 0.05; done + "$real_mv" "\$@" + rc=\$? + : > "$state/after-move" + while [ ! -e "$state/allow-cleanup" ]; do sleep 0.05; done + exit "\$rc" +fi +exec "$real_mv" "\$@" +EOF + chmod +x "$shim/mv" + a_out="$repo/a.out" + ( + cd "$repo" || exit 1 + exec env HOME="$HOME_DIR" PATH="$shim:$TEST_PATH" \ + MAESTRO_TEST_JOB_PHASE=completed \ + MAESTRO_TEST_JOB_STATUS_EXIT=0 \ + MAESTRO_TEST_STATUS="$repo/status.json" \ + bash "$LOOP" --clear-job-lock + ) > "$a_out" 2>&1 3>&1 & + a_pid=$! + PIDS="$PIDS $a_pid" + wait_for 5 test -e "$state/before-move" || + { echo "first reclaimer did not reach atomic move: $(tr '\n' ' ' < "$a_out")"; return 1; } + + b_out="$repo/b.out" + run_clear "$repo" "$b_out" completed 0; b_rc=$? + [ "$b_rc" -eq 11 ] || + { echo "second reclaimer rc=$b_rc want 11: $(tr '\n' ' ' < "$b_out")"; return 1; } + [ -f "$lock/metadata" ] || + { echo "second reclaimer removed the claimed generation"; return 1; } + + : > "$state/allow-move" + wait_for 5 test -e "$state/after-move" || + { echo "first reclaimer did not atomically move the lock"; return 1; } + write_metadata "$repo" successor 999998 session-successor write task-successor-aaaaaa + : > "$state/allow-cleanup" + wait_turn "$a_pid" 10 + a_rc=$TURN_RC + [ "$a_rc" -eq 11 ] || + { echo "first reclaimer rc=$a_rc want 11 after successor: $(tr '\n' ' ' < "$a_out")"; return 1; } + token=$(sed -n 's/^token=//p' "$lock/metadata") + [ "$token" = successor ] || + { echo "successor generation was removed or changed (token=${token:-missing})"; return 1; } + no_job_lock_generation_siblings "$lock" || return 1 +) + +t9_orphan_identity_fences_successor_publication() ( + local repo="$TEST_ROOT/t9" lock rc result evidence probe_a probe_b + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + prepare_generation "$lock" || return 1 + ( + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + result="$repo/result" + evidence="$repo/evidence" + job_lock_workspace_jobs() { + rm -rf "$lock" + prepare_generation "$lock" || return 1 + touch -t 202001010000 "$lock" + printf -v "$1" '%s' "" + return 0 + } + job_lock_clear "$result" "$evidence" >/dev/null 2>&1 + ) + rc=$? + [ "$rc" -eq 11 ] || { echo "orphan clear rc=$rc want 11"; return 1; } + [ -d "$lock" ] || { echo "successor publication directory was removed"; return 1; } + probe_a=$(mktemp -d "$repo/identity-a.XXXXXX") || return 1 + probe_b=$(mktemp -d "$repo/identity-b.XXXXXX") || return 1 + prepare_generation "$probe_a" aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa || return 1 + prepare_generation "$probe_b" bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb || return 1 + . "$COMPANION_LIB" + [ "$(job_lock_path_identity "$probe_a")" != "$(job_lock_path_identity "$probe_b")" ] || + { echo "directory identity helper does not distinguish generations"; return 1; } + [ ! -e "$lock/metadata" ] || + { echo "successor unexpectedly published metadata"; return 1; } +) +t10_publication_claim_blocks_concurrent_clear() ( + local repo="$TEST_ROOT/t10" lock identity generation record state publisher rc result evidence + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + state="$repo/interleave" + result="$repo/clear.result" + evidence="$repo/clear.evidence" + mkdir -p "$state" + prepare_generation "$lock" || return 1 + . "$COMPANION_LIB" + cd "$repo" || exit 1 + progress_init + identity=$(job_lock_path_identity "$lock") || return 1 + generation=$(cat "$lock/generation") || return 1 + record=$(printf 'token=publisher\ngeneration=%s\npid=999999\nsession=test\nclass=write\nstart=1\njob=task-publisher-aaaaaa' "$generation") + ( + mv() { + command mv "$@" || return + case "$*" in + *"$lock/metadata") + : > "$state/metadata-moved" + while [ ! -e "$state/finish-publication" ]; do sleep 0.05; done + ;; + esac + } + job_lock_publish_metadata "$lock" "$identity" publisher "$record" + ) > "$state/publisher.out" 2>&1 & + publisher=$! + if ! wait_for 5 test -e "$state/metadata-moved"; then + : > "$state/finish-publication" + wait "$publisher" 2>/dev/null || : + echo "publisher did not reach its final metadata check" + return 1 + fi + job_lock_job_state() { printf -v "$2" '%s' completed; } + job_lock_clear "$result" "$evidence" > "$state/clear.out" 2>&1 3>&1 + rc=$? + : > "$state/finish-publication" + wait "$publisher" || { echo "publisher failed after the clearer was refused"; return 1; } + [ "$rc" -eq 11 ] || { echo "concurrent clear rc=$rc want 11"; return 1; } + grep -q 'generation claim is unavailable' "$state/clear.out" || + { echo "clear did not contend on the publisher claim"; return 1; } + [ "$(job_lock_metadata_value "$lock/metadata" token)" = publisher ] || + { echo "publisher lost ownership after concurrent clear"; return 1; } +) +t11_valid_clear_identity_fences_same_token_successor() ( + local repo="$TEST_ROOT/t11" lock old replacement result evidence rc + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + old="$lock.observed" + replacement="$lock.successor" + result="$repo/result" + evidence="$repo/evidence" + write_metadata "$repo" replayed 999999 session-replayed write task-replayed-aaaaaa + ( + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + stat() { printf '7:42\n'; } + job_lock_job_state() { + command mkdir "$replacement" || return 1 + command cp "$lock/metadata" "$replacement/metadata" || return 1 + prepare_generation "$replacement" bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb || return 1 + sync_metadata_generation "$replacement" || return 1 + command mv "$lock" "$old" || return 1 + command mv "$replacement" "$lock" || return 1 + printf -v "$2" '%s' completed + return 0 + } + job_lock_clear "$result" "$evidence" >/dev/null 2>&1 + ) + rc=$? + [ "$rc" -eq 11 ] || { echo "same-token job clear rc=$rc want 11"; return 1; } + [ -f "$lock/generation" ] || + { echo "same-token job-lock successor was removed by clear"; return 1; } + [ "$(sed -n 's/^token=//p' "$lock/metadata")" = replayed ] || + { echo "same-token job-lock successor metadata changed"; return 1; } + no_job_lock_generation_siblings "$lock" || return 1 +) + +t12_release_identity_fences_same_token_successor() ( + local repo="$TEST_ROOT/t12" lock old replacement state token + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + old="$lock.observed" + replacement="$lock.successor" + state="$repo/replaced" + ( + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + stat() { printf '7:42\n'; } + job_lock_acquire write >/dev/null 2>&1 || return 1 + token=$MAESTRO_JOB_LOCK_TOKEN + claim_checks=0 + lock_claim_current_matches() { + local lock_dir="$1" identity="$2" requested_token="$3" + local current_identity current_token + claim_checks=$((claim_checks + 1)) + if [ "$claim_checks" -eq 2 ] && [ ! -e "$state" ]; then + : > "$state" || return 1 + command mkdir "$replacement" || return 1 + command cp "$lock/metadata" "$replacement/metadata" || return 1 + prepare_generation "$replacement" cccccccccccccccccccccccccccccccc || return 1 + sync_metadata_generation "$replacement" || return 1 + command mv "$lock" "$old" || return 1 + command mv "$replacement" "$lock" || return 1 + fi + current_identity=$(job_lock_path_identity "$lock_dir") || return 1 + [ "$current_identity" = "$identity" ] || return 1 + current_token=$(job_lock_metadata_value "$lock_dir/metadata" token) + [ -z "$current_token" ] || [ "$current_token" = "$requested_token" ] + } + job_lock_release + [ -f "$lock/generation" ] || + { echo "same-token job-lock successor was removed by release"; return 1; } + [ "$(job_lock_metadata_value "$lock/metadata" token)" = "$token" ] || + { echo "same-token job-lock successor metadata changed during release"; return 1; } + [ "$MAESTRO_JOB_LOCK_ACQUIRED" -eq 1 ] || + { echo "release dropped local ownership after rejecting a changed generation"; return 1; } + no_job_lock_generation_siblings "$lock" + ) +) +t13_job_publication_fences_acquisition_identity() ( + local repo="$TEST_ROOT/t13" lock old replacement rc + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + old="$lock.acquired" + replacement="$lock.successor" + ( + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + stat() { printf '7:42\n'; } + job_lock_acquire write >/dev/null 2>&1 || return 1 + command mkdir "$replacement" || return 1 + command cp "$lock/metadata" "$replacement/metadata" || return 1 + prepare_generation "$replacement" dddddddddddddddddddddddddddddddd || return 1 + sync_metadata_generation "$replacement" || return 1 + command mv "$lock" "$old" || return 1 + command mv "$replacement" "$lock" || return 1 + job_lock_publish_job task-replayed-aaaaaa >/dev/null 2>&1 + rc=$? + [ "$rc" -eq 3 ] || { echo "same-token job publication rc=$rc want 3"; return 1; } + [ -f "$lock/generation" ] || + { echo "same-token publication successor disappeared"; return 1; } + ! grep -q '^job=' "$lock/metadata" || + { echo "job id was published into a replacement generation"; return 1; } + ) +) + +t14_killed_publisher_claim_is_recoverable() ( + local repo="$TEST_ROOT/t14" lock identity generation record state publisher child result evidence rc + local shim real_mv + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + state="$repo/interleave" + result="$repo/clear.result" + evidence="$repo/clear.evidence" + mkdir -p "$state" + shim="$state/shim" + real_mv=$(command -v mv) || return 1 + mkdir -p "$shim" + cat > "$shim/mv" < "$state/child.pid" + wait "\$!" + ;; +esac +EOF + chmod +x "$shim/mv" + prepare_generation "$lock" || return 1 + . "$COMPANION_LIB" + cd "$repo" || exit 1 + progress_init + identity=$(job_lock_path_identity "$lock") || return 1 + generation=$(cat "$lock/generation") || return 1 + record=$(printf 'token=publisher\ngeneration=%s\npid=999999\nsession=test\nclass=write\nstart=1\njob=task-publisher-aaaaaa' "$generation") + ( + export PATH="$shim:$PATH" + job_lock_publish_metadata "$lock" "$identity" publisher "$record" + ) > "$state/publisher.out" 2>&1 & + publisher=$! + wait_for 5 test -s "$state/child.pid" || + { echo "publisher did not enter its fenced metadata update"; return 1; } + child=$(cat "$state/child.pid") + kill -KILL "$publisher" 2>/dev/null || return 1 + wait "$publisher" 2>/dev/null || : + kill -0 "$child" 2>/dev/null || + { echo "blocked publication child exited before claim recovery"; return 1; } + job_lock_job_state() { printf -v "$2" '%s' completed; } + job_lock_clear "$result" "$evidence" > "$state/clear.out" 2>&1 3>&1 + rc=$? + if [ "$rc" -ne 0 ]; then + kill "$child" 2>/dev/null || : + echo "clear could not recover killed publisher claim: $(tr '\n' ' ' < "$state/clear.out")" + return 1 + fi + [ ! -d "$lock" ] || + { echo "killed publisher generation survived operator clear"; return 1; } + kill "$child" 2>/dev/null || : + no_job_lock_generation_siblings "$lock" +) + +t15_platform_backend_is_deterministic() ( + local repo="$TEST_ROOT/t15" lock identity generation record marker platform expected + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + marker="$repo/wrong-backend" + prepare_generation "$lock" || return 1 + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + platform=$(uname -s) + case "$platform" in + Darwin|FreeBSD|NetBSD|OpenBSD) + expected=lockf + flock() { : > "$marker"; return 1; } + ;; + Linux) + expected=flock + lockf() { : > "$marker"; return 1; } + ;; + *) echo "unsupported claim backend platform: $platform"; return 1 ;; + esac + [ "$_MAESTRO_LOCK_CLAIM_BACKEND" = "$expected" ] || + { echo "claim backend=$_MAESTRO_LOCK_CLAIM_BACKEND want $expected"; return 1; } + identity=$(job_lock_path_identity "$lock") || return 1 + generation=$(cat "$lock/generation") || return 1 + record=$(printf 'token=backend\ngeneration=%s\npid=999999\nsession=test\nclass=write\nstart=1' "$generation") + job_lock_publish_metadata "$lock" "$identity" backend "$record" || + { echo "selected claim backend could not publish metadata"; return 1; } + [ ! -e "$marker" ] || + { echo "PATH-only alternate backend was selected"; return 1; } +) + +t16_claim_gate_uses_the_requested_lock_path() ( + local repo_a="$TEST_ROOT/t16-a" repo_b="$TEST_ROOT/t16-b" lock gate expected + new_repo "$repo_a" + new_repo "$repo_b" + repo_a=$(cd "$repo_a" && pwd -P) + repo_b=$(cd "$repo_b" && pwd -P) + lock=$(lock_path "$repo_a") + expected="${lock%/*}/maestro-generation-claim.lock" + cd "$repo_b" || exit 1 + . "$COMPANION_LIB" + gate=$(lock_claim_gate_path "$lock") || return 1 + [ "$gate" = "$expected" ] || + { echo "claim gate followed cwd: got=$gate want=$expected"; return 1; } +) + +t17_acquisition_waits_for_the_generation_gate() ( + local repo="$TEST_ROOT/t17" lock gate state holder acquirer rc + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + state="$repo/gate-state" + mkdir -p "$state" + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + gate=$(lock_claim_gate_path "$lock") || return 1 + ( + lock_claim_gate_acquire "$gate" || exit 1 + : > "$state/ready" + while [ ! -e "$state/release" ]; do sleep 0.05; done + lock_claim_unlock + ) & + holder=$! + wait_for 5 test -e "$state/ready" || + { echo "generation gate holder did not start"; return 1; } + ( + export MAESTRO_LOCK_WAIT_SEC=3 MAESTRO_LOCK_WAIT_POLL_SEC=1 + job_lock_acquire write + rc=$? + printf '%s\n' "$rc" > "$state/acquire.rc" + if [ "$rc" -eq 0 ]; then job_lock_release; fi + exit "$rc" + ) > "$state/acquire.out" 2>&1 3>&1 & + acquirer=$! + sleep 0.3 + if [ -d "$lock" ]; then + : > "$state/bypassed" + fi + : > "$state/release" + wait "$holder" || return 1 + wait "$acquirer" + rc=$? + [ "$rc" -eq 0 ] || + { echo "acquisition after gate release rc=$rc: $(tr '\n' ' ' < "$state/acquire.out")"; return 1; } + [ ! -e "$state/bypassed" ] || + { echo "acquisition created the lock while the generation gate was held"; return 1; } + [ ! -d "$lock" ] || + { echo "acquisition left a lock generation behind"; return 1; } +) + +t18_generation_gate_preserves_caller_fd19() ( + local repo="$TEST_ROOT/t18" lock output identity generation record + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + output="$repo/caller-fd19" + cd "$repo" || exit 1 + . "$COMPANION_LIB" + exec 19>> "$output" || return 1 + printf 'before\n' >&19 || return 1 + lock_claim_create "$lock" identity || + { echo "could not create generation under caller fd19"; return 1; } + [ -n "$identity" ] || return 1 + generation=$(cat "$lock/generation") || return 1 + record=$(printf 'token=fd19\ngeneration=%s\npid=999999\nsession=test\nclass=write\nstart=1' "$generation") + job_lock_publish_metadata "$lock" "$identity" fd19 "$record" || + { echo "could not publish metadata under caller fd19"; return 1; } + printf 'after-publish\n' >&19 || + { echo "metadata publication closed caller fd19"; return 1; } + lock_claim_acquire "$lock" "$identity" fd19 || + { echo "could not reacquire generation under caller fd19"; return 1; } + lock_claim_release "$lock" "$identity" fd19 || + { echo "could not release generation claim under caller fd19"; return 1; } + printf 'after-release\n' >&19 || + { echo "claim release closed caller fd19"; return 1; } + exec 19>&- + [ "$(cat "$output")" = "$(printf 'before\nafter-publish\nafter-release')" ] || + { echo "caller fd19 content was not preserved"; return 1; } +) + +t19_failed_publication_retirement_blocks() ( + local repo lock output rc + repo="$TEST_ROOT/t19" + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + output="$repo/acquire.out" + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + mv() { + case "$*" in + *"$lock/metadata.tmp."*"$lock/metadata"|*"$lock $lock.reclaim."*) return 1 ;; + esac + command mv "$@" + } + job_lock_acquire read > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 11 ] || + { echo "failed publication retirement rc=$rc want 11: $(tr '\n' ' ' < "$output")"; return 1; } + [ -d "$lock" ] || + { echo "failed retirement unexpectedly removed the canonical generation"; return 1; } + [ "$MAESTRO_JOB_LOCK_ACQUIRED" -eq 0 ] || + { echo "failed publication granted local ownership"; return 1; } + grep -q 'could not be retired' "$output" || + { echo "failed retirement diagnostic missing"; return 1; } +) + +t20_identity_failure_retirement_blocks() ( + local repo lock output rmdir_called rc + repo="$TEST_ROOT/t20" + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + output="$repo/acquire.out" + rmdir_called="$repo/rmdir.called" + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + lock_claim_path_identity() { return 1; } + rmdir() { : > "$rmdir_called"; return 1; } + job_lock_acquire read > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 11 ] || + { echo "identity failure retirement rc=$rc want 11: $(tr '\n' ' ' < "$output")"; return 1; } + [ -d "$lock" ] || + { echo "identity failure unexpectedly removed the canonical generation"; return 1; } + [ ! -e "$rmdir_called" ] || + { echo "identity failure attempted an unfenced rmdir"; return 1; } + grep -q 'generation initialization failed.*retaining fail-closed lock' "$output" || + { echo "identity failure diagnostic missing"; return 1; } +) + +t21_token_failure_precedes_creation() ( + local repo lock output rc + repo="$TEST_ROOT/t21" + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + output="$repo/acquire.out" + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + od() { return 1; } + job_lock_acquire read > "$output" 2>&1 + rc=$? + [ "$rc" -eq 3 ] || + { echo "token generation failure rc=$rc want 3: $(tr '\n' ' ' < "$output")"; return 1; } + [ ! -e "$lock" ] || + { echo "token generation failure created a canonical lock"; return 1; } +) + +t22_post_move_unlock_failure_does_not_reclassify_release() ( + local repo lock output rc + repo="$TEST_ROOT/t22" + new_repo "$repo" + repo=$(cd "$repo" && pwd -P) + lock=$(lock_path "$repo") + output="$repo/release.out" + cd "$repo" || exit 1 + . "$COMPANION_LIB" + progress_init + job_lock_acquire read >/dev/null 2>&1 || return 1 + lock_claim_unlock() { + exec 19>&- + return 1 + } + job_lock_release > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 0 ] || + { echo "post-move unlock failure rc=$rc want 0: $(tr '\n' ' ' < "$output")"; return 1; } + [ ! -d "$lock" ] || + { echo "post-move unlock failure left the canonical generation"; return 1; } + grep -q 'canonical lock retired; generation gate cleanup failed' "$output" || + { echo "post-move unlock failure diagnostic missing"; return 1; } +) + + + + check() { local fn="$1" label="$2" detail if detail=$("$fn" 2>&1); then ok "$label"; else bad "$label" "${detail:-no detail}"; fi @@ -326,5 +931,21 @@ check t3_malformed_metadata_fails_closed "malformed metadata blocks with recover check t4_publication_window_is_not_clearable "young metadata-less locks remain protected" check t5_operator_recovery_is_fail_closed "operator recovery clears only proven-safe locks" check t6_read_and_write_turns_serialize "read and write companion turns serialize" +check t7_dead_holder_uses_global_quiescence "dead holders clear after bounded global quiescence" +check t9_orphan_identity_fences_successor_publication "orphan identity fence preserves initializing successors" +check t10_publication_claim_blocks_concurrent_clear "metadata publisher excludes a clearer through its final check" +check t15_platform_backend_is_deterministic "claim backend selection is platform-stable" +check t16_claim_gate_uses_the_requested_lock_path "generation gate is derived from the requested lock path" +check t17_acquisition_waits_for_the_generation_gate "lock creation waits for the generation gate" +check t18_generation_gate_preserves_caller_fd19 "generation claims preserve caller fd 19" +check t11_valid_clear_identity_fences_same_token_successor "valid clear fences same-token replacement generations" +check t12_release_identity_fences_same_token_successor "release fences same-token replacement generations" +check t13_job_publication_fences_acquisition_identity "job publication is fenced to the acquired directory identity" +check t14_killed_publisher_claim_is_recoverable "killed publisher claims are recoverable" +check t19_failed_publication_retirement_blocks "failed publication blocks when its generation cannot retire" +check t20_identity_failure_retirement_blocks "identity failure blocks when its generation cannot retire" +check t21_token_failure_precedes_creation "token generation fails before canonical creation" +check t22_post_move_unlock_failure_does_not_reclassify_release "post-move gate cleanup failure does not reclassify release" +check t8_generation_claim_serializes_reclaimers "generation claims serialize reclaimers and preserve successors" printf '\n=== %d passed, %d failed ===\n' "$PASS" "$FAIL" [ "$FAIL" -eq 0 ] diff --git a/tests/lease.sh b/tests/lease.sh index f8f0d7c..40583bb 100755 --- a/tests/lease.sh +++ b/tests/lease.sh @@ -20,6 +20,34 @@ bad() { printf 'FAIL %s — %s\n' "$1" "$2"; FAIL=$((FAIL+1)); } ws() { local d="$TEST_ROOT/$1"; mkdir -p "$d"; printf '%s' "$d"; } +prepare_generation() { # lock [generation] + local lock="$1" generation="${2-}" + mkdir -p "$lock" || return 1 + if [ -z "$generation" ]; then + [ ! -f "$lock/generation" ] || return 0 + generation=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n') || return 1 + fi + printf '%s\n' "$generation" > "$lock/generation" +} + +sync_generation_field() { # lock [record-name] + local lock="$1" record="${2:-metadata}" generation file temp + generation=$(cat "$lock/generation") || return 1 + file="$lock/$record" + temp="$file.generation" + awk -v generation="$generation" ' + BEGIN { written = 0 } + /^generation=/ { + if (!written) print "generation=" generation + written = 1 + next + } + { print } + END { if (!written) print "generation=" generation } + ' "$file" > "$temp" || return 1 + command mv "$temp" "$file" +} + without_ps_path() { # dir local bin="$1/no-ps" mkdir -p "$bin" @@ -76,9 +104,11 @@ run_clear_lease() { # dir status stale_sec } dead_lock() { # dir job_id - mkdir -p "$1/.maestro-write.lock" - printf 'token=old\npid=999999\nprocess_start=dead\njob_id=%s\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=1\n' \ - "$2" > "$1/.maestro-write.lock/metadata" + local lock="$1/.maestro-write.lock" generation + prepare_generation "$lock" || return 1 + generation=$(cat "$lock/generation") || return 1 + printf 'token=old\ngeneration=%s\npid=999999\nprocess_start=dead\njob_id=%s\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=1\n' \ + "$generation" "$2" > "$lock/metadata" } # ---------------------------------------------------------------- deep Lease interval interface @@ -303,13 +333,15 @@ t11() ( # ---------------------------------------------------------------- step 12 # A live owner with unconfirmable identity must block rather than be stolen. t12() ( - local dir owner_pid rc; dir=$(ws no_ps_contention) + local dir lock owner_pid rc; dir=$(ws no_ps_contention) + lock="$dir/.maestro-write.lock" status_empty > "$dir/status.json" sleep 30 & owner_pid=$! trap 'kill "$owner_pid" 2>/dev/null || :; wait "$owner_pid" 2>/dev/null || :' EXIT - mkdir -p "$dir/.maestro-write.lock" + prepare_generation "$lock" || return 1 printf 'token=old\npid=%s\nprocess_start=unavailable\njob_id=task-live0000-aaaaaa\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=1\n' \ - "$owner_pid" > "$dir/.maestro-write.lock/metadata" + "$owner_pid" > "$lock/metadata" + sync_generation_field "$lock" || return 1 cd "$dir" || exit 1; . "$LIB"; companion_resolve() { printf '%s' "$FAKE"; }; progress_init export MAESTRO_TEST_STATUS="$dir/status.json" PATH=$(without_ps_path "$dir"); export PATH @@ -445,12 +477,14 @@ t19() ( # ---------------------------------------------------------------- step 20 # A live owner with unconfirmed process identity is never queueable. t20() ( - local dir owner_pid out rc started elapsed; dir=$(ws wait_identity_unconfirmed) + local dir lock owner_pid out rc started elapsed; dir=$(ws wait_identity_unconfirmed) + lock="$dir/.maestro-write.lock" sleep 30 & owner_pid=$! trap 'kill "$owner_pid" 2>/dev/null || :; wait "$owner_pid" 2>/dev/null || :' EXIT - mkdir -p "$dir/.maestro-write.lock" + prepare_generation "$lock" || return 1 printf 'token=old\npid=%s\nprocess_start=unavailable\njob_id=task-unconfirmed-aaaaaa\nsession_id=session-unconfirmed\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=%s\ndigest_before=unavailable\n' \ - "$owner_pid" "$(date +%s)" > "$dir/.maestro-write.lock/metadata" + "$owner_pid" "$(date +%s)" > "$lock/metadata" + sync_generation_field "$lock" || return 1 cd "$dir" || exit 1; . "$LIB"; progress_init export MAESTRO_LOCK_WAIT_SEC=2 MAESTRO_LOCK_WAIT_POLL_SEC=1 started=$(date +%s) @@ -581,7 +615,7 @@ t25() ( t26() ( local dir lock out rc; dir=$(ws malformed_metadata) lock="$dir/.maestro-write.lock" - mkdir -p "$lock" + prepare_generation "$lock" || return 1 : > "$lock/metadata" status_empty > "$dir/status.json" cd "$dir" || exit 1; . "$LIB"; companion_resolve() { printf '%s' "$FAKE"; }; progress_init @@ -645,7 +679,7 @@ t27() ( t28() ( local dir lock out rc; dir=$(ws absent_metadata) lock="$dir/.maestro-write.lock" - mkdir -p "$lock" + prepare_generation "$lock" || return 1 cd "$dir" || exit 1; . "$LIB"; progress_init out=$(write_lock_acquire 3>&1 >/dev/null 2>&1); rc=$? [ "$rc" -eq 11 ] || { echo "rc=$rc want 11"; return 1; } @@ -662,7 +696,7 @@ t28() ( # ---------------------------------------------------------------- step 29 # A current owner's heartbeat is visible as fresh and never changes its token. t29() ( - local dir lock heartbeat owner_token recorded_token out rc; dir=$(ws heartbeat_fresh) + local dir lock heartbeat owner_token recorded_token generation out rc; dir=$(ws heartbeat_fresh) cd "$dir" || exit 1; . "$LIB"; progress_init PATH=$(confirmed_ps_path "$dir"); export PATH export MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC=1 @@ -673,11 +707,15 @@ t29() ( heartbeat="$lock/heartbeat" owner_token=$MAESTRO_LOCK_TOKEN write_lock_heartbeat_write || { echo "heartbeat write failed"; return 1; } - [ "$(wc -l < "$heartbeat")" -eq 2 ] || - { echo "heartbeat did not contain exactly two lines"; return 1; } + [ "$(wc -l < "$heartbeat")" -eq 3 ] || + { echo "heartbeat did not contain exactly three lines"; return 1; } grep -qx "token=$owner_token" "$heartbeat" || { echo "heartbeat token missing"; return 1; } - write_lock_heartbeat_epoch "$lock" "$owner_token" | grep -Eq '^[0-9]+$' || + generation=$(cat "$lock/generation") || return 1 + grep -qx "generation=$generation" "$heartbeat" || + { echo "heartbeat generation missing"; return 1; } + write_lock_heartbeat_epoch "$lock" "$owner_token" "$MAESTRO_LOCK_IDENTITY" | + grep -Eq '^[0-9]+$' || { echo "heartbeat epoch missing"; return 1; } unset MAESTRO_LOCK_TOKEN out=$(write_lock_acquire 3>&1 >/dev/null 2>&1); rc=$? @@ -794,10 +832,12 @@ t32() ( lock="$dir/.maestro-write.lock" status="$dir/status.json" now=$(date +%s) - mkdir -p "$lock" + prepare_generation "$lock" || return 1 printf 'token=stale-writer\npid=%s\nprocess_start=old\njob_id=task-stale-writer\nsession_id=session-stale-writer\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=%s\ndigest_before=unavailable\n' \ 99999999 "$((now - 5))" > "$lock/metadata" printf 'token=stale-writer\nepoch=%s\n' "$((now - 5))" > "$lock/heartbeat" + sync_generation_field "$lock" || return 1 + sync_generation_field "$lock" heartbeat || return 1 status_running_job task-running-writer true > "$status" out=$(run_clear_lease "$dir" "$status" 1 2>&1); rc=$? [ "$rc" -eq 11 ] || { echo "rc=$rc want 11"; return 1; } @@ -815,10 +855,12 @@ t33() ( lock="$dir/.maestro-write.lock" status="$dir/status.json" now=$(date +%s) - mkdir -p "$lock" + prepare_generation "$lock" || return 1 printf 'token=stale-empty\npid=%s\nprocess_start=old\njob_id=task-stale-empty\nsession_id=session-stale-empty\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=%s\ndigest_before=unavailable\n' \ 99999999 "$((now - 5))" > "$lock/metadata" printf 'token=stale-empty\nepoch=%s\n' "$((now - 5))" > "$lock/heartbeat" + sync_generation_field "$lock" || return 1 + sync_generation_field "$lock" heartbeat || return 1 status_empty > "$status" out=$(run_clear_lease "$dir" "$status" 1 2>&1); rc=$? [ "$rc" -eq 0 ] || { echo "rc=$rc want 0: $out"; return 1; } @@ -838,10 +880,12 @@ t34() ( lock="$dir/.maestro-write.lock" status="$dir/status.json" now=$(date +%s) - mkdir -p "$lock" + prepare_generation "$lock" || return 1 printf 'token=fresh-clear\npid=%s\nprocess_start=current\njob_id=task-fresh-clear\nsession_id=session-fresh-clear\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=%s\ndigest_before=unavailable\n' \ "$$" "$now" > "$lock/metadata" printf 'token=fresh-clear\nepoch=%s\n' "$now" > "$lock/heartbeat" + sync_generation_field "$lock" || return 1 + sync_generation_field "$lock" heartbeat || return 1 status_empty > "$status" out=$(run_clear_lease "$dir" "$status" 10 2>&1); rc=$? [ "$rc" -eq 11 ] || { echo "rc=$rc want 11"; return 1; } @@ -871,10 +915,12 @@ t35() ( } trap disabled_owner_cleanup EXIT now=$(date +%s) - mkdir -p "$lock" + prepare_generation "$lock" || return 1 printf 'token=disabled\npid=%s\nprocess_start=Mon Jan 1 00:00:00 2026\njob_id=task-disabled\nsession_id=session-disabled\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=%s\ndigest_before=unavailable\n' \ "$owner_pid" "$((now - 5))" > "$lock/metadata" printf 'token=disabled\nepoch=%s\n' "$((now - 5))" > "$lock/heartbeat" + sync_generation_field "$lock" || return 1 + sync_generation_field "$lock" heartbeat || return 1 kill -STOP "$owner_pid" || return 1 cd "$dir" || exit 1; . "$LIB"; progress_init export MAESTRO_LOCK_HEARTBEAT_STALE_SEC=0 @@ -899,10 +945,12 @@ t36() ( local dir lock out rc; dir=$(ws heartbeat_unknown_age) lock="$dir/.maestro-write.lock" PATH=$(confirmed_ps_path "$dir"); export PATH - mkdir -p "$lock" + prepare_generation "$lock" || return 1 printf 'token=unknown-age\npid=%s\nprocess_start=Mon Jan 1 00:00:00 2026\njob_id=task-unknown-age\nsession_id=session-unknown-age\nstarted_at=unknown\nstarted_epoch=not-a-number\ndigest_before=unavailable\n' \ "$$" > "$lock/metadata" printf 'token=unknown-age\nepoch=1\n' > "$lock/heartbeat" + sync_generation_field "$lock" || return 1 + sync_generation_field "$lock" heartbeat || return 1 cd "$dir" || exit 1; . "$LIB"; progress_init export MAESTRO_LOCK_HEARTBEAT_STALE_SEC=1 out=$(write_lock_acquire 3>&1 >/dev/null 2>&1); rc=$? @@ -971,10 +1019,12 @@ t39() ( mkdir -p "$ps_shim" printf '#!/bin/sh\nprintf "%%s\\n" %q\n' "$owner_start" > "$ps_shim/ps" chmod +x "$ps_shim/ps" - mkdir -p "$lock" + prepare_generation "$lock" || return 1 printf 'token=stale-live\npid=%s\nprocess_start=%s\njob_id=task-stale-live\nsession_id=session-stale-live\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=%s\ndigest_before=unavailable\n' \ "$$" "$owner_start" "$((now - 5))" > "$lock/metadata" printf 'token=stale-live\nepoch=%s\n' "$((now - 5))" > "$lock/heartbeat" + sync_generation_field "$lock" || return 1 + sync_generation_field "$lock" heartbeat || return 1 status_empty > "$status" out=$(run_clear_lease "$dir" "$status" 1 2>&1); rc=$? [ "$rc" -eq 11 ] || { echo "rc=$rc want 11: $out"; return 1; } @@ -994,9 +1044,11 @@ t40() ( state="$dir/reclaim-state" shim="$state/shim" real_rm=$(command -v rm) - mkdir -p "$lock" "$shim" + mkdir -p "$shim" + prepare_generation "$lock" || return 1 status_empty > "$dir/status.json" printf 'token=old\npid=999999\nprocess_start=dead\njob_id=task-old\nsession_id=session-old\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=1\ndigest_before=unavailable\n' > "$lock/metadata" + sync_generation_field "$lock" || return 1 cat > "$shim/rm" < "$lock/metadata" + sync_generation_field "$lock" || return 1 write_lock_workspace_writers() { printf 'token=second\npid=999999\nprocess_start=dead\njob_id=task-second0-aaaaaa\nsession_id=test\nstarted_epoch=1\nquiescence=unconfirmed\nunconfirmed_job=task-second0-aaaaaa\nunconfirmed_reason=deadline\n' > "$lock/metadata" + sync_generation_field "$lock" || return 1 return 0 } write_lease_clear "$result" "$evidence"; rc=$? @@ -1147,10 +1201,11 @@ t44() ( cd "$dir" || exit 1 . "$LEASE_LIB" progress_init - mkdir "$lock" + prepare_generation "$lock" || return 1 start=$(write_lock_process_start "$$") printf 'token=live\npid=%s\nprocess_start=%s\njob_id=task-live0000-aaaaaa\nsession_id=test\nstarted_epoch=1\nquiescence=unconfirmed\nunconfirmed_job=task-live0000-aaaaaa\nunconfirmed_reason=deadline\n' \ "$$" "${start:-unavailable}" > "$lock/metadata" + sync_generation_field "$lock" || return 1 write_lock_workspace_writers() { return 0; } write_lease_clear "$result" "$evidence"; rc=$? [ "$rc" -eq 11 ] || { echo "live owner clear rc=$rc want 11"; return 1; } @@ -1163,9 +1218,10 @@ t45_publication_temp_does_not_wedge_steal() ( dir=$(ws publication_temps) lock="$dir/.maestro-write.lock" status_empty > "$dir/status.json" - mkdir -p "$lock" + prepare_generation "$lock" || return 1 printf 'token=old\npid=99999999\nprocess_start=dead\njob_id=task-stale-temp-aaaaaa\nsession_id=test\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=1\ndigest_before=unavailable\n' \ > "$lock/metadata" + sync_generation_field "$lock" || return 1 : > "$lock/heartbeat.tmp.deadtoken" cd "$dir" || exit 1 . "$LIB" @@ -1319,6 +1375,613 @@ t50_effective_poison_state_is_shared() ( { echo "staged poison state path=$selected quiescence=$quiescence"; return 1; } ) +# ---------------------------------------------------------------- wait diagnostics +t51_wait_diagnostics_identify_waiter_budget() ( + local dir out rc started_epoch metadata rewritten + dir=$(ws wait_diagnostics) + cd "$dir" || exit 1 + . "$LIB" + progress_init + PATH=$(confirmed_ps_path "$dir"); export PATH + write_lock_acquire task-wait-diagnostic-aaaaaa >/dev/null 2>&1 || return 1 + metadata="$dir/.maestro-write.lock/metadata" + started_epoch=$(date +%s) + rewritten="$metadata.rewritten" + sed "s/^started_epoch=.*/started_epoch=$((started_epoch - 60))/" \ + "$metadata" > "$rewritten" || return 1 + mv -f "$rewritten" "$metadata" || return 1 + unset MAESTRO_LOCK_TOKEN + export MAESTRO_LOCK_WAIT_SEC=2 MAESTRO_LOCK_WAIT_POLL_SEC=1 + out=$(write_lock_acquire 3>&1 >/dev/null 2>&1); rc=$? + [ "$rc" -eq 11 ] || { echo "rc=$rc want 11"; return 1; } + printf '%s\n' "$out" | grep -Eq 'lease_age=6[2-5]s' || + { echo "lease age missing or inaccurate: $out"; return 1; } + printf '%s\n' "$out" | grep -q 'wait_budget=2s' || + { echo "wait budget missing: $out"; return 1; } + printf '%s\n' "$out" | grep -Eq 'wait_elapsed=[2-3]s' || + { echo "wait elapsed missing or inaccurate: $out"; return 1; } +) + +t52_contention_progress_is_throttled() ( + local dir out rc wait_lines + dir=$(ws wait_progress) + cd "$dir" || exit 1 + . "$LIB" + progress_init + PATH=$(confirmed_ps_path "$dir"); export PATH + write_lock_acquire task-wait-progress-aaaaaa >/dev/null 2>&1 || return 1 + unset MAESTRO_LOCK_TOKEN + export MAESTRO_LOCK_WAIT_SEC=3 MAESTRO_LOCK_WAIT_POLL_SEC=1 + out=$(write_lock_acquire 3>&1 >/dev/null 2>&1); rc=$? + [ "$rc" -eq 11 ] || { echo "rc=$rc want 11"; return 1; } + wait_lines=$(printf '%s\n' "$out" | grep -c 'waiting for the write lease' || true) + [ "$wait_lines" -eq 1 ] || + { echo "wait_lines=$wait_lines want 1: $out"; return 1; } +) + +t53_initializing_wait_respects_budget() ( + local dir lock out rc started elapsed + dir=$(ws initializing_wait_budget) + lock="$dir/.maestro-write.lock" + prepare_generation "$lock" || return 1 + cd "$dir" || exit 1 + . "$LIB" + progress_init + export MAESTRO_LOCK_WAIT_SEC=1 MAESTRO_LOCK_WAIT_POLL_SEC=1 + started=$(date +%s) + out=$(write_lock_acquire 3>&1 >/dev/null 2>&1); rc=$? + elapsed=$(( $(date +%s) - started )) + [ "$rc" -eq 11 ] || { echo "rc=$rc want 11"; return 1; } + [ "$elapsed" -le 2 ] || + { echo "initializing-owner elapsed=${elapsed}s exceeded 1s cap: $out"; return 1; } + printf '%s\n' "$out" | grep -Eq 'wait_budget=1s wait_elapsed=[1-2]s' || + { echo "initializing-owner budget missing: $out"; return 1; } +) + +t54_atomic_clear_preserves_successor() ( + local dir lock first_result first_evidence second_result second_evidence + local second_rc_file rc move_rc token candidate + dir=$(ws atomic_clear_successor) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + first_result="$dir/first.result" + first_evidence="$dir/first.evidence" + second_result="$dir/second.result" + second_evidence="$dir/second.evidence" + second_rc_file="$dir/second.rc" + prepare_generation "$lock" || return 1 + printf 'token=old\npid=999999\nprocess_start=dead\njob_id=task-old00000-aaaaaa\nsession_id=test\nstarted_epoch=1\nquiescence=unconfirmed\nunconfirmed_job=task-old00000-aaaaaa\nunconfirmed_reason=deadline\n' > "$lock/metadata" + sync_generation_field "$lock" || return 1 + cd "$dir" || exit 1 + . "$LIB" + progress_init + write_lock_workspace_writers() { + printf -v "$1" '%s' "" + return 0 + } + mv() { + if [ "${1-}" = "$lock" ]; then + write_lease_clear "$second_result" "$second_evidence" >/dev/null 2>&1 + printf '%s\n' "$?" > "$second_rc_file" + command mv "$@" + move_rc=$? + prepare_generation "$lock" eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee || return 1 + printf 'token=successor\npid=999998\nprocess_start=dead\njob_id=task-successor-aaaaaa\nsession_id=test\nstarted_epoch=1\n' > "$lock/metadata" + sync_generation_field "$lock" || return 1 + return "$move_rc" + fi + command mv "$@" + } + write_lease_clear "$first_result" "$first_evidence" >/dev/null 2>&1 + rc=$? + [ "$rc" -eq 11 ] || + { echo "first clear rc=$rc want 11 after successor acquisition"; return 1; } + [ "$(cat "$second_rc_file")" -eq 11 ] || + { echo "second clear bypassed the active generation claim"; return 1; } + token=$(sed -n 's/^token=//p' "$lock/metadata") + [ "$token" = successor ] || + { echo "successor generation was removed or changed (token=${token:-missing})"; return 1; } + for candidate in "$lock".reclaim.clear-*; do + [ ! -e "$candidate" ] || + { echo "atomic clear left a reclaim generation: $candidate"; return 1; } + done +) + +t55_metadata_only_clear_is_reported() ( + local dir lock result evidence out rc + dir=$(ws metadata_only_clear) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + result="$dir/result" + evidence="$dir/evidence" + prepare_generation "$lock" || return 1 + printf 'token=metadata-only\npid=999999\nprocess_start=dead\njob_id=task-metadata-only\nsession_id=test\nstarted_epoch=1\n' > "$lock/metadata" + sync_generation_field "$lock" || return 1 + cd "$dir" || exit 1 + . "$LIB" + progress_init + write_lock_workspace_writers() { + printf -v "$1" '%s' "" + return 0 + } + export MAESTRO_LOCK_HEARTBEAT_STALE_SEC=1 + out=$(write_lease_clear "$result" "$evidence" 3>&1); rc=$? + [ "$rc" -eq 0 ] || { echo "metadata-only clear rc=$rc want 0: $out"; return 1; } + printf '%s\n' "$out" | grep -q 'clearing metadata-only write lease' || + { echo "metadata-only structure missing from diagnostics: $out"; return 1; } + [ ! -d "$lock" ] || { echo "metadata-only generation survived clear"; return 1; } +) + +t56_orphan_identity_fences_successor_publication() ( + local dir lock result evidence rc probe_a probe_b + dir=$(ws orphan_identity_successor) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + result="$dir/result" + evidence="$dir/evidence" + prepare_generation "$lock" || return 1 + touch -t 202001010000 "$lock" + cd "$dir" || exit 1 + . "$LIB" + progress_init + write_lock_workspace_writers() { + rm -rf "$lock" + prepare_generation "$lock" || return 1 + touch -t 202001010000 "$lock" + printf -v "$1" '%s' "" + return 0 + } + write_lease_clear "$result" "$evidence" + rc=$? + [ "$rc" -eq 11 ] || { echo "orphan clear rc=$rc want 11"; return 1; } + [ -d "$lock" ] || { echo "successor publication directory was removed"; return 1; } + probe_a=$(mktemp -d "$dir/identity-a.XXXXXX") || return 1 + probe_b=$(mktemp -d "$dir/identity-b.XXXXXX") || return 1 + prepare_generation "$probe_a" aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa || return 1 + prepare_generation "$probe_b" bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb || return 1 + [ "$(write_lock_path_identity "$probe_a")" != "$(write_lock_path_identity "$probe_b")" ] || + { echo "directory identity helper does not distinguish generations"; return 1; } + [ ! -e "$lock/metadata" ] || + { echo "successor unexpectedly published metadata"; return 1; } +) +t57_publication_claim_blocks_concurrent_clear() ( + local dir lock identity generation record state publisher rc result evidence i + dir=$(ws publication_claim_clear) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + state="$dir/interleave" + result="$dir/clear.result" + evidence="$dir/clear.evidence" + mkdir -p "$state" + prepare_generation "$lock" || return 1 + cd "$dir" || exit 1 + . "$LIB" + progress_init + identity=$(write_lock_path_identity "$lock") || return 1 + generation=$(cat "$lock/generation") || return 1 + record=$(printf 'token=publisher\ngeneration=%s\npid=999999\nprocess_start=dead\njob_id=task-publisher-aaaaaa\nsession_id=test\nstarted_at=2020-01-01T00:00:00Z\nstarted_epoch=1\ndigest_before=unavailable' "$generation") + ( + mv() { + command mv "$@" || return + case "$*" in + *"$lock/metadata") + : > "$state/metadata-moved" + while [ ! -e "$state/finish-publication" ]; do sleep 0.05; done + ;; + esac + } + write_lock_publish_metadata "$lock" "$identity" publisher "$record" + ) > "$state/publisher.out" 2>&1 & + publisher=$! + i=0 + while [ ! -e "$state/metadata-moved" ] && [ "$i" -lt 100 ]; do + sleep 0.05 + i=$((i + 1)) + done + if [ ! -e "$state/metadata-moved" ]; then + : > "$state/finish-publication" + wait "$publisher" 2>/dev/null || : + echo "publisher did not reach its final metadata check" + return 1 + fi + write_lock_workspace_writers() { printf -v "$1" '%s' ""; } + export MAESTRO_LOCK_HEARTBEAT_STALE_SEC=1 + write_lease_clear "$result" "$evidence" > "$state/clear.out" 2>&1 3>&1 + rc=$? + : > "$state/finish-publication" + wait "$publisher" || { echo "publisher failed after the clearer was refused"; return 1; } + [ "$rc" -eq 11 ] || { echo "concurrent clear rc=$rc want 11"; return 1; } + grep -q 'generation claim is unavailable' "$state/clear.out" || + { echo "clear did not contend on the publisher claim"; return 1; } + [ "$(write_lock_metadata_value "$lock/metadata" token)" = publisher ] || + { echo "publisher lost ownership after concurrent clear"; return 1; } +) +t58_reclaimer_diagnostics_include_wait_budget() ( + local dir lock state identity token holder out rc i + dir=$(ws reclaimer_wait_summary) + lock="$dir/.maestro-write.lock" + state="$dir/claim-holder" + dead_lock "$dir" task-reclaimer-aaaaaa + cd "$dir" || exit 1 + . "$LIB" + progress_init + identity=$(write_lock_path_identity "$lock") || return 1 + token=$(write_lock_metadata_value "$lock/metadata" token) + ( + lock_claim_acquire "$lock" "$identity" "$token" || exit 1 + : > "$state.ready" + while [ ! -e "$state.release" ]; do sleep 0.05; done + lock_claim_release "$lock" "$identity" "$token" + ) & + holder=$! + i=0 + while [ ! -e "$state.ready" ] && [ "$i" -lt 100 ]; do + sleep 0.05 + i=$((i + 1)) + done + [ -e "$state.ready" ] || { echo "claim holder did not start"; return 1; } + write_lock_workspace_writers() { + printf -v "$1" '%s' "" + return 0 + } + out=$(write_lock_acquire 3>&1 >/dev/null 2>&1); rc=$? + : > "$state.release" + wait "$holder" || return 1 + [ "$rc" -eq 11 ] || { echo "reclaimer contention rc=$rc want 11"; return 1; } + printf '%s\n' "$out" | grep -q 'wait_budget=0s wait_elapsed=0s' || + { echo "reclaimer contention omitted wait accounting: $out"; return 1; } +) + +t59_valid_clear_identity_fences_same_token_successor() ( + local dir lock metadata old replacement result evidence rc + dir=$(ws valid_clear_same_token) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + metadata="$lock/metadata" + old="$lock.observed" + replacement="$lock.successor" + result="$dir/result" + evidence="$dir/evidence" + prepare_generation "$lock" || return 1 + printf 'token=replayed\npid=999999\nprocess_start=dead\njob_id=task-replayed-aaaaaa\nsession_id=test\nstarted_epoch=1\nquiescence=unconfirmed\nunconfirmed_job=task-replayed-aaaaaa\nunconfirmed_reason=deadline\n' > "$metadata" + sync_generation_field "$lock" || return 1 + cd "$dir" || exit 1 + . "$LIB" + progress_init + stat() { printf '7:42\n'; } + write_lock_workspace_writers() { + command mkdir "$replacement" || return 1 + command cp "$metadata" "$replacement/metadata" || return 1 + prepare_generation "$replacement" bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb || return 1 + sync_generation_field "$replacement" || return 1 + command mv "$lock" "$old" || return 1 + command mv "$replacement" "$lock" || return 1 + printf -v "$1" '%s' "" + return 0 + } + write_lease_clear "$result" "$evidence" >/dev/null 2>&1 + rc=$? + [ "$rc" -eq 11 ] || { echo "same-token clear rc=$rc want 11"; return 1; } + [ -f "$lock/generation" ] || + { echo "same-token successor generation was removed"; return 1; } + [ "$(write_lock_metadata_value "$lock/metadata" token)" = replayed ] || + { echo "same-token successor metadata changed"; return 1; } +) + +t60_release_identity_fences_same_token_successor() ( + local dir lock metadata old replacement evidence token + dir=$(ws release_same_token) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + metadata="$lock/metadata" + old="$lock.observed" + replacement="$lock.successor" + evidence="$dir/evidence" + cd "$dir" || exit 1 + . "$LIB" + progress_init + stat() { printf '7:42\n'; } + write_lock_acquire task-release-aaaaaa >/dev/null 2>&1 || return 1 + token=$MAESTRO_LOCK_TOKEN + write_lock_workspace_writers() { + command mkdir "$replacement" || return 1 + command cp "$metadata" "$replacement/metadata" || return 1 + prepare_generation "$replacement" cccccccccccccccccccccccccccccccc || return 1 + sync_generation_field "$replacement" || return 1 + command mv "$lock" "$old" || return 1 + command mv "$replacement" "$lock" || return 1 + printf -v "$1" '%s' "" + return 0 + } + write_lease_end "$evidence" >/dev/null 2>&1 + [ -f "$lock/generation" ] || + { echo "same-token successor was removed during release"; return 1; } + [ "$(write_lock_metadata_value "$lock/metadata" token)" = "$token" ] || + { echo "same-token successor metadata changed during release"; return 1; } + [ "$MAESTRO_LOCK_ACQUIRED" -eq 1 ] || + { echo "release dropped local ownership after rejecting a changed generation"; return 1; } +) + +t61_killed_publisher_claim_is_recoverable() ( + local dir lock identity generation record state publisher child result evidence rc candidate i + local shim real_mv + dir=$(ws killed_publisher_claim) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + state="$dir/interleave" + result="$dir/clear.result" + evidence="$dir/clear.evidence" + mkdir -p "$state" + shim="$state/shim" + real_mv=$(command -v mv) || return 1 + mkdir -p "$shim" + cat > "$shim/mv" < "$state/child.pid" + wait "\$!" + ;; +esac +EOF + chmod +x "$shim/mv" + prepare_generation "$lock" || return 1 + cd "$dir" || exit 1 + . "$LIB" + progress_init + identity=$(write_lock_path_identity "$lock") || return 1 + generation=$(cat "$lock/generation") || return 1 + record=$(printf 'token=publisher\ngeneration=%s\npid=999999\nprocess_start=dead\njob_id=task-publisher-aaaaaa\nsession_id=test\nstarted_at=2020-01-01T00:00:00Z\nstarted_epoch=1\ndigest_before=unavailable' "$generation") + ( + export PATH="$shim:$PATH" + write_lock_publish_metadata "$lock" "$identity" publisher "$record" + ) > "$state/publisher.out" 2>&1 & + publisher=$! + i=0 + while [ ! -s "$state/child.pid" ] && [ "$i" -lt 100 ]; do + sleep 0.05 + i=$((i + 1)) + done + [ -s "$state/child.pid" ] || + { echo "publisher did not enter its fenced metadata update"; return 1; } + child=$(cat "$state/child.pid") + kill -KILL "$publisher" 2>/dev/null || return 1 + wait "$publisher" 2>/dev/null || : + kill -0 "$child" 2>/dev/null || + { echo "blocked publication child exited before claim recovery"; return 1; } + write_lock_workspace_writers() { printf -v "$1" '%s' ""; } + export MAESTRO_LOCK_HEARTBEAT_STALE_SEC=1 + write_lease_clear "$result" "$evidence" > "$state/clear.out" 2>&1 3>&1 + rc=$? + if [ "$rc" -ne 0 ]; then + kill "$child" 2>/dev/null || : + echo "clear could not recover killed publisher claim: $(tr '\n' ' ' < "$state/clear.out")" + return 1 + fi + [ ! -d "$lock" ] || + { echo "killed publisher generation survived operator clear"; return 1; } + kill "$child" 2>/dev/null || : + for candidate in "$lock".reclaim.*; do + [ ! -e "$candidate" ] || + { echo "killed publisher recovery leaked $candidate"; return 1; } + done +) + +t62_acquisition_waits_for_the_generation_gate() ( + local dir lock gate state holder acquirer rc i + dir=$(ws acquisition_generation_gate) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + state="$dir/gate-state" + mkdir -p "$state" + cd "$dir" || exit 1 + . "$LIB" + progress_init + gate=$(lock_claim_gate_path "$lock") || return 1 + ( + lock_claim_gate_acquire "$gate" || exit 1 + : > "$state/ready" + while [ ! -e "$state/release" ]; do sleep 0.05; done + lock_claim_unlock + ) & + holder=$! + i=0 + while [ ! -e "$state/ready" ] && [ "$i" -lt 100 ]; do + sleep 0.05 + i=$((i + 1)) + done + [ -e "$state/ready" ] || + { echo "generation gate holder did not start"; return 1; } + ( + write_lock_workspace_writers() { printf -v "$1" '%s' ""; } + export MAESTRO_LOCK_WAIT_SEC=3 MAESTRO_LOCK_WAIT_POLL_SEC=1 + write_lock_acquire task-gate-acquirer-aaaaaa + rc=$? + printf '%s\n' "$rc" > "$state/acquire.rc" + if [ "$rc" -eq 0 ]; then write_lock_release; fi + exit "$rc" + ) > "$state/acquire.out" 2>&1 3>&1 & + acquirer=$! + sleep 0.3 + if [ -d "$lock" ]; then + : > "$state/bypassed" + fi + : > "$state/release" + wait "$holder" || return 1 + wait "$acquirer" + rc=$? + [ "$rc" -eq 0 ] || + { echo "acquisition after gate release rc=$rc: $(tr '\n' ' ' < "$state/acquire.out")"; return 1; } + [ ! -e "$state/bypassed" ] || + { echo "acquisition created the write lock while the generation gate was held"; return 1; } + [ ! -d "$lock" ] || + { echo "acquisition left a write lock generation behind"; return 1; } +) + +t63_release_rejects_a_preexisting_same_token_successor() ( + local dir lock retired token successor_identity + dir=$(ws release_preexisting_successor) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + retired="$dir/retired" + cd "$dir" || exit 1 + . "$LIB" + progress_init + write_lock_acquire task-release-owner-aaaaaa >/dev/null 2>&1 || return 1 + token=$MAESTRO_LOCK_TOKEN + command mv "$lock" "$retired" || return 1 + prepare_generation "$lock" eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee || return 1 + cp "$retired/metadata" "$lock/metadata" || return 1 + sync_generation_field "$lock" || return 1 + successor_identity=$(write_lock_path_identity "$lock") || return 1 + write_lock_workspace_writers() { printf -v "$1" '%s' ""; } + write_lock_release + [ -d "$lock" ] || + { echo "release removed a same-token successor present before release"; return 1; } + [ "$(write_lock_path_identity "$lock")" = "$successor_identity" ] || + { echo "release replaced the same-token successor"; return 1; } + [ "$(write_lock_metadata_value "$lock/metadata" token)" = "$token" ] || + { echo "release changed successor metadata"; return 1; } + [ "$MAESTRO_LOCK_ACQUIRED" -eq 1 ] || + { echo "release dropped local ownership after rejecting a successor"; return 1; } +) + +t64_owner_mutations_reject_a_preexisting_same_token_successor() ( + local dir lock retired expected + dir=$(ws owner_mutation_preexisting_successor) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + retired="$dir/retired" + expected="$dir/successor.metadata" + cd "$dir" || exit 1 + . "$LIB" + progress_init + write_lock_acquire task-mutation-owner-aaaaaa >/dev/null 2>&1 || return 1 + command mv "$lock" "$retired" || return 1 + prepare_generation "$lock" ffffffffffffffffffffffffffffffff || return 1 + cp "$retired/metadata" "$lock/metadata" || return 1 + sync_generation_field "$lock" || return 1 + cp "$lock/metadata" "$expected" || return 1 + write_lock_set_job task-wrong-successor-aaaaaa >/dev/null 2>&1 || : + write_lock_poison task-wrong-successor-aaaaaa deadline >/dev/null 2>&1 || : + export MAESTRO_LOCK_HEARTBEAT_LAST_WRITE_EPOCH=0 + export MAESTRO_LOCK_HEARTBEAT_LAST_TOKEN="" + write_lock_heartbeat_write >/dev/null 2>&1 || : + cmp "$expected" "$lock/metadata" || + { echo "owner metadata mutation rewrote a same-token successor"; return 1; } + [ ! -e "$lock/metadata.new" ] || + { echo "owner poison mutation staged data in a same-token successor"; return 1; } + [ ! -e "$lock/heartbeat" ] || + { echo "owner heartbeat mutation wrote into a same-token successor"; return 1; } +) + +t65_poison_finalize_rejects_a_preexisting_same_token_successor() ( + local dir lock retired expected staged result evidence rc + dir=$(ws poison_finalize_preexisting_successor) + dir=$(cd "$dir" && pwd -P) + lock="$dir/.maestro-write.lock" + retired="$dir/retired" + expected="$dir/successor.metadata" + staged="$dir/successor.metadata.new" + result="$dir/result" + evidence="$dir/evidence" + cd "$dir" || exit 1 + . "$LIB" + progress_init + write_lock_acquire task-poison-owner-aaaaaa >/dev/null 2>&1 || return 1 + write_lock_poison task-poison-owner-aaaaaa deadline >/dev/null 2>&1 || return 1 + command mv "$lock" "$retired" || return 1 + prepare_generation "$lock" 11111111111111111111111111111111 || return 1 + cp "$retired/metadata" "$lock/metadata" || return 1 + cp "$retired/metadata.new" "$lock/metadata.new" || return 1 + sync_generation_field "$lock" || return 1 + sync_generation_field "$lock" metadata.new || return 1 + cp "$lock/metadata" "$expected" || return 1 + cp "$lock/metadata.new" "$staged" || return 1 + _write_lease_turn_event cancel-end task-poison-owner-aaaaaa deadline \ + "$result" "$evidence" + rc=$? + [ "$rc" -eq 11 ] || + { echo "poison finalize rc=$rc want 11 after generation replacement"; return 1; } + cmp "$expected" "$lock/metadata" || + { echo "poison finalize rewrote a same-token successor"; return 1; } + cmp "$staged" "$lock/metadata.new" || + { echo "poison finalize consumed successor staging metadata"; return 1; } +) + +t66_failed_publication_retirement_blocks() ( + local dir lock output rc + dir=$(ws failed_publication_retirement) + dir=$(cd "$dir" && pwd -P) + output="$dir/acquire.out" + cd "$dir" || exit 1 + . "$LIB" + progress_init + lock=$(write_lock_path) || return 1 + mv() { + case "$*" in + *"$lock/metadata.tmp."*"$lock/metadata"|*"$lock $lock.reclaim."*) return 1 ;; + esac + command mv "$@" + } + write_lock_acquire task-publication-failure-aaaaaa > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 11 ] || + { echo "failed publication retirement rc=$rc want 11: $(tr '\n' ' ' < "$output")"; return 1; } + [ -d "$lock" ] || + { echo "failed retirement unexpectedly removed the canonical generation"; return 1; } + [ "$MAESTRO_LOCK_ACQUIRED" -eq 0 ] || + { echo "failed publication granted local ownership"; return 1; } + grep -q 'could not be retired' "$output" || + { echo "failed retirement diagnostic missing"; return 1; } +) + +t67_identity_failure_retirement_blocks() ( + local dir lock output rmdir_called rc + dir=$(ws identity_failure_retirement) + dir=$(cd "$dir" && pwd -P) + output="$dir/acquire.out" + rmdir_called="$dir/rmdir.called" + cd "$dir" || exit 1 + . "$LIB" + progress_init + lock=$(write_lock_path) || return 1 + lock_claim_path_identity() { return 1; } + rmdir() { : > "$rmdir_called"; return 1; } + write_lock_acquire task-identity-failure-aaaaaa > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 11 ] || + { echo "identity failure retirement rc=$rc want 11: $(tr '\n' ' ' < "$output")"; return 1; } + [ -d "$lock" ] || + { echo "identity failure unexpectedly removed the canonical generation"; return 1; } + [ ! -e "$rmdir_called" ] || + { echo "identity failure attempted an unfenced rmdir"; return 1; } + grep -q 'generation initialization failed.*retaining fail-closed lock' "$output" || + { echo "identity failure diagnostic missing"; return 1; } +) + +t68_token_failure_precedes_creation() ( + local dir lock output rc + dir=$(ws token_failure_precedes_creation) + dir=$(cd "$dir" && pwd -P) + output="$dir/acquire.out" + cd "$dir" || exit 1 + . "$LIB" + progress_init + lock=$(write_lock_path) || return 1 + od() { return 1; } + write_lock_acquire task-token-failure-aaaaaa > "$output" 2>&1 + rc=$? + [ "$rc" -eq 3 ] || + { echo "token generation failure rc=$rc want 3: $(tr '\n' ' ' < "$output")"; return 1; } + [ ! -e "$lock" ] || + { echo "token generation failure created a canonical lock"; return 1; } +) + + + printf '=== Plan F green-phase verification ===\n' for t in t1 t2 t3 t4 t5 t5b t6 t7 t7b t8 t9 t9b t10a t10b t11 t12 t13 t14 t15 t16 t17 \ t18 t19 t20 t21 t22 t23 t24 t25 t26 t27 t28 t29 t30 t31 t32 t33 t34 t35 t36 t37 t38 t39 t40 t41 t42 t43 t44 \ @@ -1327,7 +1990,22 @@ for t in t1 t2 t3 t4 t5 t5b t6 t7 t7b t8 t9 t9b t10a t10b t11 t12 t13 t14 t15 t1 t47_unknown_lock_entry_is_not_deleted \ t48_prelaunch_interrupt_releases_without_poison \ t49_digest_recurses_through_nested_repositories \ - t50_effective_poison_state_is_shared; do + t50_effective_poison_state_is_shared t51_wait_diagnostics_identify_waiter_budget \ + t52_contention_progress_is_throttled t53_initializing_wait_respects_budget \ + t54_atomic_clear_preserves_successor t55_metadata_only_clear_is_reported \ + t56_orphan_identity_fences_successor_publication \ + t57_publication_claim_blocks_concurrent_clear \ + t58_reclaimer_diagnostics_include_wait_budget \ + t59_valid_clear_identity_fences_same_token_successor \ + t60_release_identity_fences_same_token_successor \ + t61_killed_publisher_claim_is_recoverable \ + t62_acquisition_waits_for_the_generation_gate \ + t63_release_rejects_a_preexisting_same_token_successor \ + t64_owner_mutations_reject_a_preexisting_same_token_successor \ + t65_poison_finalize_rejects_a_preexisting_same_token_successor \ + t66_failed_publication_retirement_blocks \ + t67_identity_failure_retirement_blocks \ + t68_token_failure_precedes_creation; do msg=$($t 2>&1) && ok "$t" || bad "$t" "${msg:-no detail}" done printf '\n=== %d passed, %d failed ===\n' "$PASS" "$FAIL" diff --git a/tests/liveness.sh b/tests/liveness.sh index 3a47a11..82af4b1 100644 --- a/tests/liveness.sh +++ b/tests/liveness.sh @@ -25,6 +25,34 @@ status_running_job() { printf '{\n "running": [\n {\n "id": "%s",\n "write": %s\n }\n ],\n "latestFinished": null\n}\n' "$1" "$2" } +prepare_generation() { # lock [generation] + local lock="$1" generation="${2-}" + mkdir -p "$lock" || return 1 + if [ -z "$generation" ]; then + [ ! -f "$lock/generation" ] || return 0 + generation=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n') || return 1 + fi + printf '%s\n' "$generation" > "$lock/generation" +} + +sync_generation_field() { # lock [record-name] + local lock="$1" record="${2:-metadata}" generation file temp + generation=$(cat "$lock/generation") || return 1 + file="$lock/$record" + temp="$file.generation" + awk -v generation="$generation" ' + BEGIN { written = 0 } + /^generation=/ { + if (!written) print "generation=" generation + written = 1 + next + } + { print } + END { if (!written) print "generation=" generation } + ' "$file" > "$temp" || return 1 + command mv "$temp" "$file" +} + new_repo() { local dir="$TEST_ROOT/$1" git init -q "$dir" @@ -148,7 +176,7 @@ t2_growth_defeats_idle() { t3_write_cancel_poisons() { local metadata="$TEST_ROOT/deadline-repo/.git/maestro-write.lock/metadata" local repo state shim pid lock poison_metadata - local stage_repo stage_state stage_shim stage_lock + local stage_repo stage_state stage_lock stage_attempt local race_state race_shim release_repo release_lock release_poison local acquire_repo acquire_lock acquire_poison race_failed=0 [ -f "$metadata" ] || { echo "poisoned metadata missing"; return 1; } @@ -177,7 +205,7 @@ t3_write_cancel_poisons() { printf ' */metadata.new)\n' printf ' count=$((count + 1))\n' printf ' printf "%%s\\\\n" "$count" > "$state"\n' - printf ' [ "$count" -eq 2 ] && exit 1\n' + printf ' [ "$count" -eq 1 ] && exit 1\n' printf ' ;;\n' printf ' esac\n' printf 'fi\n' @@ -239,31 +267,15 @@ t3_write_cancel_poisons() { stage_repo=$(new_repo poison-stage-failure-repo) stage_state="$TEST_ROOT/poison-stage-failure-state" - stage_shim="$stage_state/shim" - mkdir -p "$stage_shim" + stage_lock="$stage_repo/.git/maestro-write.lock" + mkdir -p "$stage_state" : > "$stage_state/job.log" : > "$stage_state/calls.log" status_empty > "$stage_state/status.json" - { - printf '#!/usr/bin/env bash\n' - printf 'real_mv="%s"\n' "$REAL_MV" - printf 'if [ "${1:-}" = "-f" ]; then\n' - printf ' case "${2:-}" in\n' - printf ' */metadata.new)\n' - printf ' "$real_mv" "$@"\n' - printf ' rc=$?\n' - printf ' [ "$rc" -eq 0 ] && mkdir "$2"\n' - printf ' exit "$rc"\n' - printf ' ;;\n' - printf ' esac\n' - printf 'fi\n' - printf 'exec "$real_mv" "$@"\n' - } > "$stage_shim/mv" - chmod +x "$stage_shim/mv" set -m ( cd "$stage_repo" && - env HOME="$TEST_HOME" PATH="$stage_shim:$TEST_PATH" \ + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ MAESTRO_TEST_CALL_LOG="$stage_state/calls.log" \ MAESTRO_TEST_JOB_PHASE=running \ MAESTRO_TEST_LOGFILE="$stage_state/job.log" \ @@ -275,6 +287,19 @@ t3_write_cancel_poisons() { ) > "$stage_state/output" 2>&1 & pid=$! set +m + stage_attempt=0 + while ! grep -qx 'job_id=task-fake0000-aaaaaa' "$stage_lock/metadata" 2>/dev/null; do + stage_attempt=$((stage_attempt + 1)) + if [ "$stage_attempt" -ge 50 ]; then + kill -TERM -"$pid" 2>/dev/null || : + wait_bounded "$pid" 3 + echo "write job was not published before poison staging setup" + return 1 + fi + sleep 0.05 + done + mkdir "$stage_lock/metadata.new" || + { echo "could not block poison staging"; return 1; } wait_bounded "$pid" 11 [ "$WAIT_TIMED_OUT" -eq 0 ] || { echo "pre-cancel poison staging failure hung"; return 1; } [ "$WAIT_RC" -eq 11 ] || { echo "pre-cancel staging failure rc=$WAIT_RC want 11"; return 1; } @@ -282,7 +307,6 @@ t3_write_cancel_poisons() { echo "cancel issued after poison staging failed: $(tr '\n' ' ' < "$stage_state/calls.log")" return 1 fi - stage_lock="$stage_repo/.git/maestro-write.lock" [ -d "$stage_lock" ] || { echo "lease released after poison staging failed"; return 1; } grep -q 'was not cancelled and may still be running' "$stage_state/output" && grep -q 'task-fake0000-aaaaaa' "$stage_state/output" && @@ -315,10 +339,10 @@ t3_write_cancel_poisons() { set +m wait_bounded "$pid" 4 [ "$WAIT_TIMED_OUT" -eq 0 ] || { echo "staging marker clear hung"; return 1; } - [ "$WAIT_RC" -eq 0 ] || { echo "staging marker clear rc=$WAIT_RC want 0"; return 1; } - grep -Fq "$stage_lock/metadata.new" "$stage_state/clear-output" || - { echo "staging marker clear omitted removed entry"; return 1; } - [ ! -d "$stage_lock" ] || { echo "staging marker survived clear"; return 1; } + [ "$WAIT_RC" -eq 11 ] || { echo "staging marker clear rc=$WAIT_RC want 11"; return 1; } + grep -q 'staged Lease interval metadata is malformed' "$stage_state/clear-output" || + { echo "staging marker refusal diagnostic missing"; return 1; } + [ -d "$stage_lock" ] || { echo "unsafe staging marker was cleared"; return 1; } race_state="$TEST_ROOT/late-poison-race-state" race_shim="$race_state/shim" @@ -362,17 +386,18 @@ t3_write_cancel_poisons() { wait_bounded "$pid" 8 [ "$WAIT_TIMED_OUT" -eq 0 ] || { echo "late-poison release race hung"; race_failed=1; } - [ "$WAIT_RC" -eq 0 ] || - { echo "late-poison release rc=$WAIT_RC want 0"; race_failed=1; } + [ "$WAIT_RC" -eq 11 ] || + { echo "late-poison release rc=$WAIT_RC want 11"; race_failed=1; } grep -qx 'quiescence=unconfirmed' "$release_lock/metadata" 2>/dev/null || { echo "release erased poison that arrived during liveness check"; race_failed=1; } acquire_repo=$(new_repo late-poison-acquire-repo) acquire_lock="$acquire_repo/.git/maestro-write.lock" acquire_poison="$race_state/acquire.poison" - mkdir -p "$acquire_lock" + prepare_generation "$acquire_lock" || return 1 printf 'token=late-token\npid=99999999\nprocess_start=dead\njob_id=task-acquire-race\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=1\ndigest_before=unavailable\n' \ > "$acquire_lock/metadata" + sync_generation_field "$acquire_lock" || return 1 sed -n 'p' "$acquire_lock/metadata" > "$acquire_poison" printf 'quiescence=unconfirmed\nunconfirmed_job=task-acquire-race\nunconfirmed_reason=deadline\n' \ >> "$acquire_poison" @@ -459,8 +484,9 @@ t6_clear_lease_works_and_refuses() { refuse_repo=$(new_repo clear-refuse-repo) metadata="$refuse_repo/.git/maestro-write.lock/metadata" - mkdir -p "$(dirname "$metadata")" + prepare_generation "$(dirname "$metadata")" || return 1 printf 'token=old\npid=999999\nprocess_start=dead\njob_id=task-old00000-aaaaaa\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=1\ndigest_before=unavailable\nquiescence=unconfirmed\nunconfirmed_job=task-old00000-aaaaaa\nunconfirmed_reason=deadline\n' > "$metadata" + sync_generation_field "$(dirname "$metadata")" || return 1 status_running_job task-running0-bbbbbb true > "$state/running.json" set -m ( @@ -481,7 +507,7 @@ t6_clear_lease_works_and_refuses() { wedge_repo=$(new_repo clear-wedge-repo) wedge_lock="$wedge_repo/.git/maestro-write.lock" - mkdir -p "$wedge_lock" + prepare_generation "$wedge_lock" || return 1 touch -t 202001010000 "$wedge_lock" set -m ( @@ -501,7 +527,7 @@ t6_clear_lease_works_and_refuses() { fresh_wedge_repo=$(new_repo clear-fresh-wedge-repo) fresh_wedge_lock="$fresh_wedge_repo/.git/maestro-write.lock" - mkdir -p "$fresh_wedge_lock" + prepare_generation "$fresh_wedge_lock" || return 1 set -m ( cd "$fresh_wedge_repo" && @@ -521,9 +547,10 @@ t6_clear_lease_works_and_refuses() { healthy_lock="$healthy_repo/.git/maestro-write.lock" healthy_pid=$$ healthy_now=$(date +%s) - mkdir -p "$healthy_lock" + prepare_generation "$healthy_lock" || return 1 printf 'token=healthy-token\npid=%s\nprocess_start=unavailable\njob_id=task-healthy-owner\nsession_id=sess-healthy-owner\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=%s\ndigest_before=unavailable\n' \ "$healthy_pid" "$healthy_now" > "$healthy_lock/metadata" + sync_generation_field "$healthy_lock" || return 1 set -m ( cd "$healthy_repo" && @@ -679,6 +706,78 @@ t8_verifier_boundaries() { echo "successful verifier child $child survived VERIFIED_DONE" return 1 fi + repo=$(new_repo verifier-root-repo) + state="$TEST_ROOT/verifier-root-state" + mkdir -p "$repo/deep/nested" "$state" + : > "$state/calls.log" + status_empty > "$state/status.json" + verify='test "$PWD" = "$(git rev-parse --show-toplevel)"' + set -m + ( + cd "$repo/deep/nested" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ + MAESTRO_TEST_CALL_LOG="$state/calls.log" \ + MAESTRO_TEST_JOB_PHASE=completed \ + MAESTRO_TEST_RESULT='RESULT: DONE' \ + MAESTRO_TEST_STATUS="$state/status.json" \ + bash "$LOOP" --plan "$TEST_ROOT/plan.md" --verify "$verify" \ + --max-iters 1 --poll 1 + ) > "$state/output" 2>&1 & + pid=$! + set +m + wait_bounded "$pid" 7 + [ "$WAIT_TIMED_OUT" -eq 0 ] || { echo "verifier root cwd exceeded 7s bound"; return 1; } + [ "$WAIT_RC" -eq 0 ] || + { echo "verifier root cwd rc=$WAIT_RC want 0: $(tr '\n' ' ' < "$state/output")"; return 1; } + grep -q '^MAESTRO_FINAL: LOOP VERIFIED_DONE rc=0$' "$state/output" || + { echo "verifier root cwd final missing: $(tr '\n' ' ' < "$state/output")"; return 1; } + local heartbeat first second + repo=$(new_repo verifier-heartbeat-repo) + state="$TEST_ROOT/verifier-heartbeat-state" + mkdir -p "$state" + : > "$state/calls.log" + status_empty > "$state/status.json" + verify='sleep 5' + set -m + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ + MAESTRO_TEST_CALL_LOG="$state/calls.log" \ + MAESTRO_TEST_JOB_PHASE=completed \ + MAESTRO_TEST_RESULT='RESULT: DONE' \ + MAESTRO_TEST_STATUS="$state/status.json" \ + MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC=1 \ + MAESTRO_LOCK_HEARTBEAT_STALE_SEC=2 \ + MAESTRO_VERIFY_TIMEOUT_SEC=8 \ + bash "$LOOP" --plan "$TEST_ROOT/plan.md" --verify "$verify" \ + --max-iters 1 --poll 1 + ) > "$state/output" 2>&1 & + pid=$! + set +m + heartbeat="$repo/.git/maestro-write.lock/heartbeat" + for _ in 1 2 3 4 5 6 7 8; do + if grep -q 'LOOP: RESULT: DONE on iteration 1' "$state/output" 2>/dev/null && + [ -f "$heartbeat" ]; then + break + fi + sleep 1 + done + grep -q 'LOOP: RESULT: DONE on iteration 1' "$state/output" || + { echo "verifier did not reach local verification"; return 1; } + [ -f "$heartbeat" ] || { echo "verification heartbeat missing"; return 1; } + first=$(sed -n 's/^epoch=//p' "$heartbeat" | head -1) + sleep 2 + second=$(sed -n 's/^epoch=//p' "$heartbeat" | head -1) + case "$first:$second" in + :*|*:|*[!0-9:]*) echo "invalid heartbeat epochs: first=$first second=$second"; return 1 ;; + esac + [ "$second" -gt "$first" ] || + { echo "verification heartbeat did not advance: first=$first second=$second"; return 1; } + kill -0 "$pid" 2>/dev/null || + { echo "verifier ended before heartbeat observation"; return 1; } + wait_bounded "$pid" 10 + [ "$WAIT_TIMED_OUT" -eq 0 ] || { echo "heartbeat verifier exceeded 10s"; return 1; } + [ "$WAIT_RC" -eq 0 ] || { echo "heartbeat verifier rc=$WAIT_RC"; return 1; } } t9_terminal_at_deadline_harvests() { @@ -1019,7 +1118,7 @@ t17_malformed_status_counts_as_status_loss() { ) > "$state/output" 2>&1 & pid=$! set +m - wait_bounded "$pid" 7 + wait_bounded "$pid" 11 [ "$WAIT_TIMED_OUT" -eq 0 ] || { echo "malformed status did not take bounded status-loss path"; return 1; } [ "$WAIT_RC" -eq 11 ] || { echo "rc=$WAIT_RC want 11"; return 1; } statuses=$(grep -c '^status task-fake0000-aaaaaa --json$' "$state/calls.log" || true) @@ -1113,6 +1212,58 @@ t19_waiting_contender_signal_does_not_cancel_owner() ( { echo "waiting contender cancelled the active lease owner"; return 1; } ) +t20_concurrent_clear_large_status_is_bounded() ( + local repo state lock pid1 pid2 rc1 rc2 timed1 timed2 + repo=$(new_repo concurrent-clear-repo) + state="$TEST_ROOT/concurrent-clear-state" + lock="$repo/.git/maestro-write.lock" + mkdir -p "$state" + prepare_generation "$lock" || return 1 + "$REAL_NODE" -e ' + const fs = require("node:fs"); + fs.writeFileSync(process.argv[1], JSON.stringify({ + running: [], + latestFinished: null, + padding: " ".repeat(256 * 1024) + })); + ' "$state/status.json" || return 1 + printf 'token=concurrent-clear\npid=999999\nprocess_start=dead\njob_id=task-concurrent-clear\nsession_id=session-concurrent-clear\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=1\ndigest_before=unavailable\nquiescence=unconfirmed\nunconfirmed_job=task-concurrent-clear\nunconfirmed_reason=signal-term\n' \ + > "$lock/metadata" + sync_generation_field "$lock" || return 1 + + set -m + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ + MAESTRO_TEST_STATUS="$state/status.json" \ + bash "$LOOP" --clear-lease + ) > "$state/first.out" 2>&1 & + pid1=$! + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ + MAESTRO_TEST_STATUS="$state/status.json" \ + bash "$LOOP" --clear-lease + ) > "$state/second.out" 2>&1 & + pid2=$! + set +m + + wait_bounded "$pid1" 4 + rc1=$WAIT_RC + timed1=$WAIT_TIMED_OUT + wait_bounded "$pid2" 4 + rc2=$WAIT_RC + timed2=$WAIT_TIMED_OUT + [ "$timed1" -eq 0 ] && [ "$timed2" -eq 0 ] || + { echo "concurrent clear timed out first=$timed1 second=$timed2"; return 1; } + case "$rc1:$rc2" in + 0:0|0:11|11:0) ;; + *) echo "concurrent clear rc=$rc1:$rc2 want one success and no failure"; return 1 ;; + esac + [ ! -d "$lock" ] || + { echo "concurrent clear left lock entries: $(find "$lock" -mindepth 1 -maxdepth 1 -print)"; return 1; } +) + check() { local fn="$1" label="$2" detail if detail=$("$fn" 2>&1); then @@ -1125,7 +1276,7 @@ check() { printf '=== Liveness verification ===\n' check t1_deadline_growing_log "deadline fires against a growing log" check t2_growth_defeats_idle "log growth defeats the idle timer" -check t3_write_cancel_poisons "write cancellation poisons; failed mv retains; failed staging blocks and clears; late poison wins" +check t3_write_cancel_poisons "write cancellation poisons; failed writes retain fail-closed state; late poison wins" check t4_poison_stops_redispatch "poison prevents a second dispatch" check t5_poison_blocks_acquire "poison blocks later acquisition" check t6_clear_lease_works_and_refuses "clear-lease clears safely and refuses a live writer" @@ -1142,5 +1293,6 @@ check t16_observed_cancellation_is_terminal "observed write cancellation poisons check t17_malformed_status_counts_as_status_loss "malformed nonempty status follows bounded status loss" check t18_invalid_polling_args_fail_before_launch "invalid polling args and mixed clear mode fail before launch" check t19_waiting_contender_signal_does_not_cancel_owner "waiting contender TERM never cancels the active owner" +check t20_concurrent_clear_large_status_is_bounded "concurrent clear stays bounded with a large status payload" printf '\n=== %d passed, %d failed ===\n' "$PASS" "$FAIL" [ "$FAIL" -eq 0 ] diff --git a/tests/provenance-edge.sh b/tests/provenance-edge.sh index 791d5a4..a797949 100644 --- a/tests/provenance-edge.sh +++ b/tests/provenance-edge.sh @@ -73,13 +73,14 @@ t3_untracked_nested_repository_is_observed() { } t4_release_publishes_baseline_before_handoff() { - local repo state shim real_rmdir lock holder contender_out + local repo state shim real_mv lock holder contender_out contender_rc + local handoff_out handoff_rc repo=$(new_repo release-handoff) repo=$(cd "$repo" && pwd -P) state="$TEST_ROOT/release-handoff-state" shim="$state/shim" lock="$repo/.git/maestro-write.lock" - real_rmdir=$(command -v rmdir) + real_mv=$(command -v mv) mkdir -p "$shim" # Establish a prior completed baseline. ( @@ -90,17 +91,21 @@ t4_release_publishes_baseline_before_handoff() { write_lock_acquire task-baseline >/dev/null 2>&1 || exit 1 write_lock_release >/dev/null 2>&1 ) || return 1 - cat > "$shim/rmdir" < "$shim/mv" < "$state/lock-removed" - while [ ! -e "$state/contender-done" ]; do sleep 0.05; done + case "\${2:-}" in + "$lock".reclaim.*) + : > "$state/lock-moved" + while [ ! -e "$state/allow-release" ]; do sleep 0.05; done + ;; + esac fi exit "\$rc" EOF - chmod +x "$shim/rmdir" + chmod +x "$shim/mv" ( cd "$repo" || exit 1 . "$LIB" @@ -112,21 +117,43 @@ EOF ) > "$state/holder.out" 2>&1 & holder=$! for _ in $(seq 1 600); do - [ -e "$state/lock-removed" ] && break + [ -e "$state/lock-moved" ] && break sleep 0.05 done - [ -e "$state/lock-removed" ] || { kill "$holder" 2>/dev/null || :; echo "holder never removed lock"; return 1; } + [ -e "$state/lock-moved" ] || + { kill "$holder" 2>/dev/null || :; echo "holder never moved the released generation"; return 1; } contender_out=$( cd "$repo" || exit 1 . "$LIB" progress_init + export MAESTRO_LOCK_WAIT_SEC=0 write_lock_workspace_writers() { return 0; } - write_lock_acquire task-contender 3>&1 >/dev/null 2>&1 + write_lock_acquire task-gated-contender 3>&1 >/dev/null 2>&1 + rc=$? + [ "$rc" -ne 0 ] || write_lock_release >/dev/null 2>&1 + exit "$rc" ) - : > "$state/contender-done" + contender_rc=$? + : > "$state/allow-release" wait "$holder" || return 1 - if printf '%s\n' "$contender_out" | grep -q 'BASELINE GAP'; then - echo "contender observed a false gap during release handoff: $contender_out" + [ "$contender_rc" -eq 11 ] || + { echo "contender rc=$contender_rc bypassed the release generation gate: $contender_out"; return 1; } + handoff_out=$( + cd "$repo" || exit 1 + . "$LIB" + progress_init + export MAESTRO_LOCK_WAIT_SEC=0 + write_lock_workspace_writers() { return 0; } + write_lock_acquire task-handoff-contender 3>&1 >/dev/null 2>&1 + rc=$? + [ "$rc" -ne 0 ] || write_lock_release >/dev/null 2>&1 + exit "$rc" + ) + handoff_rc=$? + [ "$handoff_rc" -eq 0 ] || + { echo "handoff contender rc=$handoff_rc: $handoff_out"; return 1; } + if printf '%s\n' "$handoff_out" | grep -q 'BASELINE GAP'; then + echo "contender observed a false gap after release handoff: $handoff_out" return 1 fi } @@ -170,11 +197,14 @@ t6_provenance_append_never_follows_symlink() { t7_orphan_baseline_is_published_before_reclaim_handoff() { local repo lock state reclaimer contender_rc + local generation=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa repo=$(new_repo orphan-handoff) lock="$repo/.git/maestro-write.lock" state="$TEST_ROOT/orphan-handoff-state" mkdir -p "$lock" "$state" - printf 'token=orphan-token\npid=999999\nprocess_start=dead\njob_id=task-orphan-old\nsession_id=orphan-session\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=1\ndigest_before=unavailable\n' > "$lock/metadata" + printf '%s\n' "$generation" > "$lock/generation" + printf 'token=orphan-token\ngeneration=%s\npid=999999\nprocess_start=dead\njob_id=task-orphan-old\nsession_id=orphan-session\nstarted_at=2026-01-01T00:00:00Z\nstarted_epoch=1\ndigest_before=unavailable\n' \ + "$generation" > "$lock/metadata" ( cd "$repo" || exit 1 . "$LIB" diff --git a/tests/stop-report.sh b/tests/stop-report.sh index c0529cc..b3fcb68 100755 --- a/tests/stop-report.sh +++ b/tests/stop-report.sh @@ -68,13 +68,15 @@ printf 'Objective: stop for answers.\n' > "$NEEDS_PLAN" printf 'Objective: finish successfully.\n' > "$DONE_PLAN" printf 'Objective: fail closed when status is lost.\n' > "$STATUS_LOSS_PLAN" printf 'Objective: carry failed evidence.\n' > "$FAILED_PLAN" +printf 'Objective: reject a result prefix.\n' > "$REPO/prefix-plan.md" +printf 'Objective: accept the last result record.\n' > "$REPO/last-record-plan.md" cp "$DONE_PLAN" "$TEST_ROOT/done-plan.before" git init -q "$REPO" ( cd "$REPO" && git config user.email p@p && git config user.name p && - git add needs-plan.md done-plan.md status-loss-plan.md failed-plan.md && + git add needs-plan.md done-plan.md status-loss-plan.md failed-plan.md prefix-plan.md last-record-plan.md && git commit -q -m init ) @@ -85,19 +87,21 @@ QUESTIONS: CONTINUATION: - Completed: fixture-stop-capsule - Evidence: the answer is missing -- Next: apply the supplied answer' +' run_loop() { - local name="$1" plan="$2" result="$3" + local name="$1" plan="$2" result="$3" max_iters="${4:-1}" + : > "$TEST_ROOT/$name.calls" ( cd "$REPO" && env HOME="$TEST_HOME" PATH="$TEST_PATH" \ MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_CALL_LOG="$TEST_ROOT/$name.calls" \ MAESTRO_TEST_JOB_PHASE=completed \ MAESTRO_TEST_RESULT="$result" \ MAESTRO_TEST_STATUS="$STATUS" \ bash "$LOOP" --plan "$plan" --verify true \ - --max-iters 1 --max-idle 2 --poll 1 + --max-iters "$max_iters" --max-idle 2 --poll 1 ) > "$TEST_ROOT/$name.stdout" \ 2> "$TEST_ROOT/$name.stderr" \ 3> "$TEST_ROOT/$name.progress" @@ -143,9 +147,14 @@ run_failed_loop() { ) > "$output" 2>&1 3>&1 & pid=$! set +m - wait_for_pid "$pid" 8 + # Two dispatches settle in ~4s; a measured tail reaches 13s on a busy host. + wait_for_pid "$pid" 30 FAILED_LOOP_RC=$WAIT_RC FAILED_LOOP_TIMED_OUT=$WAIT_TIMED_OUT + # wait_for_pid TERMs the loop it gives up on, and the loop poisons the write + # lease on TERM. $REPO is shared with the cases below, so drop the poison here + # and let the timeout surface as this case alone instead of three cascades. + [ "$FAILED_LOOP_TIMED_OUT" -eq 0 ] || rm -rf "$REPO/.git/maestro-write.lock" } run_loop needs-first "$NEEDS_PLAN" "$NEEDS_RESULT" @@ -155,9 +164,9 @@ run_loop needs-second "$NEEDS_PLAN" "$NEEDS_RESULT" SECOND_RC=$? run_loop "done" "$DONE_PLAN" 'RESULT: DONE' DONE_RC=$? -run_loop prefix "$DONE_PLAN" 'RESULT: DONEISH' +run_loop prefix "$REPO/prefix-plan.md" 'RESULT: DONEISH' 2 PREFIX_RC=$? -run_loop last-record "$DONE_PLAN" $'RESULT: FAILED\nearlier failure\nRESULT: DONE' +run_loop last-record "$REPO/last-record-plan.md" $'RESULT: FAILED\nearlier failure\nRESULT: DONE' LAST_RECORD_RC=$? run_failed_loop run_status_loss @@ -253,9 +262,12 @@ t6_status_loss_fails_closed() { } t7_result_records_are_full_line_and_last_wins() { - [ "$PREFIX_RC" -eq 12 ] || { echo "DONEISH rc=$PREFIX_RC want 12"; return 1; } - grep -q 'no RESULT line' "$TEST_ROOT/prefix.progress" "$TEST_ROOT/prefix.stderr" || - { echo "DONEISH was not rejected as a missing result"; return 1; } + local starts + [ "$PREFIX_RC" -eq 11 ] || { echo "DONEISH rc=$PREFIX_RC want 11"; return 1; } + starts=$(grep -c '^task ' "$TEST_ROOT/prefix.calls" || true) + [ "$starts" -eq 1 ] || { echo "DONEISH starts=$starts want 1"; return 1; } + grep -qx 'IMPLEMENTER_STATE: COMPANION_FAILURE' "$TEST_ROOT/prefix.stderr" || + { echo "DONEISH companion-failure marker missing"; return 1; } [ "$LAST_RECORD_RC" -eq 0 ] || { echo "last-record rc=$LAST_RECORD_RC want 0"; return 1; } grep -q 'LOOP_STATE: VERIFIED_DONE' "$TEST_ROOT/last-record.progress" || { echo "last anchored DONE record did not win"; return 1; } @@ -276,6 +288,230 @@ t8_failed_result_evidence_reaches_next_dispatch() { [ "$first" = "$second" ] && [ "${first##* }" = "reclaim=0" ] || { echo "Write turns crossed Lease generations: first=$first second=$second"; return 1; } } +t8b_stuck_attempt_history_persists() { + local blocks evidence + blocks=$(grep -c '^--- BEGIN MAESTRO ATTEMPT HISTORY ' "$FAILED_PLAN" || true) + evidence=$(grep -c 'UNIQUE-FAILED-EVIDENCE-7319' "$FAILED_PLAN" || true) + [ "$blocks" -eq 1 ] || { echo "attempt history blocks=$blocks want 1"; return 1; } + [ "$evidence" -ge 1 ] || { echo "persisted attempt history omitted failed evidence"; return 1; } +} + +t8c_result_transport_failure_blocks_without_retry() ( + local repo="$TEST_ROOT/transport-failure-repo" + local plan output="$TEST_ROOT/transport-failure.out" + local calls="$TEST_ROOT/transport-failure.calls" rc starts + mkdir -p "$repo" + plan="$repo/plan.md" + printf 'Objective: classify result transport failure.\n' > "$plan" + ( + cd "$repo" && + git init -q && + git config user.email p@p && + git config user.name p && + git add plan.md && + git commit -q -m init + ) || return 1 + : > "$calls" + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ + MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_CALL_LOG="$calls" \ + MAESTRO_TEST_JOB_PHASE=completed \ + MAESTRO_TEST_RESULT='RESULT-TRANSPORT-STDOUT-7319' \ + MAESTRO_TEST_RESULT_STDERR='RESULT-TRANSPORT-STDERR-8426' \ + MAESTRO_TEST_RESULT_EXIT=1 \ + MAESTRO_TEST_STATUS="$STATUS" \ + bash "$LOOP" --plan "$plan" --verify true \ + --max-iters 2 --max-idle 2 --poll 1 + ) > "$output" 2>&1 3>&1 + rc=$? + starts=$(grep -c '^task ' "$calls" || true) + [ "$rc" -eq 11 ] || { echo "transport failure rc=$rc want 11"; return 1; } + [ "$starts" -eq 1 ] || { echo "transport failure starts=$starts want 1"; return 1; } + grep -qx 'IMPLEMENTER_STATE: COMPANION_FAILURE' "$output" || + { echo "companion failure marker missing"; return 1; } + grep -q 'post-launch companion/process/result failure' "$output" || + { echo "companion failure guidance missing"; return 1; } + grep -q 'RESULT-TRANSPORT-STDOUT-7319' "$output" || + { echo "result transport stdout evidence missing"; return 1; } + grep -q 'RESULT-TRANSPORT-STDERR-8426' "$output" || + { echo "result transport stderr evidence missing"; return 1; } +) +t8d_unparseable_launch_retains_both_locks() ( + local repo="$TEST_ROOT/unparseable-launch-repo" plan output calls rc starts + local write_lock job_lock + mkdir -p "$repo" + plan="$repo/plan.md" + output="$repo/output" + calls="$repo/calls" + write_lock="$repo/.git/maestro-write.lock" + job_lock="$repo/.git/maestro-job-lock" + printf 'Objective: retain ownership after an unparseable launch response.\n' > "$plan" + ( + cd "$repo" && + git init -q && + git config user.email p@p && + git config user.name p && + git add plan.md && + git commit -q -m init + ) || return 1 + : > "$calls" + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ + MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_CALL_LOG="$calls" \ + MAESTRO_TEST_TASK_RESPONSE_RAW='accepted-without-job-id' \ + MAESTRO_TEST_STATUS="$STATUS" \ + bash "$LOOP" --plan "$plan" --verify true \ + --max-iters 2 --max-idle 2 --poll 1 + ) > "$output" 2>&1 3>&1 + rc=$? + starts=$(grep -c '^task ' "$calls" || true) + [ "$rc" -eq 11 ] || { echo "unparseable launch rc=$rc want 11"; return 1; } + [ "$starts" -eq 1 ] || { echo "unparseable launch starts=$starts want 1"; return 1; } + grep -qx 'IMPLEMENTER_STATE: COMPANION_FAILURE' "$output" || + { echo "unparseable launch companion-failure marker missing"; return 1; } + [ -f "$write_lock/metadata.new" ] && + grep -qx 'quiescence=unconfirmed' "$write_lock/metadata.new" || + { echo "unparseable launch did not poison and retain the write lease"; return 1; } + [ -f "$job_lock/metadata" ] && + ! grep -q '^job=' "$job_lock/metadata" || + { echo "unparseable launch did not retain an unpublished companion job lock"; return 1; } +) +t8e_attempt_history_is_byte_bounded() ( + local repo="$TEST_ROOT/bounded-history-repo" plan output result_file rc history_bytes + mkdir -p "$repo" + plan="$repo/plan.md" + output="$repo/output" + result_file="$repo/result.txt" + printf 'Objective: bound persisted attempt evidence.\n' > "$plan" + "$REAL_NODE" -e ' + process.stdout.write( + "RESULT: FAILED\nBOUNDARY-START-" + "x".repeat(100000) + "-BOUNDARY-END\n" + ); + ' > "$result_file" || return 1 + ( + cd "$repo" && + git init -q && + git config user.email p@p && + git config user.name p && + git add plan.md && + git commit -q -m init + ) || return 1 + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ + MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_JOB_PHASE=completed \ + MAESTRO_TEST_RESULT_FILE="$result_file" \ + MAESTRO_TEST_STATUS="$STATUS" \ + bash "$LOOP" --plan "$plan" --verify true \ + --max-iters 1 --max-idle 2 --poll 1 + ) > "$output" 2>&1 3>&1 + rc=$? + history_bytes=$(sed -n \ + '/^--- BEGIN MAESTRO ATTEMPT HISTORY /,/^--- END MAESTRO ATTEMPT HISTORY ---$/p' \ + "$plan" | wc -c | tr -d ' ') + [ "$rc" -eq 12 ] || { echo "oversized history rc=$rc want 12"; return 1; } + [ "$history_bytes" -le 65536 ] || + { echo "persisted history bytes=$history_bytes exceed the 65536-byte ceiling"; return 1; } + grep -q 'MAESTRO_ATTEMPT_HISTORY_TRUNCATED' "$plan" || + { echo "persisted history lacks an explicit truncation marker"; return 1; } + grep -q -- '-BOUNDARY-END' "$plan" || + { echo "persisted history did not retain the useful evidence tail"; return 1; } + ! grep -q 'BOUNDARY-START' "$plan" || + { echo "persisted history retained the oversized evidence prefix"; return 1; } +) + + + + +t8f_attempt_history_quotes_embedded_delimiters() ( + local repo="$TEST_ROOT/history-delimiter-repo" plan output result_file rc ends + mkdir -p "$repo" + plan="$repo/plan.md" + output="$repo/output" + result_file="$repo/result.txt" + printf 'Objective: preserve attempt-history framing.\n' > "$plan" + printf 'RESULT: FAILED\n%s\nAFTER-INJECTED-END-9184\n' \ + '--- END MAESTRO ATTEMPT HISTORY ---' > "$result_file" + ( + cd "$repo" && + git init -q && + git config user.email p@p && + git config user.name p && + git add plan.md && + git commit -q -m init + ) || return 1 + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ + MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_JOB_PHASE=completed \ + MAESTRO_TEST_RESULT_FILE="$result_file" \ + MAESTRO_TEST_STATUS="$STATUS" \ + bash "$LOOP" --plan "$plan" --verify true \ + --max-iters 1 --max-idle 2 --poll 1 + ) > "$output" 2>&1 3>&1 + rc=$? + ends=$(grep -c '^--- END MAESTRO ATTEMPT HISTORY ---$' "$plan" || true) + [ "$rc" -eq 12 ] || { echo "embedded delimiter rc=$rc want 12"; return 1; } + [ "$ends" -eq 1 ] || + { echo "embedded payload created $ends closing delimiters want 1"; return 1; } + grep -qx '> --- END MAESTRO ATTEMPT HISTORY ---' "$plan" || + { echo "embedded closing delimiter was not quoted"; return 1; } + grep -qx '> AFTER-INJECTED-END-9184' "$plan" || + { echo "payload after embedded delimiter escaped the quoted history"; return 1; } +) + +t8g_attempt_history_commit_failure_preserves_plan() ( + local repo="$TEST_ROOT/history-commit-repo" plan before output result_file shim rc + repo="$TEST_ROOT/history-commit-repo" + mkdir -p "$repo" + plan="$repo/plan.md" + before="$repo/plan.before" + output="$repo/output" + result_file="$repo/result.txt" + shim="$repo/shim" + mkdir -p "$shim" + printf 'Objective: preserve the plan on history commit failure.\n' > "$plan" + cp "$plan" "$before" + printf 'RESULT: FAILED\nATOMIC-HISTORY-EVIDENCE-5731\n' > "$result_file" + { + printf '#!/usr/bin/env bash\n' + printf 'if [ "${1:-}" = "-f" ]; then\n' + printf ' case "${2:-}" in */.maestro-history.*) exit 73 ;; esac\n' + printf 'fi\n' + printf 'exec "%s" "$@"\n' "$REAL_MV" + } > "$shim/mv" + chmod +x "$shim/mv" + ( + cd "$repo" && + git init -q && + git config user.email p@p && + git config user.name p && + git add plan.md && + git commit -q -m init + ) || return 1 + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$shim:$TEST_PATH" \ + MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_JOB_PHASE=completed \ + MAESTRO_TEST_RESULT_FILE="$result_file" \ + MAESTRO_TEST_STATUS="$STATUS" \ + bash "$LOOP" --plan "$plan" --verify true \ + --max-iters 1 --max-idle 2 --poll 1 + ) > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 12 ] || { echo "history commit failure rc=$rc want 12"; return 1; } + cmp -s "$plan" "$before" || + { echo "history commit failure partially changed the plan"; return 1; } + grep -q 'could not append attempt history' "$output" || + { echo "history commit failure warning missing"; return 1; } +) t9_loop_is_independent_of_single_shot_adapter() { local bundle="$TEST_ROOT/peer-hooks" repo="$TEST_ROOT/peer-repo" @@ -372,6 +608,31 @@ t10_public_adapter_exit_mappings() { [ "$rc" -eq 3 ] && grep -q '^MAESTRO_FINAL: LOOP FAILED rc=3$' "$output" || { echo "loop invalid mapping rc=$rc"; return 1; } + output="$TEST_ROOT/loop-unquoted-verify.out" + ( + cd "$mapping_repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ + bash "$LOOP" --plan "$DONE_PLAN" --verify true extra --max-iters 1 + ) > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 3 ] && + grep -q '^LOOP_ERROR: unknown argument: extra$' "$output" || + { echo "unquoted verifier residue rc=$rc: $(tr '\n' ' ' < "$output")"; return 1; } + + output="$TEST_ROOT/loop-verifier-flag-string.out" + ( + cd "$mapping_repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" \ + MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_JOB_PHASE=completed \ + MAESTRO_TEST_RESULT='RESULT: DONE' \ + MAESTRO_TEST_STATUS="$STATUS" \ + bash "$LOOP" --plan "$DONE_PLAN" --verify 'true --max-iters 4' --max-iters 1 + ) > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 0 ] && + grep -q '^MAESTRO_FINAL: LOOP VERIFIED_DONE rc=0$' "$output" || + { echo "flag-like verifier command rc=$rc: $(tr '\n' ' ' < "$output")"; return 1; } } t11_signal_cancels_unpublished_writer_and_prints_result() ( @@ -409,7 +670,9 @@ t11_signal_cancels_unpublished_writer_and_prints_result() ( grep -q '^status task-fake0000-aaaaaa --json$' "$state/calls.log" || { kill -KILL "$pid" 2>/dev/null || :; echo "watchdog never entered polling"; return 1; } kill -TERM "$pid" || return 1 - wait_for_pid "$pid" 8 + # Signalled shutdown cancels two writers before exiting; keep the hang guard + # well clear of that work so a busy host reads as slow, not as rc=124. + wait_for_pid "$pid" 30 rc=$WAIT_RC [ "$rc" -eq 125 ] || { echo "signal rc=$rc want 125: $(tr '\n' ' ' < "$state/output")"; return 1; } grep -qx 'cancel task-fake0000-aaaaaa' "$state/calls.log" || @@ -464,6 +727,172 @@ t12_passing_verification_survives_unwritable_fact() ( grep -qx 'MAESTRO_FINAL: LOOP VERIFIED_DONE rc=0' "$state/output" || { echo "VERIFIED_DONE final missing"; return 1; } ) +t13_release_failure_overrides_verified_done() ( + local repo="$TEST_ROOT/release-failure-repo" + local plan="$TEST_ROOT/release-failure-plan.md" + local output="$TEST_ROOT/release-failure.out" rc + mkdir -p "$repo" || return 1 + printf 'Objective: fail closed when lease release is unsafe.\n' > "$plan" + ( + cd "$repo" || exit 1 + git init -q + git config user.email p@p + git config user.name p + printf 'seed\n' > seed + git add seed + git commit -q -m init + ) || return 1 + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_JOB_PHASE=completed MAESTRO_TEST_RESULT='RESULT: DONE' \ + MAESTRO_TEST_STATUS="$STATUS" \ + bash "$LOOP" --plan "$plan" \ + --verify 'touch .git/maestro-write.lock/foreign-entry' \ + --max-iters 1 --max-idle 2 --poll 1 + ) > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 11 ] || + { echo "unsafe release rc=$rc want 11: $(tr '\n' ' ' < "$output")"; return 1; } + grep -q 'unrecognized entry' "$output" || + { echo "unsafe release diagnostic missing"; return 1; } + grep -qx 'MAESTRO_FINAL: LOOP BLOCKED rc=11' "$output" || + { echo "unsafe release did not override the success final"; return 1; } + ! grep -q '^MAESTRO_FINAL: LOOP VERIFIED_DONE rc=0$' "$output" || + { echo "unsafe release emitted a verified success final"; return 1; } + ! grep -q '^LOOP_STATE: VERIFIED_DONE' "$output" || + { echo "unsafe release emitted a verified success state"; return 1; } + grep -q '^LOOP_STATE: BLOCKED' "$output" || + { echo "unsafe release omitted the terminal blocked state"; return 1; } +) + +t13b_failed_verifier_release_failure_has_one_terminal_state() ( + local repo="$TEST_ROOT/release-failure-stuck-repo" + local plan="$TEST_ROOT/release-failure-stuck-plan.md" + local output="$TEST_ROOT/release-failure-stuck.out" rc states + mkdir -p "$repo" || return 1 + printf 'Objective: fail verification and retain an unsafe lease.\n' > "$plan" + ( + cd "$repo" || exit 1 + git init -q + git config user.email p@p + git config user.name p + printf 'seed\n' > seed + git add seed + git commit -q -m init + ) || return 1 + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_JOB_PHASE=completed MAESTRO_TEST_RESULT='RESULT: DONE' \ + MAESTRO_TEST_STATUS="$STATUS" \ + bash "$LOOP" --plan "$plan" \ + --verify 'touch .git/maestro-write.lock/foreign-entry; false' \ + --max-iters 1 --max-idle 2 --poll 1 + ) > "$output" 2>&1 3>&1 + rc=$? + states=$(grep -c '^LOOP_STATE:' "$output" || true) + [ "$rc" -eq 11 ] || + { echo "failed verifier unsafe release rc=$rc want 11"; return 1; } + [ "$states" -eq 1 ] || + { echo "failed verifier emitted $states terminal states: $(grep '^LOOP_STATE:' "$output")"; return 1; } + grep -q '^LOOP_STATE: BLOCKED' "$output" || + { echo "failed verifier unsafe release omitted blocked state"; return 1; } + ! grep -q '^LOOP_STATE: STUCK' "$output" || + { echo "failed verifier emitted stale STUCK state"; return 1; } +) + +t14_release_cleanup_residue_does_not_reclassify_success() ( + local repo="$TEST_ROOT/release-cleanup-repo" + local plan="$TEST_ROOT/release-cleanup-plan.md" + local state="$TEST_ROOT/release-cleanup-state" + local shim="$state/shim" output="$state/output" real_rm candidate + local rc residue=0 + mkdir -p "$repo" "$shim" || return 1 + printf 'Objective: release the canonical lease despite cleanup residue.\n' > "$plan" + ( + cd "$repo" || exit 1 + git init -q + git config user.email p@p + git config user.name p + printf 'seed\n' > seed + git add seed + git commit -q -m init + ) || return 1 + real_rm=$(command -v rm) || return 1 + { + printf '#!/usr/bin/env bash\n' + printf 'for arg in "$@"; do\n' + printf ' case "$arg" in */maestro-write.lock.reclaim.*) exit 1 ;; esac\n' + printf 'done\n' + printf 'exec "$MAESTRO_TEST_REAL_RM" "$@"\n' + } > "$shim/rm" || return 1 + chmod +x "$shim/rm" || return 1 + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$shim:$TEST_PATH" MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_REAL_RM="$real_rm" \ + MAESTRO_TEST_JOB_PHASE=completed MAESTRO_TEST_RESULT='RESULT: DONE' \ + MAESTRO_TEST_STATUS="$STATUS" \ + bash "$LOOP" --plan "$plan" --verify true \ + --max-iters 1 --max-idle 2 --poll 1 + ) > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 0 ] || + { echo "cleanup residue rc=$rc want 0: $(tr '\n' ' ' < "$output")"; return 1; } + grep -q '^MAESTRO_LOCK_CLEANUP: canonical lock released;' "$output" || + { echo "cleanup residue diagnostic missing"; return 1; } + grep -qx 'MAESTRO_FINAL: LOOP VERIFIED_DONE rc=0' "$output" || + { echo "cleanup residue reclassified verified success"; return 1; } + [ ! -d "$repo/.git/maestro-write.lock" ] || + { echo "canonical lease survived atomic retirement"; return 1; } + for candidate in "$repo/.git/maestro-write.lock.reclaim."*; do + [ ! -e "$candidate" ] || residue=$((residue + 1)) + done + [ "$residue" -eq 1 ] || + { echo "cleanup residue count=$residue want 1"; return 1; } +) + +t15_job_lock_release_failure_blocks_completed_turn() ( + local repo="$TEST_ROOT/job-release-failure-repo" + local plan="$TEST_ROOT/job-release-failure-plan.md" + local output="$TEST_ROOT/job-release-failure.out" rc + local job_lock write_lock + mkdir -p "$repo" || return 1 + printf 'Objective: block if the companion job lock cannot be released.\n' > "$plan" + ( + cd "$repo" || exit 1 + git init -q + git config user.email p@p + git config user.name p + printf 'seed\n' > seed + git add seed + git commit -q -m init + ) || return 1 + job_lock="$repo/.git/maestro-job-lock" + write_lock="$repo/.git/maestro-write.lock" + ( + cd "$repo" && + env HOME="$TEST_HOME" PATH="$TEST_PATH" MAESTRO_LOCK_WAIT_SEC=0 \ + MAESTRO_TEST_JOB_PHASE=completed MAESTRO_TEST_RESULT='RESULT: DONE' \ + MAESTRO_TEST_STATUS="$STATUS" \ + MAESTRO_TEST_MUTATE_JOB_LOCK_METADATA="$job_lock/metadata" \ + bash "$LOOP" --plan "$plan" --verify true \ + --max-iters 1 --max-idle 2 --poll 1 + ) > "$output" 2>&1 3>&1 + rc=$? + [ "$rc" -eq 11 ] || + { echo "job-lock release failure rc=$rc want 11: $(tr '\n' ' ' < "$output")"; return 1; } + grep -q 'MAESTRO_JOB_LOCK:.*release' "$output" || + { echo "job-lock release failure diagnostic missing"; return 1; } + grep -qx 'MAESTRO_FINAL: LOOP BLOCKED rc=11' "$output" || + { echo "job-lock release failure did not block public adapter"; return 1; } + [ -d "$job_lock" ] || + { echo "foreign job-lock generation was removed"; return 1; } + [ ! -d "$write_lock" ] || + { echo "write lease remained after job-lock-only failure"; return 1; } +) + check() { local fn="$1" label="$2" detail if detail=$("$fn" 2>&1); then @@ -483,9 +912,19 @@ check t5_verified_done_unchanged "VERIFIED_DONE appends nothing" check t6_status_loss_fails_closed "status loss blocks after one dispatch and retains poison" check t7_result_records_are_full_line_and_last_wins "RESULT records are anchored and the last record wins" check t8_failed_result_evidence_reaches_next_dispatch "FAILED result evidence reaches the next dispatch" +check t8b_stuck_attempt_history_persists "STUCK persists bounded attempt history in the plan" +check t8c_result_transport_failure_blocks_without_retry "result transport failure blocks without billing a retry" +check t8d_unparseable_launch_retains_both_locks "unparseable successful launch retains both ownership locks" +check t8e_attempt_history_is_byte_bounded "STUCK attempt history has a hard byte ceiling" +check t8f_attempt_history_quotes_embedded_delimiters "attempt history quotes embedded closing delimiters" +check t8g_attempt_history_commit_failure_preserves_plan "failed history commit leaves the plan byte-identical" check t9_loop_is_independent_of_single_shot_adapter "loop and single-shot adapters are peers" check t10_public_adapter_exit_mappings "public adapter exits and final markers stay exact" check t11_signal_cancels_unpublished_writer_and_prints_result "signal cancellation scans all writers and prints the blocked result" check t12_passing_verification_survives_unwritable_fact "passing verification survives an unwritable fact file" +check t13_release_failure_overrides_verified_done "unsafe lease release overrides public adapter success" +check t13b_failed_verifier_release_failure_has_one_terminal_state "failed verification plus unsafe release emits one blocked state" +check t14_release_cleanup_residue_does_not_reclassify_success "post-retirement cleanup residue does not reclassify success" +check t15_job_lock_release_failure_blocks_completed_turn "completed turn blocks when its job lock cannot be released" printf '\n=== %d passed, %d failed ===\n' "$PASS" "$FAIL" [ "$FAIL" -eq 0 ]