app_worker_cleanup should scan Redis for orphaned locks from unknown owners

[dkliban]

Problem

The current app_worker_cleanup() only discovers dead workers via AppStatus.objects.missing(). If an AppStatus record is deleted before cleanup runs (e.g., by a previous cleanup pass, by the worker's own shutdown() method, or because the pod was killed without graceful shutdown), the cleanup finds nothing and the Redis locks persist indefinitely.

During the 2026-07-24 incident, worker 8qc25 had no AppStatus record when cleanup ran. Its 543 task locks and 524 resource locks were invisible to app_worker_cleanup and required manual intervention via the release-task-locks debug API.

Proposed Fix

Add a Redis-first reconciliation step to app_worker_cleanup():

1. Scan all Redis lock keys (task:* and pulp:resource_lock:*) and collect unique owner names
2. Get all worker names from AppStatus.objects.all() (not just online — includes stale heartbeats)
3. Any lock owner with no AppStatus record at all has orphaned locks — release them
4. Fail any associated tasks that are not in a final state

Important: only release locks for owners with no AppStatus record, NOT for owners with stale heartbeats. A worker with a stale heartbeat may still be alive and executing a task (e.g., DB contention preventing heartbeat writes). Releasing its locks while the task is running would break exclusive resource protection.

This runs under the existing WORKER_CLEANUP_LOCK advisory lock, so only one worker performs the scan at a time.

Add a shared cleanup_locks_for_owner() function in redis_locks.py that scans for and releases all locks owned by a given name. Reuse this function for both startup cleanup (#7919) and periodic reconciliation.

References

• RCA: 2026-07-24/25 tasking degradation
• RedisWorker should clean up stale Redis locks from previous incarnation at startup #7919 — Startup self-cleanup (uses the same cleanup_locks_for_owner function)