Skip to content

RedisWorker should clean up stale Redis locks from previous incarnation at startup #7919

Description

@dkliban

Problem

When a Kubernetes pod restarts within the same ReplicaSet, the new worker gets the same name (e.g., 1@pulp-worker-abc123-xyz). The new worker creates a fresh AppStatus but Redis still holds locks from the previous incarnation under the same name. These locks block resources indefinitely.

During the 2026-07-24 incident, worker 8qc25 was killed without graceful shutdown. Its 543 task locks and 524 resource locks persisted in Redis. When new workers came online, they had different names (different ReplicaSet), so they couldn't release the old locks. The app_worker_cleanup mechanism also missed them because the AppStatus record was already gone.

Proposed Fix

Add release_stale_locks_for_self() to RedisWorker.__init__(), called after redis_conn.ping() and before startup_hook():

  1. Guard: skip if another live AppStatus has our name (should not happen under normal K8s semantics, but defensive)
  2. Scan Redis for all task:* and pulp:resource_lock:* keys owned by our worker name
  3. Release them (delete string keys, srem from set keys)
  4. Log a warning with counts if any stale locks were found

This handles pod-restart-with-same-name immediately at startup, before accepting any tasks.

References

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions