RedisWorker fetch_task() head-of-line blocking: workers starved when queue head is serialized on one resource (WORKER_TYPE=redis)

[dkliban]

Summary

fetch_task() in RedisWorker always scans from FIFO position 0 and returns the first claimable task. When the queue head is dominated by tasks needing the same exclusive resource, all workers compete for the same blocked tasks and never reach tasks with free resources further down the queue.

Real-world impact (2026-07-24)

• 17,000+ general_create tasks for public-trusted-libraries all needing exclusive access to the same Python repository
• 10 workers online, but only 1 could work — the other 9 scanned the same FIFO head, found everything blocked, and were idle
• Tasks from other domains (lightwell, hummingbird) with completely free resources were starved for hours
• The fetch_limit doubling (20 → 40 → 80 → ...) only triggers when NO task in the batch is claimable. Since a few COPR tasks were mixed in, workers grabbed those and never triggered the doubling far enough to reach the starved domains

Current algorithm (redis_worker.py lines 424-510)

fetch_limit = 20

while True:
    reset taken_exclusive, taken_shared
    query oldest `fetch_limit` waiting tasks (FIFO from position 0)
    
    for each task:
        skip if resources conflict with taken_exclusive/taken_shared
        add resources to taken_exclusive/taken_shared
        try acquire_locks() → if blocked, continue
        try DB claim → if failed, release locks, continue
        return task  ← stops scanning immediately
    
    if batch was full: fetch_limit *= 2, restart from position 0
    else: break

Problems:

1. Every worker starts from position 0 every time
2. return task on first claim stops scanning — no attempt to find parallelizable work
3. The doubling re-queries from position 0, re-examining already-checked tasks
4. taken_exclusive/taken_shared knowledge is discarded on each doubling iteration

Proposed fix: blocked-resource DB-level exclusion

Track resources that acquire_locks reports as blocked. On subsequent batch queries, exclude tasks needing those resources at the DB level using PostgreSQL's array overlap operator:

blocked_resources = set()
offset = 0

while True:
    qs = Task.objects.filter(state=WAITING, app_lock=None)
    if blocked_resources:
        qs = qs.exclude(reserved_resources_record__overlap=list(blocked_resources))
    
    waiting_tasks = qs.order_by("pulp_created")[offset:offset + FETCH_TASK_LIMIT]
    
    for each task:
        ... same lock acquisition logic ...
        if acquire_locks fails:
            blocked_resources.add(failed_resources)  # key change
            continue
        return task
    
    offset += FETCH_TASK_LIMIT  # advance instead of doubling from 0

This leverages the existing GIN index (pulp_task_resources_index) on reserved_resources_record. In the incident scenario, the second query skips all 17,000 blocked tasks at the DB level and returns tasks with free resources. Total: 2 queries instead of ~10 exponentially growing queries.

Key properties

• FIFO fairness within same resource preserved — tasks for the same resource still execute in creation order
• No changes to Redis lock mechanism — acquire_locks Lua script unchanged
• No changes to handle_tasks() interface — fetch_task() still returns Task or None
• blocked_resources is local to each fetch_task() call — starts fresh after each task completes, so newly-unblocked resources are not missed
• Conservative resource exclusion — only raw resource names added (not shared: prefixed), so tasks needing shared access to an exclusively-locked resource are still tried (they fail fast at acquire_locks)

Files involved

• pulpcore/tasking/redis_worker.py — fetch_task() method (lines 424-510) is the only method that changes
• pulpcore/app/models/task.py — existing GIN index on reserved_resources_record (lines 382-387) supports the __overlap exclusion query
• pulpcore/tasking/redis_locks.py — no changes needed; acquire_locks return value already provides blocked resource names