Blocked resource detector can wedge forked children: get_aggregated_resources timeout does not bound the wait

[can019]

Describe your environment

Reproduced on Linux (in a container, which is where it actually bites) and on macOS:

OS: Debian GNU/Linux 13 (trixie), kernel 6.8.0-1055-gke  # container image, GKE node
Python version: 3.12.13
SDK version: 1.44.0
API version: 1.44.0

OS: macOS (Darwin 25.4.0)
Python version: 3.12.13
SDK version: 1.44.0
API version: 1.44.0

Not platform specific — it only depends on threading.Lock + os.register_at_fork semantics.

What happened?

Two related problems, one is the cause of the other.

1. get_aggregated_resources() timeout does not bound the wait.

The detectors run in a with concurrent.futures.ThreadPoolExecutor(...) block. future.result(timeout=timeout) only bounds how long we wait for the result — leaving the with block calls Executor.shutdown(wait=True), which joins the still-running worker. So a detector that takes longer than timeout still blocks the whole call for its full duration, and a detector that blocks forever makes the call never return. The Detector ... took longer than N seconds, skipping warning is emitted, which makes it look like the timeout worked, but it did not.

https://github.com/open-telemetry/opentelemetry-python/blob/v1.44.0/opentelemetry-sdk/src/opentelemetry/sdk/resources/__init__.py

2. Because of 1, a forked child can hang permanently inside the at_fork handler added in 1.44.

1.44 (#5280, motivated by #5257) added process-dependent resource refresh:

def _handle_fork(self) -> None:
    self._tracers_lock = threading.Lock()
    self._update_resource(_get_process_dependent_resource())

registered via os.register_at_fork(after_in_child=...) on both TracerProvider and MeterProvider. _get_process_dependent_resource() goes through get_aggregated_resources(), and ServiceInstanceIdResourceDetector.detect() acquires the module-level _service_instance_id_lock (a plain threading.Lock).

If any thread in the parent holds that lock at the moment another thread calls fork(), the child inherits it locked with no owner left to release it. The pool worker blocks forever, shutdown(wait=True) never returns, and the child never returns from fork() — it is stuck inside the at_fork handler before any user code runs.

Under a prefork worker model this is fatal rather than merely slow: the child never signals readiness, so the supervisor kills it (e.g. Celery prefork SIGKILLs the child), producing a crash loop. It happens on every fork, including worker recycling (celery --max-tasks-per-child, gunicorn --max-requests).

The critical section is tiny (uuid4() + a pid comparison), so the race is unlikely — but it is unrecoverable when it happens, and the fork path is exercised constantly in prefork deployments.

Steps to Reproduce

Part 1 — the timeout does not bound the wait (public API only, deterministic):

import time
from opentelemetry.sdk.resources import Resource, ResourceDetector, get_aggregated_resources

class SlowDetector(ResourceDetector):
    def detect(self):
        time.sleep(8)
        return Resource.get_empty()

t0 = time.time()
get_aggregated_resources([SlowDetector()], Resource.get_empty(), timeout=1)
print(f"returned after {time.time() - t0:.1f}s")

Part 2 — forked child hangs. The _service_instance_id_lock is touched directly only to make the race deterministic; in production the same state occurs whenever a thread happens to be inside ServiceInstanceIdResourceDetector.detect() while another thread forks.

import os, threading, time
import opentelemetry.sdk.resources as res
from opentelemetry.sdk.trace import TracerProvider

provider = TracerProvider()   # registers TracerProvider._handle_fork via os.register_at_fork

started = threading.Event()

def hold_lock():
    with res._service_instance_id_lock:
        started.set()
        time.sleep(30)

threading.Thread(target=hold_lock, daemon=True).start()
started.wait()

pid = os.fork()
if pid == 0:
    print("child: returned from fork, reached user code")
    os._exit(0)

deadline = time.time() + 10
while time.time() < deadline:
    if os.waitpid(pid, os.WNOHANG)[0]:
        print("child exited normally")
        break
    time.sleep(0.1)
else:
    print("child is STILL STUCK 10s after fork")
    os.kill(pid, 9); os.waitpid(pid, 0)

Expected Result

Part 1: get_aggregated_resources(..., timeout=1) returns in about 1s, skipping the slow detector.

Part 2: the child returns from fork() and reaches user code, at worst without the process-dependent attribute.

Actual Result

Part 1 — the warning fires at 1s but the call blocks for the detector's full 8s:

Detector <__main__.SlowDetector object at 0x100f02840> took longer than 1 seconds, skipping
returned after 8.0s

Part 2 — the child never returns from fork(); child: returned from fork, reached user code is never printed:

Detector <opentelemetry.sdk.resources.ServiceInstanceIdResourceDetector object at 0x103f4e300> took longer than 5 seconds, skipping
child is STILL STUCK 10s after fork

Additional context

Any of these would fix Part 2; the first two also fix Part 1:

1. Don't join blocked workers — executor.shutdown(wait=False, cancel_futures=True) instead of relying on the with block. Leaks a thread, but the timeout becomes real.
2. Don't use a thread pool on the fork path. Right after fork() the child is single-threaded, so spawning a pool there buys nothing, and the timeout isolation it is meant to provide does not work anyway (Part 1).
3. Re-initialize the lock in the child — have opentelemetry.sdk.resources register os.register_at_fork(after_in_child=...) to replace _service_instance_id_lock with a fresh Lock(). Most local fix, but leaves Part 1 in place.

Note that a userland workaround is not really available. The usual pattern of bracketing the fork (before=lock.acquire, after_in_*=lock.release) does not help here: after_in_child handlers run in registration order, and opentelemetry-instrument creates the providers at interpreter startup, so the SDK's _handle_fork always runs before any handler an application can register.

Possibly related: #4345 (deadlock reaching metric reader storage under gunicorn+gevent), #4215, #3307. #3885 is the metrics-under-fork problem that per-pid service.instance.id addresses, so the fork refresh path matters and is worth keeping — it just should not be able to wedge the child.

Would you like to implement a fix?

No

[can019]

Two follow-ups.

Confirmed on Linux, inside a container. Same Part 2 script, run in a Debian 13 (trixie) container image on a GKE node, Python 3.12.13, SDK 1.44.0:

platform: Linux 6.8.0-1055-gke
Detector <opentelemetry.sdk.resources.ServiceInstanceIdResourceDetector object at 0x75a420106000> took longer than 5 seconds, skipping
RESULT: child is STILL STUCK 10s after fork

I have updated the environment section accordingly. This is the environment that matters for the report — a Celery prefork worker in a container — since a wedged child there never signals readiness and gets SIGKILLed into a crash loop.

Prior art, in case it helps triage. There is an existing line of work on exactly this class of bug that has stalled:

• fix: reinitialize locks after fork to prevent deadlocks in child processes #4626 "fix: reinitialize locks after fork to prevent deadlocks in child processes" — broad register_at_fork(after_in_child=...) + _at_fork_reinit() across all Lock/RLock usages, including opentelemetry-api's util/_once.py. Opened 2025-06-09, the author asked for maintainer input several times, received no review, and it was closed by the stale bot on 2026-03-20.
• fix(metrics): Add fork-safety to SynchronousMeasurementConsumer #4767 "fix(metrics): Add fork-safety to SynchronousMeasurementConsumer" — the narrower successor the same author opened instead. Still open, last pinged 2026-07-30, also being marked stale.
• Deadlock accessing metric reader storage when running under Flask/Gunicorn with gevent workers #4345 is the report fix(metrics): Add fork-safety to SynchronousMeasurementConsumer #4767 addresses.

Worth noting that #4626 predates this specific problem: _service_instance_id_lock and the provider _handle_fork path only landed in 1.44 (#5280), so even if #4626 had merged it would not have covered the lock involved here. And the Part 1 defect — get_aggregated_resources()'s timeout not bounding the wait, because leaving the with ThreadPoolExecutor(...) block joins the worker — is independent of lock reinitialization entirely. Fixing Part 1 fixes Part 2 as a side effect, which is why I would lean toward options 1 or 2 rather than adding another lock-reinit hook.