Skip to content

[Fix] Self-hosted Docker tasks fail after two minutes when SleepCheck cannot reach Docker#618

Closed
roomote-roomote[bot] wants to merge 2 commits into
developfrom
fix/bullmq-docker-access-sleep-check-04y3kmxk047aw
Closed

[Fix] Self-hosted Docker tasks fail after two minutes when SleepCheck cannot reach Docker#618
roomote-roomote[bot] wants to merge 2 commits into
developfrom
fix/bullmq-docker-access-sleep-check-04y3kmxk047aw

Conversation

@roomote-roomote

Copy link
Copy Markdown
Contributor

Opened on behalf of Daniel Riccio. View the task or mention @roomote-roomote for follow-up asks.

Related issue

Fixes #616

Why this PR exists

  • A maintainer explicitly invited this PR in the linked issue or discussion
  • I am a maintainer / this is internal Roomote work

On self-hosted Docker compute with the installer-generated docker-socket-proxy compose path, SleepCheck runs inside bullmq but only controller could reach Docker. Every docker inspect failed, mapped to status unknown, and healthy task runs were force-failed within ~WORKER_HEARTBEAT_STALE_MS (2 minutes) — verified operator workaround was giving bullmq the same proxy access.

What changed

  • Grant bullmq the same Docker socket-proxy path as the controller in production Compose and Coolify: DOCKER_HOST=tcp://docker-proxy:2375, membership on the internal docker-api network, and depends_on the proxy (still no raw /var/run/docker.sock on bullmq).
  • Harden the Docker adapter: missing container → stopped; daemon/CLI connectivity failures throw instead of becoming unknown.
  • SleepCheck no longer finalizes runs when instance status probing throws; it logs, clears in-flight sleep/snapshot request stamps, and retries next cycle.
  • Extend compose security + deployment artifact contracts so bullmq’s proxy wiring cannot regress unnoticed.
  • Unit tests for adapter status mapping and SleepCheck probe-failure behavior.

Impact: Self-hosted Docker task runs can stay alive past the SleepCheck heartbeat window when using the proxy-based install path. Misconfigured Docker access no longer silently fails every run.

How it was tested

  • deploy/scripts/validate-compose-security.sh — passed
  • node deploy/ci/validate-deployment-artifacts.mjs — passed
  • pnpm --filter @roomote/compute-providers exec vitest run src/adapters/docker.test.ts — 5 passed
  • pnpm exec dotenvx run -f .env.test -- pnpm --filter @roomote/bullmq exec vitest run src/scheduled-jobs/__tests__/sleep-check.test.ts — 36 passed
  • Pre-push oxlint / check-types:fast / knip — passed

Operator follow-up after upgrade: recreate bullmq, confirm DOCKER_HOST + docker inspect succeed, and run a task past 2 minutes without force-fail.

Checklist

  • The PR title follows the repo convention: [Fix], [Feat], [Improve], [Refactor], [Docs], or [Chore] followed by a user-facing description
  • This PR is small and scoped to one change
  • pnpm lint and pnpm check-types pass locally
  • I added tests or included a clear manual validation note above
  • I removed secrets, tokens, private keys, and customer data from code, logs, and screenshots
  • If this change should appear in the changelog, I ran pnpm changeset

Linked work items

Closes #616

Installer/proxy compose only gave controller DOCKER_HOST; SleepCheck in
bullmq mapped failed inspect to unknown and force-failed healthy runs.
Wire bullmq through docker-proxy, harden getInstanceStatus, skip destructve
recovery on probe errors.
@roomote-roomote

roomote-roomote Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

No code issues found. See task

Reviewed 4f62ec5

@daniel-lxs

Copy link
Copy Markdown
Member

Thanks for putting this together. Closing it for two reasons:

  1. The Compose/Coolify half of this PR landed separately via [Improve] Validate environment button for Local Docker settings #627 (shipped in v0.15.0), which resolves the outage reported in [Self-hosting]: Docker compute: bullmq has no Docker access, so SleepCheck reads every running worker as unknown and force-fails healthy task runs #616 — that's why this branch now conflicts with develop.
  2. We've decided not to take the adapter/SleepCheck behavior change. Making status-probe failures non-fatal flips the failure mode: if a container is genuinely gone but the error text isn't recognized as a missing-object error, the run could never be finalized — even the hard-limit path depends on a successful probe. We'd rather keep that failure visible than risk runs that hang indefinitely.

The CI assertions guarding bullmq's Docker-proxy wiring are a good idea on their own and would be welcome as a separate, focused PR.

@daniel-lxs daniel-lxs closed this Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Self-hosting]: Docker compute: bullmq has no Docker access, so SleepCheck reads every running worker as unknown and force-fails healthy task runs

1 participant