docs: Document Serverless worker fitness checks (SDK v1.9.0)#705
docs: Document Serverless worker fitness checks (SDK v1.9.0)#705promptless[bot] wants to merge 3 commits into
Conversation
… (PR #536) A failed fitness check now force-kills the worker via os._exit(1) instead of sys.exit(1), so the previous example that caught SystemExit around run_fitness_checks() no longer works. Call check functions directly and add a warning about the runner's uncatchable force-kill.
![promptless[bot]](https://avatars.githubusercontent.com/in/979718?s=60&v=4){kind=small}
|
|
||
| ## Register a custom fitness check | ||
|
|
||
| Use the `@runpod.serverless.register_fitness_check` decorator to register your own checks. A check is a function that raises an exception (typically `RuntimeError`) to signal failure. If it returns without raising, the check passes. Both synchronous and asynchronous functions are supported. |
There was a problem hiding this comment.
Sourced the @runpod.serverless.register_fitness_check decorator API, sync/async support, startup-only execution in registration order, and the failure behavior (worker exits code 1, marked unhealthy) from the upstream worker_fitness_checks.md doc. Rewritten to fit Runpod docs style rather than copied.
Source: https://github.com/runpod/runpod-python/blob/main/docs/serverless/worker_fitness_checks.md
| """Verify required environment variables are set.""" | ||
| required_vars = ["API_KEY", "MODEL_PATH", "CONFIG_URL"] | ||
| missing = [var for var in required_vars if not os.environ.get(var)] | ||
|
|
There was a problem hiding this comment.
Sourced the automatic built-in system checks and their environment variables/defaults (RUNPOD_MIN_MEMORY_GB, RUNPOD_MIN_DISK_PERCENT, RUNPOD_NETWORK_CHECK_TIMEOUT, RUNPOD_MIN_CUDA_VERSION, RUNPOD_GPU_BENCHMARK_TIMEOUT, GPU test tuning vars) and the disable flags (RUNPOD_SKIP_AUTO_SYSTEM_CHECKS, RUNPOD_SKIP_GPU_CHECK) from the upstream doc.
Source: https://github.com/runpod/runpod-python/blob/main/docs/serverless/worker_fitness_checks.md
There was a problem hiding this comment.
(Line 7)
Confirmed fitness checks shipped in the runpod-python v1.9.0 release, which sets the SDK version requirement (runpod>=1.9.0) noted in the release-notes entry and the new page.
Source: https://github.com/runpod/runpod-python/releases/tag/v1.9.0
| asyncio.run(main()) | ||
| ``` | ||
|
|
||
| <Warning> |
There was a problem hiding this comment.
PR #536 routes the unhealthy exit in rp_fitness.py through a new _terminate_unhealthy helper that calls os._exit(1) instead of sys.exit(1). Because os._exit terminates the process without raising SystemExit or running except/finally blocks, I rewrote the local-testing example to call check functions directly and added a <Warning> that run_fitness_checks() force-kills the worker uncatchably.
Source: runpod/runpod-python#536
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Open this suggestion in Promptless to view citations and reasoning process
Adds a new Serverless development page documenting custom fitness checks (the register_fitness_check decorator API) and the automatic built-in system checks with their environment-variable configuration, introduced in runpod-python v1.9.0. Also adds a release-notes entry. Updated the "Test fitness checks locally" section to reflect runpod-python PR #536: a failed check now force-kills the worker via os._exit(1) (not sys.exit(1)), so the failure is uncatchable and the prior SystemExit-based example was corrected to call check functions directly, with a warning about the runner's force-kill.
Trigger Events
Tip: Assign suggestions to team members in the Promptless dashboard to claim work 👥