diff --git a/docs.json b/docs.json index 474ced9a..c41ff4ef 100644 --- a/docs.json +++ b/docs.json @@ -117,6 +117,7 @@ "serverless/workers/deploy", "serverless/workers/github-integration", "serverless/storage/overview", + "serverless/development/volume-cache", "serverless/development/dual-mode-worker" ] }, diff --git a/serverless/development/optimization.mdx b/serverless/development/optimization.mdx index 5b9b39fe..9ee7d443 100644 --- a/serverless/development/optimization.mdx +++ b/serverless/development/optimization.mdx @@ -14,6 +14,7 @@ Optimization involves measuring performance with [benchmarking](/serverless/deve |----------|--------|-------------| | [Use cached models](/serverless/endpoints/model-caching) | ⬇️ Cold start (major) | Models on Hugging Face | | [Bake models into image](/serverless/workers/create-dockerfile#including-models-and-files) | ⬇️ Cold start | Private models | +| [Cache files to a network volume](/serverless/development/volume-cache) | ⬇️ Cold start | Downloaded weights, attached volume | | [Set active workers > 0](/serverless/endpoints/endpoint-configurations#active-workers) | ⬇️ Cold start (eliminates) | Latency-sensitive apps | | [Select multiple GPU types](/serverless/endpoints/endpoint-configurations#gpu-configuration) | ⬆️ Availability | Production workloads | | [Increase max workers](/serverless/endpoints/endpoint-configurations#max-workers) | ⬆️ Throughput | High concurrency | diff --git a/serverless/development/volume-cache.mdx b/serverless/development/volume-cache.mdx new file mode 100644 index 00000000..cc140537 --- /dev/null +++ b/serverless/development/volume-cache.mdx @@ -0,0 +1,80 @@ +--- +title: "Cache files to a network volume" +sidebarTitle: "Volume cache" +description: "Persist model weights and other files to an attached network volume to speed up Serverless worker cold starts." +--- + +import { ColdStartTooltip } from "/snippets/tooltips.jsx"; + +The Runpod Python SDK includes `VolumeCache`, a helper that warms local directories across Serverless workers using an attached network volume. It keeps a browsable mirror of your cache directories on the volume: on cold start it restores previously cached files into place, and after your code runs it copies newly written files back. This turns a repeated multi-GB model download on every cold start into a one-time cost per endpoint, which reduces times. + +`VolumeCache` is best-effort and self-contained, and it never affects the outcome of a job. If any part of the cache fails, the worker falls back to a normal cold start. You add it explicitly by wrapping the code that populates your cache, so nothing runs until you opt in. + +## Requirements + +- A [network volume](/storage/network-volumes#network-volumes-for-serverless) attached to your endpoint, mounted at `/runpod-volume`. The mirror is stored on the volume, so every operation is a safe no-op when no volume is mounted. +- The Runpod Python SDK installed in your worker image. +- An endpoint namespace to scope the mirror. On Serverless this defaults to `RUNPOD_ENDPOINT_ID`, which Runpod sets automatically. + +## Usage + +`VolumeCache` is a context manager. Wrapping it around your model load hydrates the cache before the block runs and syncs any changes back afterward: + +```python title="handler.py" +import runpod +from runpod.serverless import VolumeCache + +def handler(job): + ... + +# Hydrate from the volume, download only what's missing, then sync back +with VolumeCache(dirs=["/root/.cache/huggingface"]): + model = load_model() # downloads land in the cached directory + +runpod.serverless.start({"handler": handler}) +``` + +When you enter the block, `hydrate()` copies files that are missing or newer on the volume mirror into the container. When you exit, `sync()` copies files that are missing or newer in the container back onto the mirror. By default the sync runs on a background daemon thread and returns immediately, so the `with` block doesn't wait on it. A process-exit hook completes any outstanding syncs, so short-lived processes still finish syncing before they exit. + +You can also call the phases directly when they happen at different points in your worker's lifecycle: + +```python +vc = VolumeCache(dirs=["/data/models"], namespace="my-model-cache") + +vc.hydrate() # restore cached files (for example, at startup) +model = load_model() # populate the cache +vc.sync(background=False) # persist new files back to the volume, inline +``` + +## Constructor + +`VolumeCache` accepts the following arguments: + +| Argument | Default | Description | +|----------|---------|-------------| +| `dirs` | Required | A list of local directories to cache. | +| `namespace` | `RUNPOD_ENDPOINT_ID` | Isolation key for the on-volume mirror. Must be a single safe path component. | +| `volume_path` | `/runpod-volume` | Mount point of the network volume. | +| `best_effort` | `True` | When `True`, cache errors are logged and swallowed instead of raised. Set to `False` while debugging. | +| `max_workers` | `min(32, (os.cpu_count() or 4) * 4)` | Thread count for the parallel copy of large files. The work is I/O-bound, so the default oversubscribes the CPU count. | + +## How it works + +`VolumeCache` adapts its transport to the size of your files, which keeps both many-small-file caches and multi-GB weight files fast. + +- **Size-bucketed mirror:** Cached files live at `{volume_path}/.cache/{namespace}` on the volume. Files below 256 KiB are packed into a single `small.tar` archive, which collapses the per-file metadata round-trips that make many small files slow on a network volume. Larger files are copied unpacked into a `big/` subdirectory, which preserves their original relative paths so the large-file subtree stays browsable. A versioned `manifest.json`, written last, records the size and modification time of every cached file and marks the mirror as complete. A mirror without a valid, current manifest is treated as absent, so a sync self-heals. +- **Incremental large files, whole-archive small files:** Large-file transfers are diffed per file against the manifest, so unchanged files are skipped. The `small.tar` archive is repacked whole whenever any small file changes, since unpacking and re-diffing many tiny files individually is slower than the volume's per-file overhead. +- **Parallel copy:** Large-file transfers run across a thread pool sized by `max_workers`. The work is I/O-bound, so the default oversubscribes the CPU count. +- **Safety:** Symbolic links are never followed or copied. Every archive member and every large-file destination is checked to resolve inside one of your configured `dirs` before it is written, so a mirror entry can't write outside your cached directories. + +## Limitations + +- **Concurrent cold-start write amplification:** If several workers cold-start at the same time, each may miss the still-empty mirror, download the model, and sync a full copy back. There's no coordination between concurrent syncs, so the mirror reflects whichever worker synced most recently. +- **Background sync on short-lived processes:** `sync()` schedules the copy on a daemon thread. If the process exits without a normal interpreter shutdown (for example, `os._exit` or `SIGKILL`), the exit hook never runs and the sync may not complete. +- **Orphaned large files aren't pruned:** If you delete or rename a large file locally, its copy under `big/` stays on the volume. Hydration is manifest-driven and ignores it, but volume usage grows as you swap model versions. + +## Next steps + +- [Optimize your endpoints](/serverless/development/optimization): Combine the volume cache with other strategies to reduce cold starts. +- [Cached models](/serverless/endpoints/model-caching): Use Runpod's platform-level model cache for Hugging Face models. +- [Storage options](/serverless/storage/overview): Compare container disks, network volumes, and S3-compatible storage.