fix: harden media server under load by richiemcilroy · Pull Request #2007 · CapSoftware/Cap

richiemcilroy · 2026-07-15T14:51:46Z

Summary

make media admission aware of cgroup CPU and memory limits while preserving nominal concurrency
stream large segment downloads and mux inputs through disk, preserve input ordering, abort failed transfers, and retry transient upstream failures
preserve signed child URLs for HLS and DASH audio probes with bounded cancellation and cleanup
expose resource pressure in health and structured job logs
pin the media runtime for reproducible deployments

Incident addressed

A long-running segment mux could retain multiple complete media copies while Linux page cache grew outside the process RSS signal used for admission. This allowed a Railway replica to exceed its memory limit under sustained work. A separate upstream object-storage failure could also leave a worker doing unnecessary transfer work.

Validation

full media-server suite: 142 passed, 0 failed
scoped TypeScript typecheck passed
scoped Biome check passed
production Docker image built successfully
exact-image media and cgroup suite: 21 passed, 0 failed
512 MB cgroup check throttled at 86% and rejected new work at 90.4%
2-CPU cgroup check reported the quota instead of the 16-CPU host

Greptile Summary

This PR hardens the media server against memory and CPU exhaustion by replacing RSS-based admission with real cgroup v1/v2 metrics, streaming segment downloads and concatenations through disk instead of buffering full files in memory, aborting in-flight transfers on fatal failures, and retrying transient upstream errors with exponential backoff.

New cgroup modules (container-cpu.ts, container-memory.ts) read kernel-reported limits and usage; job-manager.ts wires them into admission thresholds (throttle at 85%, reject at 90%) with a CPU utilisation signal from delta-sampled cpu.stat.
muxSegmentsAsync pipeline streams each segment to disk, returns ordered paths, early-deletes input directories to release page cache, and wraps the FFmpeg mux in withMuxMemoryGuard for a 1 s polling abort.
checkHasAudioTrack materialises HLS/MPD manifests to disk before probing with a bounded AbortSignal; resolveResourceUrl now blocks non-HTTP protocols with a post-hoc manifest scan as defence-in-depth.

Confidence Score: 5/5

Safe to merge; the hardening logic is well-tested and the two observations are conservative design choices rather than correctness defects.

The cgroup-reading modules are straightforward and have dedicated unit tests with injected paths. The streaming pipeline correctly orders segment output, aborts in-flight downloads on the first fatal error, and cleans up intermediate files eagerly. The two flagged items are edge cases that cause unnecessary retries rather than data loss or corruption.

The withMuxMemoryGuard block and the two init-file downloadUrlToFile calls in muxSegmentsAsync in video.ts are worth a second look if retry pressure under sustained load becomes observable in production.

Important Files Changed

Filename	Overview
apps/media-server/src/routes/video.ts	Major refactor: streaming segment downloads with ordered output paths, retry logic, abort propagation, and memory-guard wrapping around FFmpeg mux. Two design observations: init-file downloads don't pass the abort signal, and the memory guard may discard a valid completed mux.
apps/media-server/src/lib/job-manager.ts	Replaces flat RSS-based admission with cgroup-aware CPU and memory pressure; introduces hasCriticalMemoryPressure(). getMaxConcurrentVideoProcesses() calls getContainerMemoryMetrics() separately from getSystemResources(), causing a double filesystem read per health or status poll.
apps/media-server/src/lib/media-video.ts	Adds abortSignal threading through HLS/MPD materialization, expands MPD URL rewriting to BaseURL/Location/sourceURL/xlink:href/href, and blocks file: protocol in resolveResourceUrl with a post-hoc manifest scan.
apps/media-server/src/lib/media-audio.ts	Rewrites checkHasAudioTrack to materialise HLS/MPD inputs to disk before probing and adds retry loop with exponential backoff; cancellation and cleanup via AbortController with bounded scope.
apps/media-server/src/lib/container-cpu.ts	New module: reads cgroup v2 cpu.max and v1 cfs_quota_us/cfs_period_us for CPU limit; reads cpu.stat/cpuacct.usage for usage. Clean implementation with injectable paths for testing.
apps/media-server/src/lib/container-memory.ts	New module: reads cgroup v2 memory.max/memory.current and v1 equivalents; respects MEDIA_SERVER_MEMORY_LIMIT_MB env override. Plausible limit guard prevents phantom limits on unlimited cgroups.
apps/media-server/src/lib/media-operations.ts	Audio and probe admission now also gate on hasCriticalMemoryPressure(); straightforward addition.
apps/media-server/Dockerfile	Pins bun:1 to bun:1.3.14 for reproducibility; updates healthcheck to a promise-chain form that also catches network failures.
apps/media-server/src/routes/health.ts	Exposes getSystemResources() into the system block; status now reflects effectiveMax===0 as degraded.
docker-compose.yml	Replaces wget-based healthcheck with bun fetch across all three compose files; change is consistent and correct.

Comments Outside Diff (1)

apps/media-server/src/lib/job-manager.ts, line 581-619 (link)

getContainerCpuLimit() read twice per getSystemResources() call

getSystemResources() first calls getCpuCapacity() (which reads cpu.max / cfs_quota_us), then passes the result as cpuCapacity to getCpuPressure(), where getContainerCpuLimit() is called a second time to check cpuLimit > 0. On every /health poll or /video/status fetch, two synchronous filesystem reads are issued to the same cgroup path. The redundant read can be eliminated by using the already-computed cpuCapacity to infer whether a container limit is present: if cpuCapacity !== hostCpuCount, a limit exists and the usage path should be taken. Alternatively, getCpuCapacity() can return a structured result that includes the limit flag.

Prompt To Fix With AI

This is a comment left during a code review.
Path: apps/media-server/src/lib/job-manager.ts
Line: 581-619

Comment:
**`getContainerCpuLimit()` read twice per `getSystemResources()` call**

`getSystemResources()` first calls `getCpuCapacity()` (which reads `cpu.max` / `cfs_quota_us`), then passes the result as `cpuCapacity` to `getCpuPressure()`, where `getContainerCpuLimit()` is called a second time to check `cpuLimit > 0`. On every `/health` poll or `/video/status` fetch, two synchronous filesystem reads are issued to the same cgroup path. The redundant read can be eliminated by using the already-computed `cpuCapacity` to infer whether a container limit is present: if `cpuCapacity !== hostCpuCount`, a limit exists and the usage path should be taken. Alternatively, `getCpuCapacity()` can return a structured result that includes the limit flag.

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (11): Last reviewed commit: "fix: decode DASH resource attributes" | Re-trigger Greptile}

cursor · 2026-07-15T14:51:50Z

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

richiemcilroy · 2026-07-15T18:23:52Z