task: Performance & memory on large repositories

[DeusData]

Scope

Umbrella tracker for performance & memory on large repositories — indexing that never finishes, times out, or grows memory without bound on big codebases. Distinct from #390 (crashes/segfaults/memory-safety) and #391 (silent corruption): here indexing is too slow or too memory-hungry, not crashing or lying about success.

Sub-issues

• Linux: cbm_system_info / cbm_default_worker_count don't respect cgroup CPU/memory limits #363 — Linux: cbm_system_info / cbm_default_worker_count don't respect cgroup CPU/memory limits (container over-subscription)
• GLR ambiguity-merge is O(n²) for deeply-nested ambiguous grammars (e.g. Perl), even with the recursion-depth cap #471 — GLR ambiguity-merge is O(n²) for deeply-nested ambiguous grammars (e.g. Perl), even with the recursion-depth cap
• constant timeouts #474 — constant timeouts
• never finishes indexing large python project #524 — never finishes indexing large Python project
• Index failed in large repo #563 — index failed in large repo
• Memory leak: process grows to 50+ GB virtual memory over hours/days, crashes Windows #581 — memory leak: process grows to 50+ GB virtual memory over hours/days, crashes Windows

Related

• Dump-phase crash on large TS monorepo (v0.6.1, darwin-arm64) — pipeline completes through semantic_edges then terminates abnormally before gbuf.dump #317 — dump-phase crash on large TS monorepo (also tracked under stability task: Stability — crashes, segfaults, and memory safety during indexing (14 issues) #390)
• Feature request: --max-memory flag for RAM-constrained hosts #580 — --max-memory / CBM_MAX_MEMORY_MB budget override — tracked via PR feat(mem): CBM_MAX_MEMORY_MB explicit memory-budget override (#580) #586

Acceptance

1. Indexing of a large public repo completes within a bounded time and memory envelope (capture a baseline corpus).
2. Worker-count / memory budget honor container (cgroup) limits and the explicit override.
3. No unbounded growth over long-running sessions — reproduce the leak in a soak test before fixing.

Why one task

These share the indexing throughput + memory-budget infrastructure (worker pool sizing, GLR merge, dump phase, long-running watcher). Grouping them keeps the large-repo experience coherent.