Skip to content

Improve js-memory leak signal scoping and confidence#19

Merged
V3RON merged 3 commits into
masterfrom
issue-17-js-memory-leak-signal
May 13, 2026
Merged

Improve js-memory leak signal scoping and confidence#19
V3RON merged 3 commits into
masterfrom
issue-17-js-memory-leak-signal

Conversation

@V3RON
Copy link
Copy Markdown
Contributor

@V3RON V3RON commented May 12, 2026

Summary

  • agent-cdp users can now run memory usage leak-signal --since SAMPLE_ID to evaluate one bounded interaction instead of accidentally scoring against older daemon samples.
  • The leak check now reports confidence, scope, and quality notes so mixed history, missing GC checkpoints, and too-few samples read as weak evidence instead of looking conclusive.
  • The heuristic now treats post-GC retained growth as the primary signal and keeps directing users to heap snapshots for confirmation.

Context

The js-memory leak signal previously scored against the full stored sample history and could produce a strong-looking result from mixed workflows. This change keeps the existing sampling workflow, adds an opt-in bounded analysis window, and reworks the scoring/output so retained post-GC growth carries the most weight while trend-only evidence is explicitly downgraded.

Backward-compatibility analysis:

  • Existing memory usage sample, list, summary, diff, trend, and leak-signal workflows continue to work.
  • memory usage leak-signal keeps the same command name and remains heuristic, but its default text output is now richer because it includes confidence/scope/quality notes.
  • The new --since SAMPLE_ID flag is additive and does not change daemon lifecycle or stored sample format.

Risks:

  • Scripts that parse the old compact memory usage leak-signal text output may need updates because the default formatter now prints confidence and quality-note text.
  • The heuristic is still based on coarse heap-usage checkpoints, so workloads without a clear baseline/action/GC sequence can still produce low-confidence noise.
  • Because bounded leak analysis is opt-in, users who keep using full-history mode may still see mixed-session warnings until they adopt --since.

Manual testing:

  • Setup: start the daemon, select a target, and leave existing sample history in place to confirm mixed-history messaging still appears.
  • Run agent-cdp memory usage sample --label baseline --gc, trigger the suspicious interaction, run agent-cdp memory usage sample --label action, then run agent-cdp memory usage sample --label cleanup --gc.
  • Run agent-cdp memory usage leak-signal and verify the output calls out full-history scope or other low-confidence quality notes when applicable.
  • Run agent-cdp memory usage leak-signal --since <baseline-sample-id> and verify the output reports bounded scope, includes confidence, and explains the post-GC retained-growth evidence for just that interaction window.
  • Expected user-visible outcome: bounded leak checks read as more trustworthy, while weak or mixed evidence is explicitly labeled as such.

Fixes #17

@V3RON
Copy link
Copy Markdown
Contributor Author

V3RON commented May 13, 2026

Manual verification against the PR branch on iOS simulator looks good.

Setup

  • built agent-cdp from this PR
  • restarted the daemon from the PR build
  • launched the React Native playground app in the iPhone 17 Pro simulator
  • drove the new Memory Leak Test screen with agent-device

Scenarios run

1. Mixed history + temporary allocation

Preloaded daemon history with unrelated samples, then ran a bounded temporary-allocation workflow.

Commands:

pnpm agent-cdp memory usage sample --label old-1
pnpm agent-cdp memory usage sample --label old-2 --gc
pnpm agent-cdp memory usage sample --label temp-baseline --gc
pnpm agent-cdp memory usage sample --label temp-action
pnpm agent-cdp memory usage sample --label temp-cleanup --gc
pnpm agent-cdp memory usage leak-signal
pnpm agent-cdp memory usage leak-signal --since jm_3

Observed:

NONE confidence:medium score:0 scope:full-history samples:5
note: This result spans all stored samples in the daemon. Mixed workflows can skew the signal; rerun with --since SAMPLE_ID for one bounded check.
NONE confidence:medium score:0 scope:bounded samples:3
note: Only 3 samples in this window; leak confidence is limited.

This matches the intended behavior: full-history mode warns about mixed samples, while the bounded temporary-allocation window stays negative.

2. Retained allocation positive case

Used the Allocate Retained 20 MB x5 action to force obvious retained growth.

Commands:

pnpm agent-cdp memory usage sample --label retained-baseline --gc
pnpm agent-cdp memory usage sample --label retained-action
pnpm agent-cdp memory usage sample --label retained-cleanup --gc
pnpm agent-cdp memory usage leak-signal --since jm_6
pnpm agent-cdp memory usage leak-signal --since jm_6 --verbose

Observed compact output:

HIGH confidence:medium score:5 scope:bounded samples:3
note: Only 3 samples in this window; leak confidence is limited.

Observed verbose output:

JS Memory Leak Signal:
  Level:  HIGH (score: 5)
  Confidence: MEDIUM
  Scope: bounded window (3 samples, jm_6 -> jm_8)

Evidence:
  - Window jm_6 -> jm_8: baseline 6.7 MB, peak 108.0 MB, latest 107.0 MB.
  - Post-GC checkpoint jm_8 is +100.3 MB vs baseline after +0.9 MB of recovery from the peak.
  - Most peak growth remains after a GC-assisted checkpoint, which is a strong retention signal.
  - Latest heap is still 1500.0% above the baseline.

Quality notes:
  - Only 3 samples in this window; leak confidence is limited.
  - Samples oscillate instead of following a clean progression, which lowers confidence.

Caveat: This is a heuristic signal based on heap usage checkpoints, not proof of a memory leak. Use heap snapshots for confirmation.

This is the strongest confirmation for the PR: the positive result is driven by the post-GC retained-growth checkpoint, and the output explicitly communicates confidence, scope, quality notes, and caveat text.

3. Trend-only bounded run without a GC checkpoint

Ran a bounded two-sample retained-growth window without any GC-assisted sample in the window.

Commands:

pnpm agent-cdp memory usage sample --label trend-baseline
pnpm agent-cdp memory usage sample --label trend-action
pnpm agent-cdp memory usage leak-signal --since jm_11
pnpm agent-cdp memory usage leak-signal --since jm_11 --verbose

Observed:

MEDIUM confidence:low score:3 scope:bounded samples:2
note: Only 2 samples in this window; leak confidence is limited.

Verbose quality note correctly called out the missing GC checkpoint:

- No GC-assisted checkpoint in this window; the signal is trend-based and lower confidence.

This also matches the PR intent.

4. Too few samples

Command:

pnpm agent-cdp memory usage leak-signal --since jm_12

Observed:

NONE confidence:low score:0 scope:bounded samples:1
note: Capture a bounded baseline and follow-up sample before using leak-signal.

5. Invalid sample id

Command:

pnpm agent-cdp memory usage leak-signal --since does-not-exist

Observed:

Sample does-not-exist not found

Conclusion

Manual device testing supports the change set:

  • --since SAMPLE_ID correctly scopes analysis to one bounded interaction window
  • full-history mode warns about mixed daemon history
  • retained post-GC growth drives strong signals
  • trend-only and too-few-sample windows are explicitly lower-confidence
  • invalid --since fails clearly

One minor observation: the mixed full-history temporary case reported confidence:medium rather than low, even though it did include the correct mixed-history warning. Worth double-checking whether that confidence level is exactly what we want, but the core behavior looks correct.

@V3RON V3RON merged commit d598ba8 into master May 13, 2026
4 checks passed
@V3RON V3RON deleted the issue-17-js-memory-leak-signal branch May 13, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scope and harden js-memory leak detection

1 participant