Skip to content

SSE client drops every ~16s on stable 1.92.1 — reconnect backoff runaways to 120s (no keepalive, no 429 handling) #606

Description

@liuahiqun8888

SSE client drops every ~16s on stable @evomap/evolver@1.92.1 — reconnect backoff runaways to 120s (no keepalive, no 429 handling)

Summary

On the current stable @evomap/evolver@1.92.1 (published 2026-07-16), the daemon's SSE connection to the hub drops roughly every 16 seconds, and the client's reconnect backoff then runs away exponentially (5s → 10s → … → 120s) and never recovers — so the event stream is effectively dead until something external resets it.

I reproduced and root-caused this locally. This is filed as an Issue (not a PR) because src/gep/a2aProtocol.js in this repo appears to be the obfuscated/minified release artifact (the file I fetched decodes to _0x… obfuscated code, not source), so I can't open a clean PR diff against it. See "Repo state note" below.

Environment

  • Package: @evomap/evolver@1.92.1 (latest dist-tag, 2026-07-16)
  • SSE transport: eventsource package (v4.x) via require('eventsource') inside src/gep/a2aProtocol.jshubOpenEventStream() / startEventStream()
  • Hub: https://evomap.ai/a2a/events/stream?node_id=…&duration_ms=…
  • OS: win32 (Node 22)

Symptoms

  1. [SSE] Event stream connected appears, then ~16s later the socket is killed (intermediate proxy/firewall drops idle long-lived streams).
  2. onerror fires; reconnect is scheduled via _sseReconnectMs which doubles each error and is never reset on a successful (re)connect.
  3. Over a few minutes the reconnect gap climbs to the 120s cap and stays there → SSE is "fake-dead".
  4. Hub rate-limits SSE connections hard (reproduced frequent HTTP 429), which the client does not honor (no Retry-After handling, no jitter) → tight retry storm makes the 429s worse.

Root causes (all reproduced locally)

  1. No client-side keepalive / idle probe. When the hub is idle it sends zero bytes, so the proxy kills the idle socket at ~16s. A client ping/heartbeat on the SSE socket is impossible because SSE is server→client only — the fix must come from the hub sending periodic : keepalive comment frames.
  2. _sseReconnectMs only grows, never resets. The backoff starts at 5000ms and doubles on every error, but _sseReconnectMs is not reset to base on a successful onopen, so it saturates at 120000ms and stays there.
  3. No 429 / Retry-After handling, no jitter. Rapid reconnects trigger hub 429s; the client ignores them and retries immediately, creating a reconnect storm across nodes.

Proposed fixes

Client (a2aProtocol.js, once source is available):

  • Reset _sseReconnectMs to a base (_sseReconnectBaseMs, e.g. 5000) inside onopen on every successful (re)connect — this is the core runaway fix.
  • In onerror, honor HTTP 429 / Retry-After, apply ±30% jitter, and cap at _sseMaxReconnectMs to avoid synchronized storms.

Server (hub, /a2a/events/stream) — required for 100% uptime:

  • Send a : keepalive\n\n comment frame every ~10s (comment frames don't trigger onmessage but keep the TCP socket non-idle, dodging the 16s proxy kill).
  • Add X-Accel-Buffering: no + Cache-Control: no-transform so nginx/CDN don't buffer the stream.
  • Return Retry-After on 429 to coordinate with the client backoff.

Note: the existing POST /a2a/heartbeat is a separate TCP connection (node registration) and does not keep the SSE socket alive — it's a different path.

Local diagnostics

  • curl -N against /a2a/events/stream returned HTTP 429 in ~1.6s with no keepalive bytes in 22s → confirms hub sends nothing when idle and rate-limits aggressively.
  • Observed pre-patch reconnect gaps: 7s → 5s → 12s → 48s → 118s (runaway to the 120s cap). Post local client-side patch (bounded backoff + onopen reset): stable 2–4s, no 429 storm, no crash.

Repo state note (why Issue, not PR)

When I fetched src/gep/a2aProtocol.js from main it decodes to obfuscated code (const _0x1f36f7=_0x447a;…), so there is no readable source to diff against for a PR. Meanwhile, test files in this repo (test/heartbeatResilienceRound3.test.js, test/issue600EventSourceFallback.test.js) suggest the SSE client has been reworked on a 2.0.0 line (fetch-based stream + a round-3 audit that reset backoff on wake).

Questions for maintainers:

  • Is the fix already shipping in the 2.0.0-beta series (latest 2.0.0-beta.5, 2026-07-21)? If so, can stable 1.92.1 get a backport, or should users move to beta?
  • Can the hub emit : keepalive frames so idle SSE survives proxies?

Local workaround (for others hitting this now)

A non-invasive preload that wraps the eventsource EventSource with a bounded-backoff reconnect facade, keeping the daemon stable on 1.92.1 until an official fix lands. Applied via NODE_OPTIONS="--require sse-idle-watchdog.js". Effectively bounds reconnect to 2–4s with no 429 storm.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions