Skip to content

rendezvous: bound the wait for a dial, so a dead path is noticed - #51

Merged
Bitflash-sh merged 1 commit into
mainfrom
fix/rendezvous-keepalive
Jul 30, 2026
Merged

rendezvous: bound the wait for a dial, so a dead path is noticed#51
Bitflash-sh merged 1 commit into
mainfrom
fix/rendezvous-keepalive

Conversation

@Bitflash-sh

Copy link
Copy Markdown
Owner

Fixes the node going deaf after a while — the thing behind the memory note "relay caído não é redescoberto em runtime", except the relay is not what dies.

What happens

A node registers at a relay and waits to be dialled. That wait had no bound, and ConnectTo() clears the socket timeout deliberately: a registration is idle by design and may sit for a long time.

What it cannot survive is the path dying with neither end saying so — the ordinary case, not the exotic one. A NAT drops the idle mapping, a relay restarts, a route changes. No FIN arrives, nothing becomes readable, recv() never returns. The thread stays alive and asleep, so ThreadBtfAccept never reaches the continue that would register again.

if (!btf::RvServiceWaitPaired(rv))   // ← parked here forever
{
    btf::RvClose(rv);
    continue;                         // ← never reached
}

The node is unreachable and every outward sign says it is healthy.

Caught in production on the bootstrap seed: 27 minutes without a block, eight behind the network, the relay reachable throughout, a restart fixing it instantly. Nothing in the log marked the moment it went deaf — the same shape as #29, #30, #42 and #46.

Keepalive does not fix this

It was my first attempt and it is measured, not assumed. With probes every 5s and a 2-probe limit, a socket whose replies were being dropped was still ESTAB after four minutes, probe counter stuck at zero:

   30s  timer:(keepalive,,0)
   60s  timer:(keepalive,644ms,0)
  ...
  240s  timer:(keepalive,1.608sec,0)
  VERDICT: still blocked after 240s

That approach is not in this PR. The bound has to be ours, at the one place that knows what waiting means.

The change

RvServiceWaitPaired takes a timeout. Expiring is not an error — it means "nobody came, go round again", which the loop already handles. Cost is one reconnect per relay per five minutes. The socket returns to blocking once a dial lands, because from then on it is a data tunnel and tunnels are allowed to sit quiet.

Verification

Path killed silently — dropping only the relay's replies, so no RST or FIN is ever seen. Test build bound at 60s:

15:59:12  path killed
     40s  rendezvous: no dial at 92.246.128.180:8434 within 60s
          (or it dropped us), re-registering
          registrations 1 → 2, node carried on

Under the same treatment before this change, the node was still blocked past four minutes with no log line at all.

  • Linux: recovers on its own, logs the reason
  • Windows: registers and syncs normally, no regression
  • Production value is 300s; 60s was only to make the test converge

Not covered here

The relay's own side. bitflashd holds a registration socket per waiting client and has the mirror-image problem — a client whose path dies leaves a dead entry that the relay then hands out, which is one source of "ghost" addresses. The relay is built from a separate tree on the VPS and redeploying it is an infrastructure change, so it belongs in its own piece of work rather than being folded in here.

A node registers at a relay and waits to be dialled. The wait
had no bound, and ConnectTo() clears the socket timeout on
purpose -- a registration is idle by design and may sit for a
long time.

What that cannot survive is the path dying with neither end
saying so, which is the ordinary case and not the exotic one:
a NAT drops the idle mapping, a relay restarts, a route
changes. No FIN arrives, nothing becomes readable, and recv()
never returns. The thread stays alive and asleep, so
ThreadBtfAccept never reaches the `continue` that would
register again. The node is unreachable and every outward sign
says it is fine.

Caught on the bootstrap seed: 27 minutes without a block,
eight behind the network, relay reachable the whole time, and
a restart fixing it instantly. Nothing in the log marked the
moment it went deaf.

I tried TCP keepalive first and it does not work here. With
probes every 5s and a 2-probe limit, a socket whose replies
were being dropped was still ESTAB after four minutes with the
probe counter stuck at zero. It is measured, not assumed, and
that approach is not in this change.

So the bound is ours to enforce, at the one place that knows
what waiting means. RvServiceWaitPaired takes a timeout;
expiring is not an error but "nobody came, go round again",
which the loop above already handles. Cost is one reconnect
per relay per five minutes.

Verified by killing a registration's path silently -- dropping
only the relay's replies, so no RST or FIN is ever seen. With
a 60s bound for the test:

  rendezvous: no dial at 92.246.128.180:8434 within 60s
              (or it dropped us), re-registering

Registrations went 1 -> 2 and the node carried on. Before this,
the same treatment left it blocked past four minutes with no
log line at all.

The socket goes back to blocking once a dial lands, because
from that point it is a data tunnel and a tunnel is allowed to
sit quiet between messages.
@Bitflash-sh

Copy link
Copy Markdown
Owner Author

Branch name is a leftover: it says fix/rendezvous-keepalive because keepalive was where I started. The measurement above is why it is not what shipped. Renaming the branch would break the PR, so it stays — the code has no keepalive in it.

@Bitflash-sh
Bitflash-sh merged commit df4f615 into main Jul 30, 2026
@Bitflash-sh Bitflash-sh mentioned this pull request Jul 30, 2026
Bitflash-sh added a commit that referenced this pull request Jul 30, 2026
Rendezvous no longer waits forever for a dial (#51), keys can be
exported and imported as text (#49), and mining uses every core
instead of one (#52).

#51 is the reason not to sit on this: without it a node goes
silently deaf after a while and only a restart brings it back.
Bitflash-sh added a commit that referenced this pull request Jul 30, 2026
…t check (#55)

A connection whose path dies silently -- NAT drops an idle mapping, a relay
restarts, a route changes -- never delivers FIN. Nothing becomes readable,
no error is raised, and the node simply stops hearing from that peer while
the socket looks perfectly healthy. #51 bounded this for the rendezvous
registration; the connection that comes out of the rendezvous had the same
hole, and so did every ordinary peer.

Stamp nLastRecv and nLastSend on the socket path and act on them:

  - nothing at all within 60s of connecting, disconnect. In testing this
    fired twice per run against dead .btf descriptors handed out by the
    relay, which is where the ghosts have been coming from.
  - queued to send for 10 minutes with nothing going out, disconnect.
  - nothing received for 30 minutes, disconnect. A healthy peer relays a
    block inventory about every two minutes, so this is fifteen intervals
    of silence. Bitcoin used ninety minutes against ten-minute blocks.

Disconnecting is the whole cure: reconnection already works, and a peer we
cannot hear is worth what no peer is worth.

A node with nothing to relay would now look dead to a peer applying the
same rule, so send "ping" after ten minutes of having said nothing. It
carries no payload and expects no answer -- arriving is the entire content,
because the receive path has already stamped nLastRecv by the time it is
parsed. Requiring an answer would have read every node predating this
change as dead, since those ignore unknown commands rather than replying.

Verified on the live network: synced 0 to 3790 with five peers and no
spurious disconnects, then again with the interval cut to 20s, which put 54
pings on the wire with no peer dropping us and no exceptions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant