rendezvous: bound the wait for a dial, so a dead path is noticed - #51
Merged
Conversation
A node registers at a relay and waits to be dialled. The wait
had no bound, and ConnectTo() clears the socket timeout on
purpose -- a registration is idle by design and may sit for a
long time.
What that cannot survive is the path dying with neither end
saying so, which is the ordinary case and not the exotic one:
a NAT drops the idle mapping, a relay restarts, a route
changes. No FIN arrives, nothing becomes readable, and recv()
never returns. The thread stays alive and asleep, so
ThreadBtfAccept never reaches the `continue` that would
register again. The node is unreachable and every outward sign
says it is fine.
Caught on the bootstrap seed: 27 minutes without a block,
eight behind the network, relay reachable the whole time, and
a restart fixing it instantly. Nothing in the log marked the
moment it went deaf.
I tried TCP keepalive first and it does not work here. With
probes every 5s and a 2-probe limit, a socket whose replies
were being dropped was still ESTAB after four minutes with the
probe counter stuck at zero. It is measured, not assumed, and
that approach is not in this change.
So the bound is ours to enforce, at the one place that knows
what waiting means. RvServiceWaitPaired takes a timeout;
expiring is not an error but "nobody came, go round again",
which the loop above already handles. Cost is one reconnect
per relay per five minutes.
Verified by killing a registration's path silently -- dropping
only the relay's replies, so no RST or FIN is ever seen. With
a 60s bound for the test:
rendezvous: no dial at 92.246.128.180:8434 within 60s
(or it dropped us), re-registering
Registrations went 1 -> 2 and the node carried on. Before this,
the same treatment left it blocked past four minutes with no
log line at all.
The socket goes back to blocking once a dial lands, because
from that point it is a data tunnel and a tunnel is allowed to
sit quiet between messages.
Owner
Author
|
Branch name is a leftover: it says |
Merged
This was referenced Jul 30, 2026
Bitflash-sh
added a commit
that referenced
this pull request
Jul 30, 2026
…t check (#55) A connection whose path dies silently -- NAT drops an idle mapping, a relay restarts, a route changes -- never delivers FIN. Nothing becomes readable, no error is raised, and the node simply stops hearing from that peer while the socket looks perfectly healthy. #51 bounded this for the rendezvous registration; the connection that comes out of the rendezvous had the same hole, and so did every ordinary peer. Stamp nLastRecv and nLastSend on the socket path and act on them: - nothing at all within 60s of connecting, disconnect. In testing this fired twice per run against dead .btf descriptors handed out by the relay, which is where the ghosts have been coming from. - queued to send for 10 minutes with nothing going out, disconnect. - nothing received for 30 minutes, disconnect. A healthy peer relays a block inventory about every two minutes, so this is fifteen intervals of silence. Bitcoin used ninety minutes against ten-minute blocks. Disconnecting is the whole cure: reconnection already works, and a peer we cannot hear is worth what no peer is worth. A node with nothing to relay would now look dead to a peer applying the same rule, so send "ping" after ten minutes of having said nothing. It carries no payload and expects no answer -- arriving is the entire content, because the receive path has already stamped nLastRecv by the time it is parsed. Requiring an answer would have read every node predating this change as dead, since those ignore unknown commands rather than replying. Verified on the live network: synced 0 to 3790 with five peers and no spurious disconnects, then again with the interval cut to 20s, which put 54 pings on the wire with no peer dropping us and no exceptions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the node going deaf after a while — the thing behind the memory note "relay caído não é redescoberto em runtime", except the relay is not what dies.
What happens
A node registers at a relay and waits to be dialled. That wait had no bound, and
ConnectTo()clears the socket timeout deliberately: a registration is idle by design and may sit for a long time.What it cannot survive is the path dying with neither end saying so — the ordinary case, not the exotic one. A NAT drops the idle mapping, a relay restarts, a route changes. No FIN arrives, nothing becomes readable,
recv()never returns. The thread stays alive and asleep, soThreadBtfAcceptnever reaches thecontinuethat would register again.The node is unreachable and every outward sign says it is healthy.
Caught in production on the bootstrap seed: 27 minutes without a block, eight behind the network, the relay reachable throughout, a restart fixing it instantly. Nothing in the log marked the moment it went deaf — the same shape as #29, #30, #42 and #46.
Keepalive does not fix this
It was my first attempt and it is measured, not assumed. With probes every 5s and a 2-probe limit, a socket whose replies were being dropped was still ESTAB after four minutes, probe counter stuck at zero:
That approach is not in this PR. The bound has to be ours, at the one place that knows what waiting means.
The change
RvServiceWaitPairedtakes a timeout. Expiring is not an error — it means "nobody came, go round again", which the loop already handles. Cost is one reconnect per relay per five minutes. The socket returns to blocking once a dial lands, because from then on it is a data tunnel and tunnels are allowed to sit quiet.Verification
Path killed silently — dropping only the relay's replies, so no RST or FIN is ever seen. Test build bound at 60s:
Under the same treatment before this change, the node was still blocked past four minutes with no log line at all.
Not covered here
The relay's own side.
bitflashdholds a registration socket per waiting client and has the mirror-image problem — a client whose path dies leaves a dead entry that the relay then hands out, which is one source of "ghost" addresses. The relay is built from a separate tree on the VPS and redeploying it is an infrastructure change, so it belongs in its own piece of work rather than being folded in here.