feat(rdma): add negotiated zero-copy v2 transport

[ketor]

Summary

Add an opt-in, per-connection negotiated RDMA v2 data plane while preserving TCP and the existing RDMA v1 protocol.

• v2 PUT uses IBV_WR_RDMA_WRITE_WITH_IMM into a process-wide registered receive segment.
• v2 GET advertises client {addr,rkey,len} targets; the server RDMA-WRITEs payload bytes and completes with a small ordered SEND.
• v1/v2 negotiation is connection-local. New clients probe and fall back to v1 against old/forced-v1 servers.
• HCA context/PD and large pool MRs are shared per device; QP/CQ remain per connection.
• Add active-HCA discovery, explicit local whitelists, NUMA-aware selection, bounded connection pools, idle reclaim, and an interruptible shutdown path.
• Add true windowed scalar/SG/batch PUT, GET, Range and Exist paths with strict in-order replies.
• Keep the default build portable: DFKV_WITH_RDMA=OFF has no verbs dependency and retains TCP behavior.

The design borrows the proven memory-registration, single-sided payload, topology, and submit/wait principles from Mooncake Transfer Engine without importing Mooncake Store's master/replica/object semantics into dfkv's existing MDS + consistent-hash architecture.

Protocol and compatibility

The existing device bootstrap frame gains an optional versioned DCP tail:

• legacy/plain and DCP1 connections use RDMA v1 SEND/RECV;
• DCP2 requests v2 and carries the declared max block geometry;
• wire prefix, QP metadata, slot geometry, target count/capacity, opcode, immediate flag, and completion byte length are validated fail-closed;
• DFKV_RDMA_PROTOCOL=1 forces client v1;
• DFKV_RDMA_SERVER_PROTOCOL=1 forces server v1;
• auto-v2 falls back by creating a fresh v1 connection, never by changing protocol on a live QP.

An explicit DFKV_RDMA_DEV is treated as a hard local fabric selection in both RDMA and TCP-only builds: an unsatisfied whitelist returns no transport instead of silently using TCP. Without a whitelist, each peer independently selects its first ACTIVE local HCA; host-local HCA names are not assumed to be peer identities.

Memory and lifetime model

• RecvSegment allocates aligned depth * slot_size leases from one process-wide region.
• Startup anchor endpoints register that region once on every selected shared PD.
• Pool MRs are PD-owned and survive individual connection reclaim.
• API-owned buffers use transient MRs, released after the corresponding completion; endpoint teardown owns failed/timed-out cleanup.
• Read-only Cache sources are registered without remote/local-write requirements.
• RAM-hit pins are released only on the signaled SEND completion, after all prior unsignaled WRITEs on that QP.
• Server Stop() wakes CQ waiters and joins all Serve threads before owner destruction.

Capacity and observability

The v2 slot geometry is:

slot_size = align4K(4096 + sizeof(ValueHeader) + max(declared_block, 4096))

The shared-segment budget is approximately:

slot_size * negotiated_depth * (active + pooled data connections)

New metrics expose v1/v2 connections, v2 PUT/GET writes, per-rail connections, pool/ad-hoc MR usage, registered receive-segment rails/free bytes, completion errors, active connections, and idle reclaims. Documentation includes rollout, rollback, memlock, NUMA and sizing guidance.

Validation

Final branch was rebased onto upstream/main@eadb3fd and verified as follows:

• DFKV_WITH_RDMA=ON Debug build on a real ConnectX host: 464/464 CTest passed.
• DFKV_WITH_RDMA=OFF Debug build: 406/406 CTest passed.
• ThreadSanitizer RDMA protocol/segment/loopback matrix: 40/40 passed.
• Two independent final reviews found no remaining blocking protocol or integration findings.
• Python syntax check passed for the rebased vLLM connector files.

TSan note: the unchanged baseline RamTier destructor emits a GNU 11 TSan double lock diagnostic for its condition-variable stop/join sequence. RdmaLoopback.RamTierZeroCopyServe is therefore covered by the normal 464-test ConnectX run and excluded from the 40-test race matrix; this patch does not modify ram_tier.{h,cc}.

Rollout

1. Upgrade servers while forcing DFKV_RDMA_SERVER_PROTOCOL=1; restart all changed server processes.
2. Remove the force-v1 setting, restart servers, and verify segment/connection metrics.
3. Enable auto-v2 on one client process, restart it, and verify v2 write/connection counters plus zero completion errors.
4. Expand by rank/node.
5. Roll back by forcing v1 and restarting every affected client/server. Removing the force variables also requires restarts because existing pooled QPs never renegotiate.