Add access logging + host TCP sampler to diagnose pypi download failures - #362
Merged
Merged
Conversation
Halide CI intermittently fails to reach pypi.halide-lang.org with uv "client error (Connect): operation timed out" (3 retries, ~47s). The appliance had no request-level visibility, so failures couldn't be localized. This adds two complementary, low-overhead data sources: - Caddy JSON access log for the pypi site -> dedicated caddy-logs volume, rotated, kept off the ACME/TLS log stream. Distinguishes a truncated download (status 200, size < content-length) from a request that never arrived (absent), and shows whether the server was serving other clients during an incident. - monitor/sample.sh: a network_mode:host sidecar (netmon) that samples host TCP stack counters every 15s (accept-queue drops, conntrack, SYN_RECV, retransmits, load) to net-samples.tsv. Tells us whether lost SYNs were dropped on this host vs upstream of it. - Caddy global `metrics` on 127.0.0.1:2019 for on-demand in-flight snapshots. Correlating these against a live failure showed the server serving identical requests from other runners in the same window while the failing runner's request never reached Caddy and every host drop counter stayed flat -- i.e. the loss is upstream of the host (network path / OpenStack NAT), not the server software. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
alexreinking
added a commit
to halide/Halide
that referenced
this pull request
Jul 29, 2026
The existing retry() only spanned 60s + 120s of backoff (3 attempts total), which wasn't enough to ride out every pypi.halide-lang.org connect-timeout blackhole (halide/build_bot#362). - Retry harder: 5 attempts, backoff doubling from 60s up to a 300s cap (worst case ~12 extra minutes instead of ~3). - On each failed attempt, log a UTC timestamp, the runner's egress IP (via api.ipify.org), and a path trace to pypi.halide-lang.org (tracepath/traceroute/tracert, whichever is available), folded into a collapsible log group. MIT network ops asked for an exact 5-tuple and timestamp from a live incident to correlate against their own border logs; this makes the next incident self-documenting without needing to reproduce it. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Halide CI intermittently (~10%) fails to reach
pypi.halide-lang.org, with uv reportingclient error (Connect): operation timed out(3 retries, ~47s) — on both wheel downloads and the tiny/simple/index. The appliance had no request-level visibility, so failures could not be localized.What this adds
caddy-logsvolume, rotated, kept off the ACME/TLS log stream. Distinguishes a truncated download (200withsize< content-length) from a request that never arrived (absent), and shows whether the server was serving other clients during an incident.monitor/sample.sh— anetwork_mode: hostsidecar (netmon) sampling host TCP stack counters every 15s (accept-queue drops, conntrack, SYN_RECV, retransmits, load) tonet-samples.tsv. Tells us whether lost SYNs were dropped on this host vs upstream of it.metricson127.0.0.1:2019for on-demand in-flight snapshots.What the data already showed
Correlated against a live failure (2026-07-29): during the failing runner's 47s retry window, Caddy served the identical
/simple/halide-llvm/request from three other runners (200in ~15ms), the failing runner's request never reached Caddy, and every host drop counter stayed flat (listen_drops/overflows/reqq_full_drop = 0, conntrack ~20/262144, load < 0.5).Conclusion: the packet loss is upstream of the host (network path / OpenStack floating-IP NAT), not Caddy/pypiserver/host config. This instrumentation is what let us establish that, and stays in place to triage future incidents.
🤖 Generated with Claude Code