fix(s3): add HTTP/2 keep-alive and write timeouts to avoid wedged requests (#1490)

[Alexays]

Problem

Fixes #1490.

The S3 client sets no per-request timeout. When a connection stops making progress — a throttling / back-pressuring endpoint (e.g. Scaleway Object Storage) that stops reading the request or sending a response — an in-flight request wedges forever with no way to recover.

During object-disk server-side copy this is very visible: dozens of concurrent UploadPartCopy requests are multiplexed over a few HTTP/2 connections, and once one stalls, the goroutines pile up in net/http.(*http2clientStream).writeRequest (serialized per connection). A goroutine dump from a wedged create_remote showed ~218 requests stuck there, 0 network, and the backup never progressing (in watch mode the whole daemon wedges). It reproduces intermittently under load and never self-recovers.

http_idle_conn_timeout doesn't help here — it only covers idle connections, not an active request whose write/response has stalled.

Fix

Always build a custom transport and configure the HTTP/2 transport with health checks and a write timeout:

• ReadIdleTimeout = 30s + PingTimeout = 15s — PING the peer when the connection goes idle; if there's no response, the connection is closed.
• WriteByteTimeout = 60s — close the connection if a single write stalls for too long (this is exactly the writeRequest stall above).

When a connection stalls it is now dropped, the in-flight request fails, and it is retried on a fresh connection instead of blocking indefinitely. Validated against Scaleway Object Storage: object-disk backups that previously wedged indefinitely now complete.

The timeouts are conservative constants here for a minimal change; happy to expose them as s3 config options if you'd prefer.

[Slach]

Thanks — the diagnosis in #1490 holds up (218 goroutines parked in http2clientStream.writeRequest, never recovering), and dropping a stalled h2 connection is the right direction. The implementation needs changes before it can land:

1. Hardcoded constants. 30s / 15s / 60s baked into the code is the main blocker. Every other transport knob here is configurable (http_max_idle_conns*, http_write_buffer_size, http_idle_conn_timeout), these must be too — s3.http2_send_ping_timeout / http2_ping_timeout / http2_write_byte_timeout, empty = disabled.

2. Unconditional custom transport is a regression of #1376. That guard was not cosmetic. http.DefaultTransport.Clone() gives Go's defaults — MaxIdleConnsPerHost = 2. The AWS SDK's own transport (aws/transport/http/client.go) uses MaxIdleConnsPerHost = 10, MaxConnsPerHost = 2048, plus its dialer timeouts and dial tracing. Since all http_max_* default to 0, this PR silently drops every user — including plain AWS S3 — from 10 idle conns per host to 2, i.e. TCP+TLS re-established for most requests at upload_concurrency > 2. Build the transport from awshttp.NewBuildableClient().GetTransport() instead of http.DefaultTransport.

3. golang.org/x/net/http2 isn't needed. This repo is go 1.26; http.Transport.HTTP2 (Go 1.24+) exposes SendPingTimeout / PingTimeout / WriteByteTimeout for the bundled h2 stack, no extra dependency and no ConfigureTransports ordering concerns.

4. h2Err is silently swallowed — if ConfigureTransports fails the fix quietly does nothing.

5. For the record, scope: AWS S3 does not negotiate h2 over ALPN, so these settings only take effect on S3-compatible endpoints (Scaleway, MinIO, Ceph). Plain HTTP/1.1 stalls are still uncovered — no ResponseHeaderTimeout anywhere. Worth a follow-up, not a blocker here.

I'm landing an adjusted version of this with the config options and the SDK-default transport; will credit this PR. Thanks for the thorough goroutine dump — that's what made it diagnosable.