[Bug]: RegistryClient blob push retry restarts upload from byte 0, causing ECR BLOB_UPLOAD_INVALID (416)

[sadysnaat]

Summary

RegistryClient blob push to AWS ECR can fail with HTTP 416 and error code BLOB_UPLOAD_INVALID when a large blob upload is interrupted or retried. ECR reports that the client restarted from byte 0 while the upload session already had bytes committed.

This is the underlying implementation issue tracked upstream in the container CLI repo: apple/container#1895

---

Steps to reproduce

Reproducible via container image push (which uses this library). Minimal flow:

aws ecr get-login-password --region <region> \
  | container registry login --username AWS --password-stdin \
    <account-id>.dkr.ecr.<region>.amazonaws.com

container build \
  --platform linux/amd64,linux/arm64 \
  -t <account-id>.dkr.ecr.<region>.amazonaws.com/<repo>:<tag> \
  .

container image push <account-id>.dkr.ecr.<region>.amazonaws.com/<repo>:<tag>

Most likely with large multi-arch images when a blob upload is partially sent then retried.

---

Current behavior

Example error:

416 Range Not Satisfiable
{"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"First byte of the layer part is 0 instead of 11304960"}]}

---

Suspected root cause

Sources/ContainerizationOCI/Client/RegistryClient+Push.swift

Blob push flow:

1. POST /v2/<name>/blobs/uploads/ → obtain upload session URL
2. Monolithic PUT of entire blob to .../blobs/uploads/<uuid>?digest=sha256:...

The code passes a bodyClosure that resets the stream on retry:

// We have to pass a body closure rather than a body to reset the stream when retrying.
let bodyClosure = {
    let stream = try streamGenerator()
    return HTTPClientRequest.Body.stream(stream, length: .known(descriptor.size))
}
try await request(components: components, method: .PUT, bodyClosure: bodyClosure, ...)

Sources/ContainerizationOCI/Client/RegistryClient.swift

The generic request() loop retries by re-invoking bodyClosure() (stream from byte 0) against the same request URL:

while true {
    request.body = try bodyClosure()
    let _response = try await client.execute(request, ...)
    // retries on 5xx (shouldRetry) and network errors
    continue
}

For blob uploads, if PUT partially succeeds then retries (network error or generic retry), the registry expects continuation from the last committed byte, not a restart from byte 0.

Similar issues in other OCI clients:

• google/go-containerregistry#431
• GoogleContainerTools/kaniko#607

---

Expected behavior

On blob upload failure or BLOB_UPLOAD_INVALID:

1. Abandon the stale upload session and issue a fresh POST /blobs/uploads/, then retry the full blob; or
2. Resume via distribution-spec chunked upload (GET upload status + PATCH with Content-Range).

Generic request() retry must not restart a blob PUT from byte 0 on the same upload session URL.

---

Suggested fix

1. Refactor push() in RegistryClient+Push.swift to separate initiateBlobUpload() and uploadBlob().
2. Wrap blob upload in an outer retry loop; on BLOB_UPLOAD_INVALID / HTTP 416, always start a new upload session (POST).
3. Disable or bypass generic request() retry for blob PUT, or pass retryOptions: nil for that call.
4. Add tests in Tests/ContainerizationOCITests/ simulating partial upload + 416 → successful retry with new session.
5. (Optional) Implement PATCH + Content-Range resume per the distribution spec.

---

Related

• Upstream report (container CLI): [Bug]: container image push to ECR fails with 416 BLOB_UPLOAD_INVALID after partial blob upload retry container#1895
• Related container issues: [Bug]: container image push fails with 401 on manifest PUT to ECR; pull from same registry succeeds container#1707 (ECR manifest PUT 401), [Bug]: container image push fails to register manifest digests referenced in OCI Index, causing 404s on pull container#1001 (OCI index manifest registration)