EOF When pushing a large image to a remote repository

[sachouche]

Actual Behavior

Hi team,

• I am having an issue when using "docker push" from my local box to an Oracle Cloud registry.
• Most of the layers would succeed with the exception of 4 layers which are larger than 80MB.
• These layers keep retrying and eventually the whole push command fails with an EOF
• Rancher logs don't show any error related to the push command.

My ENV:

• Mac OS M2 pro.
• Rancher version: 1.14.1

Remote Repo: phx.ocir.io

Would appreciate if you could help me get over this issue as the Google search returns mostly fixes for Docker's based tools which I don't find within the Rancher's bin directory.

Steps to Reproduce

docker push phx.ocir.io//:
OR
nerdctl

REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
phx.ocir.io// 381ee68f9217 40 hours ago linux/amd64 4.093GB 1.833GB

Result

Using Dockerd(moby)
5f70bf18a086: Layer already exists
1531edb90414: Layer already exists
b890a7bd6f13: Layer already exists
a9e536e25695: Layer already exists
a9c4c539560c: Layer already exists
4f1127fcf098: Layer already exists
63f54e534871: Layer already exists
c7cb49f197a6: Layer already exists
cd17b1548c5b: Layer already exists
e2a70b92c686: Layer already exists
e697a158b8d7: Layer already exists
4611c374932e: Layer already exists
139726fa9140: Layer already exists
3ca0bb783055: Layer already exists
9d56e05bddc3: Layer already exists
dbbfffb09792: Layer already exists
4a7182279151: Layer already exists
b07cf3591f47: Layer already exists
b3c6d7267c88: Pushing [==================================================>] 341.5MB
8ac06828bc68: Layer already exists
ba37115a2af8: Layer already exists
02248c474edf: Layer already exists
51867ea8cb29: Layer already exists
b0f33d61a87d: Layer already exists
0e3b348a50c1: Layer already exists
e400ff8cd4f5: Layer already exists
d099ab2193c3: Layer already exists
d7a8c26a4663: Layer already exists
fb21852b2463: Layer already exists
52f242e3577a: Layer already exists
3a194b2f557f: Layer already exists
e7b74c04f015: Layer already exists
691ee7d8a97c: Layer already exists
8d957ef797ef: Layer already exists
5aa01872ea42: Pushing [==================================================>] 402.3MB
e1b64e3a0372: Layer already exists
9c898758c962: Layer already exists
c97a5e6af785: Layer already exists
7bb76c57aee1: Layer already exists
1b34d827b724: Layer already exists
8b66daa0c7f6: Pushing [==================================================>] 82.06MB
5be8c18ee625: Layer already exists
7c08d9940027: Layer already exists
545356f5059a: Pushing [========> ] 414.7MB/2.551GB
323da2963581: Layer already exists
aa2f171fe990: Layer already exists
7534ae9858d9: Layer already exists
54a9f9ff7ca8: Layer already exists
8900ef3aaf79: Pushed
e42549559b59: Pushing [==================================================>] 118.1MB

Large layers (more than 80MB) would keep on retrying till the command eventually fails.

Using containerd
elapsed: 61.1s total: 1.7 Gi (28.5 MiB/s)
FATA[0061] failed commit on ref "layer-sha256:128cf272b5639f8243c9e559d1f65892b8b46a9362cb0cb44d16d0b7399daaaf": failed to do request: Put "https://phx.ocir.io/v2//blobs/uploads/b8eb988a-ebc8-45f2-b0de-7a3bff4280cd?digest=sha256%3A128cf272b5639f8243c9e559d1f65892b8b46a9362cb0cb44d16d0b7399daaaf": EOF
Error: exit status 1

Expected Behavior

The image push command should succeed.

Additional Information

No response

Rancher Desktop Version

1.14.1

Rancher Desktop K8s Version

NA

Which container engine are you using?

moby (docker cli)

What operating system are you using?

macOS

Operating System / Build Version

Sequoia 15.1.1

What CPU architecture are you using?

arm64 (Apple Silicon)

Linux only: what package format did you use to install Rancher Desktop?

None

Windows User Only

No response

[sachouche]

Connected with @jandubois about this issue through slack: https://rancher-users.slack.com/archives/C0200L1N1MM/p1738348505750709

• I reported observing 401 within the debug logs (not reported by the CLI).
• @jandubois suggested to avoid the "Allowed Images" experimental feature as it internally uses a proxy.

My problem disappeared as soon as I disabled this feature and my 1.7GB layer (image was 4GB) got successfully uploaded after 20min.

[jandubois]

Maybe related issue: docker/hub-feedback#1944

[Nino-K]

Also, it might be related to: #4258