Duplicate put requests to registry trigger errors

[jtama]

Environment:

• Jib version: Jib core 0.27.0
• Build tool: maven 3.9.1
• OS: Ubuntu 22.04.4 LTS (in a container)

Description of the issue:
During our ci pipeline, we have an error occuring randomly. Jib triggers the same patch request which generates an error.

We know that the first patch takes a bit long to (1m12s) successfully process, but in the mean time Jib sends a second patch requests. This second request is the one that generates an error.

Expected behavior:

Should ends up fine.

Steps to reproduce:

Unable to reproduce surely the error.

Log output:

Mon, 01 Jul 2024 09:05:44 GMT Caused by: com.google.cloud.tools.jib.http.ResponseException: 404 Not Found
Mon, 01 Jul 2024 09:05:44 GMT PATCH https://.................../blobs/uploads/f7b334b6-be1f-4bfc-bd12-ca739c994a9b?_state=Z4RE2P18shUE5tW_Pk_XmXKYTOAtUlOoUA_G5isOd3l7Ik5hbWUiOiJwcm92b2x5L2RhdGEtdmlydCIsIlVVSUQiOiJmN2IzMzRiNi1iZTFmLTRiZmMtYmQxMi1jYTczOWM5OTRhOWIiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDctMDFUMDk6MDQ6MTUuMTQ1NTc4NzcxWiJ9
Mon, 01 Jul 2024 09:05:44 GMT {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"blob upload invalid"}]}
Mon, 01 Jul 2024 09:05:44 GMT
Mon, 01 Jul 2024 09:05:44 GMT at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:355)
Mon, 01 Jul 2024 09:05:44 GMT at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:266)
Mon, 01 Jul 2024 09:05:44 GMT at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:138)
Mon, 01 Jul 2024 09:05:44 GMT ... 48 more
Mon, 01 Jul 2024 09:05:44 GMT Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
Mon, 01 Jul 2024 09:05:44 GMT PATCH https://........................./blobs/uploads/f7b334b6-be1f-4bfc-bd12-ca739c994a9b?_state=Z4RE2P18shUE5tW_Pk_XmXKYTOAtUlOoUA_G5isOd3l7Ik5hbWUiOiJwcm92b2x5L2RhdGEtdmlydCIsIlVVSUQiOiJmN2IzMzRiNi1iZTFmLTRiZmMtYmQxMi1jYTczOWM5OTRhOWIiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDctMDFUMDk6MDQ6MTUuMTQ1NTc4NzcxWiJ9
Mon, 01 Jul 2024 09:05:44 GMT {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"blob upload invalid"}]}
Mon, 01 Jul 2024 09:05:44 GMT
Mon, 01 Jul 2024 09:05:44 GMT at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
Mon, 01 Jul 2024 09:05:44 GMT at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)
Mon, 01 Jul 2024 09:05:44 GMT at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:349)
Mon, 01 Jul 2024 09:05:44 GMT ... 50 more

Additional Information: The registry is a privately managed harbor

[chanseokoh]

I don't think they are duplicate PATCH requests. It's just that Jib does concurrent blob uploads. I think for whatever reason, the harbor registry sometimes fails to handle concurrent uploads correctly. I guess retrying Jib in a loop or even completely disabling concurrency in Jib (slow, of course, particularly bad for first-time build and uploads) would be a workaround.

[ddittmar]

I don't think they are duplicate PATCH requests. It's just that Jib does concurrent blob uploads. I think for whatever reason, the harbor registry sometimes fails to handle concurrent uploads correctly. I guess retrying Jib in a loop or even disabling concurrency in Jib (slow, of course, particularly for initial uploads) would be a workaround.

Hi @chanseokoh ... I think this looks like the problem we have too. I could not find howto "disable concurrency" in Jib.

[chanseokoh]

@ddittmar to disable Jib concurrency, set the system property jib.serialize to false, like -Djib.serialize=false on the command line. See this example. Note it's a system property only; no Maven or Gradle configuration for this Jib system property.

[ldetmer]

@ddittmar / @jtama does disabling concurrency solve this issue for you?

[ddittmar]

@ldetmer no ... the answer was a configuration of our Harbor registry ... our Harbor use an S3 compatible backend and we basically had to tell Harbor that it should wait until all the S3 nodes have written the data. Sorry I don't have more information because I don't configure our harbor. That's what I've heard from our administrators.

[jtama]

I can't say for sure, but we also have an managed harbor with S3 backend storage, so it could be that this is the exact same issue. Let me check please (If have enough rights)

[jtama]

So the answer is no, it did not, but as I said just before, it could be that the fault is not on jib side...