jib gets stuck at 'processing base image' when the base image comes from local docker daemon

[EternalWind]

Description of the issue

When reading the base image from local docker daemon, if the base image is built from a dockerfile consisting of a relatively huge amount of instructions(example here), jib will stuck at 'processing base image' forever like below.

Root cause

When inspecting the base image from docker daemon, jib invokes Process.waitFor() immediately without reading from stdout. It only tries to read from stdout after waitFor() returns, i.e. the process terminates.

If the size of the output is significant enough to overwhelm the buffer, the process would become pending until someone read from its stdout to clear out the buffer. This results in a deadlock that makes waitFor() never returns.

Proposed fix

Updates the implementation of CliDockerClient.inspect() as below.

[athtsio]

Hello all,

I am facing the same issue as described, running on a Jenkins instance as gradle task with plugin on version 3.3.1. Briefly these are the stages:

1. Create a docker image, using an another existing base docker image which is located to a remote registry (creating an extra layer via a Dockerfile with some stuff on top of that base docker image).

2. Use jibDockerBuild gradle task having as:
   from { image = "docker://" + [dockerImageName that was just built on stage1] }

3. Stuck having the exact described issue. It gets stuck at 25% > processing base image {IMAGE_NAME}

This is occured like 3/5 pipeline triggers and I need to find a workaround or get the provided fix for this bug ASAP - since this is getting really annoying :/ .

Thanks a lot, and especially you @EternalWind for providing a solution for that bug :) .

[EternalWind]

@athtsio Since they are unlikely to be merging my fix any time soon, one workaround we have been using is docker's multi-stage build.

We base the final image on a much simpler base image, e.g. a minimal Alpine image, instead of the desired base image. Then, we use the desired base image as a builder image and copy out all of the desired files from it since the instructions from a builder image are not going into the final image.

Of course, this is based on the assumption that most of the instructions you want from the desired base image are installing packages or copying files.

Hopefully the fix could be merged soon.

[athtsio]

@EternalWind thanks for the response/suggestion.
We are using some kind of builder pattern.

The base image is really minimal. It is built and published in a registry. This means that we do not have to build this base image each time we need to dockerize the app. We take it/use it as is in the one single FROM in the Dockerfile.

On top of that base image, we built the image that will be used from the jib task which is like the base image plus 3 new packages installed, copy and run a script that imports the aws rds certificates. Nothing more !

My point is that the instructions are literally 10 lines.
From my point of view, I see nothing extreme in the Dockerfile to switch to multi stage build pattern :/

I do not know if by adding --no-cache on docker build will bring any difference...or make things worse 😭

[artemik]

Any progress?

On 3.4.3 is getting stuck on "building image to Docker daemon step". Don't know if it's the same.

Executing tasks:
[=======================       ] 75,0% complete
> building image to Docker daemon

<=========----> 75% EXECUTING [3m 16s]
> :jibDockerBuild

taonity/vk-forwarder-bot#13

[athtsio]

@artemik
I am not sure if this is the same reason behind this behavior, but I've also faced the same issue after upgrading to 3.4.3
Issue is gone by going back to 3.4.2
Hope this will be resolved in a future release

[time4tea]

Reverting to 3.4.2 fixed indefinite hang. idk what the problem was, it wasn't happening, then it was 100% repeatable - couldn't build anything.

was hung here, but i couldn't find any obvious running thread doing anything...

"Execution worker Thread 5" #259 prio=5 os_prio=0 cpu=52.26ms elapsed=153.66s tid=0x00007f8c1c0053b0 nid=0x1c36e waiting on condition  [0x00007f8d585ec000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x00000000eddfe370> (a com.google.common.util.concurrent.TrustedListenableFutureTask)
	at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:561)
	at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.run(StepsRunner.java:222)
	at com.google.cloud.tools.jib.api.Containerizer.run(Containerizer.java:406)
	at com.google.cloud.tools.jib.api.JibContainerBuilder.containerize(JibContainerBuilder.java:610)
	at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild(JibBuildRunner.java:229)
	at com.google.cloud.tools.jib.gradle.BuildDockerTask.buildDocker(BuildDockerTask.java:126)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:77)