Multibranch Pipeline Branch Scanning Hang(Gitlab Server Error case)

[ish8758]

Jenkins and plugins versions report

Environment

OS: Windows Server 2019 - 10.0
---
Office-365-Connector:4.15.2
ace-editor:1.1
active-directory:2.25.1
analysis-model-api:10.9.4
ansicolor:1.0.1
ant:1.13
antisamy-markup-formatter:2.7
apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61
artifact-manager-s3:670.v0558a_cb_c82c2
audit-trail:3.10
authentication-tokens:1.4
authorize-project:1.4.0
aws-credentials:191.vcb_f183ce58b_9
aws-global-configuration:106.v106dc1d8d86e
aws-java-sdk-ec2:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-minimal:1.12.287-357.vf82d85a_6eefd
badge:1.9.1
bitbucket:223.vd12f2bca5430
bootstrap4-api:4.6.0-3
bootstrap5-api:5.1.3-6
bouncycastle-api:2.25
branch-api:2.1044.v2c007e51b_87f
build-blocker-plugin:1.7.8
build-discarder:60.v1747b0eb632a
build-timeout:1.20
build-timestamp:1.0.3
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
cctray-xml:1.1
checks-api:1.7.4
cloudbees-bitbucket-branch-source:757.vddedc5f2589a_
cloudbees-folder:6.714.v79e858ef76a_2
command-launcher:1.6
conditional-buildstep:1.4.2
config-file-provider:3.10.0
configurationslicing:430.v966357576543
credentials:1087.1089.v2f1b_9a_b_040e4
credentials-binding:1.27.1
data-tables-api:1.11.4-4
display-url-api:2.3.6
docker-commons:1.19
docker-workflow:1.28
durable-task:495.v29cd95ec10f2
echarts-api:5.3.3-1
email-ext:2.87
embeddable-build-status:2.0.4
envinject:2.839.v52c702c10635
envinject-api:1.180.v98d833b_27470
extended-read-permission:3.2
external-monitor-job:191.v363d0d1efdf8
font-awesome-api:6.0.0-1
forensics-api:1.12.0
git:4.11.3
git-client:3.11.0
git-server:1.10
github:1.34.3
github-api:1.301-378.v9807bd746da5
github-branch-source:1598.v91207e9f9b_4a_
gitlab-api:5.0.1-72.vb_8c272862e86
gitlab-branch-source:625.v85cf3a_400cfe
gitlab-plugin:1.5.35
global-slack-notifier:1.5
gradle:1.38
groovy:2.4
groovy-postbuild:2.5
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
jackson2-api:2.13.3-285.vc03c0256d517
javadoc:217.v905b_86277a_2a_
javax-activation-api:1.2.0-3
javax-mail-api:1.6.2-5
jaxb:2.3.6-1
jdk-tool:1.5
jersey2-api:2.36-2
jjwt-api:0.11.2-9.c8b45b8bb173
jnr-posix-api:3.1.7-3
job-dsl:1.79
jquery:1.12.4-1
jquery-detached:1.2.1
jquery-ui:1.0.2
jquery3-api:3.6.0-4
jsch:0.1.55.2
junit:1119.1121.vc43d0fc45561
ldap:2.8
ldapemail:0.8
lockable-resources:2.14
mailer:408.vd726a_1130320
mapdb-api:1.0.9.0
matrix-auth:2.6.11
matrix-project:771.v574584b_39e60
maven-plugin:3.18
mercurial:2.16.2
momentjs:1.1.1
monitoring:1.90.0
msbuild:1.30
mstest:1.0.0
multibranch-scan-webhook-trigger:1.0.9
next-build-number:1.8
okhttp-api:4.9.3-105.vb96869f8ac3a
p4:1.12.2
pam-auth:1.7
parameterized-scheduler:1.0
permissive-script-security:0.7
persistent-parameter:1.3
pipeline-build-step:2.16
pipeline-github-lib:36.v4c01db_ca_ed16
pipeline-graph-analysis:188.v3a01e7973f2c
pipeline-groovy-lib:593.va_a_fc25d520e9
pipeline-input-step:449.v77f0e8b_845c4
pipeline-milestone-step:100.v60a_03cd446e1
pipeline-model-api:2.2064.v5eef7d0982b_e
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2064.v5eef7d0982b_e
pipeline-model-extensions:2.2064.v5eef7d0982b_e
pipeline-rest-api:2.23
pipeline-stage-step:291.vf0a8a7aeeb50
pipeline-stage-tags-metadata:2.2064.v5eef7d0982b_e
pipeline-stage-view:2.23
pipeline-utility-steps:2.12.0
plain-credentials:1.8
plugin-util-api:2.17.0
popper-api:1.16.1-2
popper2-api:2.11.4-1
powershell:1.7
prism-api:1.26.0-2
publish-over:0.22
publish-over-cifs:0.16
pvs-studio:7.18
resource-disposer:0.18
role-strategy:484.v8a_a_e4b_d785fd
run-condition:1.5
rundeck:3.6.11
schedule-build:244.vce2a_2e926127
scm-api:608.vfa_f971c5a_a_e9
script-security:1175.v4b_d517d6db_f0
show-build-parameters:1.0
simple-theme-plugin:103.va_161d09c38c7
skip-certificate-check:1.0
slack:608.v19e3b_44b_b_9ff
snakeyaml-api:1.29.1
ssh-agent:1.24.1
ssh-credentials:277.v95c2fec1c047
ssh-slaves:1.806.v2253cedd3295
sshd:3.1.0
strict-crumb-issuer:2.1.0
structs:324.va_f5d6774f3a_d
subversion:2.15.5
throttle-concurrents:2.7
timestamper:1.17
token-macro:293.v283932a_0a_b_49
trilead-api:1.0.13
unity3d-plugin:1.3
variant:1.4
vstestrunner:1.0.8
warnings-ng:9.11.1
windows-slaves:1.8.1
workflow-aggregator:2.7
workflow-api:1200.v8005c684b_a_c6
workflow-basic-steps:941.vdfe1b_a_132c64
workflow-cps:2725.v7b_c717eb_12ce
workflow-cps-global-lib:588.v576c103a_ff86
workflow-durable-task-step:1128.v8c259d125340
workflow-job:1174.vdcb_d054cf74a_
workflow-multibranch:711.vdfef37cda_816
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:625.vd896b_f445a_f8
workflow-support:820.vd1a_6cc65ef33
zulip:1.3.0

What Operating System are you using (both controller, and any agents involved in the problem)?

OS: Windows Server 2019 (Controller / Agent Both)

Reproduction steps

Step1: Create Multibranch Pipeline Job
Step2: While scanning branch indexing, Gitlab SCM Server occur
Step3: Scanning Task Hang

Expected Results

After minitues, scanning task exception or timeout exception

Actual Results

while fetching branch indexing in a multibrach pipeline job, it getting stuck at scanning branch for days
(As shown in the attached image)

Problmatic thread got stuck in SocketInputStream.socketread0 method(this thread didn't kill by groovy script interrupt or monitoring plugin thread kill/stop)

stack trace: 
Executor #-1 for Built-In Node : executing BranchIndexing[Faceplay/UnityClient]
java.base@11.0.13/java.net.SocketInputStream.socketRead0(Native Method)
java.base@11.0.13/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
java.base@11.0.13/java.net.SocketInputStream.read(SocketInputStream.java:168)
java.base@11.0.13/java.net.SocketInputStream.read(SocketInputStream.java:140)
java.base@11.0.13/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
java.base@11.0.13/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
java.base@11.0.13/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
java.base@11.0.13/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1374)
java.base@11.0.13/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:985)
java.base@11.0.13/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
java.base@11.0.13/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
java.base@11.0.13/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
java.base@11.0.13/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:754)
java.base@11.0.13/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
java.base@11.0.13/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615)
java.base@11.0.13/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
java.base@11.0.13/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
java.base@11.0.13/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:378)
org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:267)
org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297)
org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662)
org.glassfish.jersey.client.JerseyInvocation$$Lambda$1124/0x0000000800ef4840.call(Unknown Source)
org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697)
org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691)
org.glassfish.jersey.client.JerseyInvocation$$Lambda$1127/0x0000000800ef5440.call(Unknown Source)
org.glassfish.jersey.internal.Errors.process(Errors.java:292)
org.glassfish.jersey.internal.Errors.process(Errors.java:274)
org.glassfish.jersey.internal.Errors.process(Errors.java:205)
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691)
org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661)
org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:413)
org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:313)
org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:416)
org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:404)
org.gitlab4j.api.AbstractApi.get(AbstractApi.java:214)
org.gitlab4j.api.RepositoryFileApi.getFile(RepositoryFileApi.java:169)
org.gitlab4j.api.RepositoryFileApi.getFile(RepositoryFileApi.java:122)
io.jenkins.plugins.gitlabbranchsource.GitLabSCMFile.type(GitLabSCMFile.java:93)
jenkins.scm.api.SCMFile.getType(SCMFile.java:265)
io.jenkins.plugins.gitlabbranchsource.GitLabSCMSource$4.stat(GitLabSCMSource.java:686)
org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:75)
jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:286)
io.jenkins.plugins.gitlabbranchsource.GitLabSCMSource.retrieve(GitLabSCMSource.java:357)
jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:641)
com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032)
hudson.model.ResourceController.execute(ResourceController.java:101)
hudson.model.Executor.run(Executor.java:442)

Anything else?

Timeouts setting(org.gitlab4j.api.GitLabApiClien setRequestTimeout API call in gitlabbranchsource plugin) are needed to solve the above problem.

The same issue can be identified in Bitbuckt/Github source plugin(JENKINS-36724](https://issues.jenkins.io/browse/JENKINS-36724), Bitbucket Hang Timeout Setting Code Commit)

Additonally, it is necessary to provide a jvm system property to set the gitlab timeout for timeout value.(example, java -Dio.jenkins.plugins.gitlabbranchsource.config.readTimeout -Dio.jenkins.plugins.gitlabbranchsource.config.connectTimeout)

If you can, we propose a common SCMSourceRequest interface. This issue occurs in many jenkins sCM source plugin such as gitblab/bitbucket/github, so we need a common interface at Jenkins level that sets the request timeout based on system properties

[ravage84]

I observe an issue that looks like this one. A scan triggered every minute is now running for over a week.

From the log:

Build was timed triggered. {Translated from German}

[Mi. Jan. 18 23:49:06 UTC 2023] Starting branch indexing...

Checking branches.. 

{branches got detected}

5 branches were processed

Checking merge requests..

0 merge requests were processed

Checking tags..

{some tags got detected}

Checking tag {random tag}

[ravage84]

Just updated another, similar Jenkins job on the same Jenkins server, connected to the same GitLab server, checking from the same owner to use the GitLab Branch Source plugin and it scanned through several times without hanging.

Even manually starting a scan while another, time-triggered one is still running does not lead to a hang.

[ravage84]

After a restart of Jenkins, both Jenkins jobs ran through the scan sucessfully several times, again.

[sIuv]

Hello,

I have created a branch where I've enabled the timeouts, do you have a good way to reproduce the issue ?
Branch: https://github.com/sIuv/gitlab-branch-source-plugin/tree/feat/gitlab-api-timeout

I've set the connect- & request-timeout to 60s.

[grazius]

We also be impacted by the same bug / problem
Jenkins 2.346.3
GitLab Branch Source Plugin Version 647.vdee7766b_cfa_e

[redgkl77]

@sIuv We are also impacted by this issue frequently and are forced to restart Jenkins on every occurrence of the issue. In order to reproduce the issue please perform the following:

1. Kick off a branch indexing scan and let it begin identifying branches. (Note that this issue does NOT occur when Gitlab is unreachable BEFORE the scan is initiated).
2. Bring down Gitlab network connectivity during the scan.
3. Branch index scan hang will occur.

I would really appreciate a fix for this issue, thanks.

(Jenkins Version 2.426.3
GitLab Branch Source Plugin Version 704.vc7f1202d7e14)

[ravage84]

In our case it happened because repo polling was set to 1 minute.
That propably led to unfinished/overlapping requests which then led to blockage and timeouts and then hung the scan.

We configured the Jenkins integration on (our self-hosted) GitLab's side, which informs Jenkins about pushes etc, we increased repo polling to 1 hour.
I haven't noticed a single hang since.

But I can imagine not everybody is able to do that configuration.