Sometimes a pipeline job hangs around checkout.
Lately this seems to be after git rev-list.
> git rev-list --no-walk db25209b84b12cca5214227a7999fe280c82017e # timeout=10
It will hang here until job is killed. A restart is fine - and it only happens occasionally.
This has been more commonly reported since 2018 February/January, a recent update.
A support bundle of this when it was happening: support_2018-02-07_22.54.25.zip
IT was happening on slave-05 at the time (can see in its thread dump a few waiting things). Timeout does not work, it will wait indefinitely.
My pipeline job is getting sporadically hung on checkout step.
Our git repo is relatively small and clone should not take more than 1 min.
The problem occurring really often once a couple builds. Once it's occurring, the only way that I know to get rid of this state so far is, is disconnect the slave and re-connect it back.
Build log:
Started by user <USER>
Checking out git <REPO> to read Jenkinsfile.bootstrap
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url <REPO> # timeout=10
Fetching upstream changes from <REPO>
> git --version # timeout=10
using GIT_SSH to set credentials jenkinsslave SSH key
> git fetch --tags --progress <REPO> +refs/heads/:refs/remotes/origin/
> git rev-parse refs/remotes/origin/devops_jenkins^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/devops_jenkins^{commit} # timeout=10
Checking out Revision 2fd998fbe8b23f814b57d8c6eb7b46004268eedc (refs/remotes/origin/devops_jenkins)
> git config core.sparsecheckout # timeout=10
> git checkout -f 2fd998fbe8b23f814b57d8c6eb7b46004268eedc
> git rev-list 2fd998fbe8b23f814b57d8c6eb7b46004268eedc # timeout=10
[Pipeline] timestamps
[Pipeline] {
[Pipeline] node
00:00:01.249 Running on <SLAVE> in <WORKSPACE>
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Load Jenkinsfile)
[Pipeline] echo
00:00:01.267 Branch: origin/devops_jenkins Repo: <REPO>
[Pipeline] checkout
00:03:27.565 > git rev-parse -is-inside-work-tree # timeout=10 <- This and further lines are being printed only after slave disconnect.
00:03:27.566 Fetching changes from the remote Git repository
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] End of Pipeline
hudson.remoting.RemotingSystemException: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:272)
at com.sun.proxy.$Proxy98.setRemoteUrl(Unknown Source)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:794)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:221)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:191)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:256)
... 16 more
Caused by: hudson.remoting.Channel$OrderlyShutdown
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1121)
at hudson.remoting.Channel$1.handle(Channel.java:526)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused by: Command close created at
at hudson.remoting.Command.<init>(Command.java:59)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1115)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1113)
at hudson.remoting.Channel.close(Channel.java:1273)
at hudson.remoting.Channel.close(Channel.java:1255)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1120)
... 2 more
Finished: FAILURE
- duplicates
-
JENKINS-48357 Binary Compatibility between JIRA Plugin and Apache HttpComponents Client 4.x API
-
- Closed
-
- is related to
-
-
- Closed
-
[JENKINS-43106] pipeline job hangs forever at checkout GitSCM - after git rev-list (sporadic)
vrobert78 I see the same thing - git rev-list too... I haev a support core bundle for it too - taking a look
My support bundle support_2018-02-07_22.54.25.zip
Slave-05 is the one that was hanging at the time if that helps.
Hey markwaite any ideas on this? This seems to be a recently popped up thing, and it is biting more people (perhaps I should close and re-open this ticket, as the original one doesn't seem to be quite the same).
could this be the same as https://issues.jenkins-ci.org/browse/JENKINS-46503 - in which case the solution is to use JIRA plugin 2.5.0 (not 2.5.1) - still not totally resolved.
Vincent Robert
added a comment - Hi michaelneale,
i just disabled JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean.
We will see during the day if the problem persists or not.
We will keep you in touch.
Best regards,
Vincent Robert
added a comment - Hi michaelneale ,
i just disabled JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean.
We will see during the day if the problem persists or not.
We will keep you in touch.
Best regards, vrobert78 I don't think it is enough to disable the Jira plugin. I believe that you need to downgrade the Jira plugin from 2.5.1 to 2.5.0 or earlier. There was a discussion of "blacklisting" Jira plugin 2.5.1 and that has been implemented about a week ago in the Jenkins update center.
Refer to JENKINS-48357 for the details of the types of damage done to critical libraries by the 2.5.1 release.
Vincent Robert
added a comment - markewaite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday.
Vincent Robert
added a comment - markewaite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday. Vincent Robert
added a comment - We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them.
Indeed, versions has changed:
- Blue Ocean 1.4.1
- JIRA Integration for Blue Ocean 1.4.1
- JIRA Plugin 2.5 (was 2.5.1)
We hope this will be as fine as yesterday.
Vincent Robert
added a comment - We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them.
Indeed, versions has changed:
Blue Ocean 1.4.1
JIRA Integration for Blue Ocean 1.4.1
JIRA Plugin 2.5 (was 2.5.1)
We hope this will be as fine as yesterday. vrobert78 thanks very much for your investigation and for sharing your results. I'd like to close this bug as a duplicate of JENKINS-48357. JENKINS-48357 will remain open until the fix is released in the Jira plugin. Is that OK?
Hi,
we have the same problem with a very updated version:
Jenkins 2.89.3 (jenkins/jenkins:2.89.3-alpine)
GKE (k8s 1.8.7 - COS image)
The build come in a infinite loop just after the "git rev-list".
We have to kill the build to stop it.
The problem appears sporadically, in general a replay successed.
The problem has appeared in the last 3 weeks.
Push event to branch dev15:11:08 Connecting to https://api.github.com using xxxx/******Obtained Jenkinsfile from xxxx
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] lock
Trying to acquire lock on [xxxx]
Lock acquired on [xxxx]
[Pipeline] {
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
Waiting for next available executor on jenkins-slave-w9k9c-5dghv
Running on
jenkins-slave-w9k9c-zh83m
in /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ
[Pipeline] {
[Pipeline] checkout
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository
https://github.com/xxx/xxx.git
> git init /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ # timeout=10
Fetching upstream changes from
https://github.com/ xxx/xxx .git
> git --version # timeout=10
using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress
https://github.com/ xxx/xxx .git
+refs/heads/dev:refs/remotes/origin/dev
> git config remote.origin.url
https://github.com/ xxx/xxx .git
> git config --add remote.origin.fetch +refs/heads/dev:refs/remotes/origin/dev # timeout=10
> git config remote.origin.url
https://github.com/ xxx/xxx .git
Fetching without tags
Fetching upstream changes from
https://github.com/ xxx/xxx .git
using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress
https://github.com/ xxx/xxx .git
+refs/heads/dev:refs/remotes/origin/dev
Checking out Revision xxxx (dev)
> git config core.sparsecheckout # timeout=10
> git checkout -f xxxx
Commit message: "Merge tag 'xxxx' into dev"
> git rev-list --no-walk xxxx # timeout=10