Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43106

pipeline job hangs forever at checkout GitSCM - after git rev-list (sporadic)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Duplicate
    • core, git-plugin
    • None

    Description

      Sometimes a pipeline job hangs around checkout. 

      Lately this seems to be after git rev-list. 

       

       > git rev-list --no-walk db25209b84b12cca5214227a7999fe280c82017e # timeout=10

      It will hang here until job is killed. A restart is fine - and it only happens occasionally. 

      This has been more commonly reported since 2018 February/January, a recent update. 

       

      A support bundle of this when it was happening: support_2018-02-07_22.54.25.zip

      IT was happening on slave-05 at the time (can see in its thread dump a few waiting things). Timeout does not work, it will wait indefinitely. 

       

       


      My pipeline job is getting sporadically hung on checkout step.

      Our git repo is relatively small and clone should not take more than 1 min.

      The problem occurring really often once a couple builds. Once it's occurring, the only way that I know to get rid of this state so far is, is disconnect the slave and re-connect it back.

      Build log:

      Started by user <USER>
      Checking out git <REPO> to read Jenkinsfile.bootstrap
      > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
      > git config remote.origin.url <REPO> # timeout=10
      Fetching upstream changes from <REPO>
      > git --version # timeout=10
      using GIT_SSH to set credentials jenkinsslave SSH key
      > git fetch --tags --progress <REPO> +refs/heads/:refs/remotes/origin/
      > git rev-parse refs/remotes/origin/devops_jenkins^{commit} # timeout=10
      > git rev-parse refs/remotes/origin/origin/devops_jenkins^{commit} # timeout=10
      Checking out Revision 2fd998fbe8b23f814b57d8c6eb7b46004268eedc (refs/remotes/origin/devops_jenkins)
      > git config core.sparsecheckout # timeout=10
      > git checkout -f 2fd998fbe8b23f814b57d8c6eb7b46004268eedc
      > git rev-list 2fd998fbe8b23f814b57d8c6eb7b46004268eedc # timeout=10
      [Pipeline] timestamps
      [Pipeline] {
      [Pipeline] node
      00:00:01.249 Running on <SLAVE> in <WORKSPACE>
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Load Jenkinsfile)
      [Pipeline] echo
      00:00:01.267 Branch: origin/devops_jenkins Repo: <REPO>
      [Pipeline] checkout
      00:03:27.565 > git rev-parse -is-inside-work-tree # timeout=10 <- This and further lines are being printed only after slave disconnect.
      00:03:27.566 Fetching changes from the remote Git repository
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] End of Pipeline
      hudson.remoting.RemotingSystemException: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:272)
      at com.sun.proxy.$Proxy98.setRemoteUrl(Unknown Source)
      at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:794)
      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
      at hudson.security.ACL.impersonate(ACL.java:221)
      at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
      at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:191)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:256)
      ... 16 more
      Caused by: hudson.remoting.Channel$OrderlyShutdown
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1121)
      at hudson.remoting.Channel$1.handle(Channel.java:526)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
      Caused by: Command close created at
      at hudson.remoting.Command.<init>(Command.java:59)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1115)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1113)
      at hudson.remoting.Channel.close(Channel.java:1273)
      at hudson.remoting.Channel.close(Channel.java:1255)
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1120)
      ... 2 more
      Finished: FAILURE

      Attachments

        1. Capture.PNG
          96 kB
          Lev Tartakovsky
        2. config.xml
          2 kB
          Brandon Parker
        3. config.xml
          5 kB
          Lev Tartakovsky
        4. Jenkinsfile
          2 kB
          Lev Tartakovsky
        5. threadDump.txt
          68 kB
          Lev Tartakovsky

        Issue Links

          Activity

            markewaite Mark Waite added a comment - - edited

            vrobert78 I don't think it is enough to disable the Jira plugin. I believe that you need to downgrade the Jira plugin from 2.5.1 to 2.5.0 or earlier. There was a discussion of "blacklisting" Jira plugin 2.5.1 and that has been implemented about a week ago in the Jenkins update center.

            Refer to JENKINS-48357 for the details of the types of damage done to critical libraries by the 2.5.1 release.

            markewaite Mark Waite added a comment - - edited vrobert78 I don't think it is enough to disable the Jira plugin. I believe that you need to downgrade the Jira plugin from 2.5.1 to 2.5.0 or earlier. There was a discussion of "blacklisting" Jira plugin 2.5.1 and that has been implemented about a week ago in the Jenkins update center . Refer to JENKINS-48357 for the details of the types of damage done to critical libraries by the 2.5.1 release.

            markewaite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday.

            vrobert78 Vincent Robert added a comment - markewaite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday.

            We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them.

            Indeed, versions has changed:

            • Blue Ocean 1.4.1
            • JIRA Integration for Blue Ocean 1.4.1
            • JIRA Plugin 2.5 (was 2.5.1)

            We hope this will be as fine as yesterday.

            vrobert78 Vincent Robert added a comment - We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them. Indeed, versions has changed: Blue Ocean 1.4.1 JIRA Integration for Blue Ocean 1.4.1 JIRA Plugin 2.5 (was 2.5.1) We hope this will be as fine as yesterday.
            markewaite Mark Waite added a comment - - edited

            vrobert78 thanks very much for your investigation and for sharing your results. I'd like to close this bug as a duplicate of JENKINS-48357. JENKINS-48357 will remain open until the fix is released in the Jira plugin. Is that OK?

            markewaite Mark Waite added a comment - - edited vrobert78 thanks very much for your investigation and for sharing your results. I'd like to close this bug as a duplicate of JENKINS-48357 . JENKINS-48357 will remain open until the fix is released in the Jira plugin. Is that OK?
            vgaidarji Veaceslav Gaidarji added a comment - svanoort ping

            People

              vrobert78 Vincent Robert
              levt Lev Tartakovsky
              Votes:
              8 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: