Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43106

pipeline job hangs forever at checkout GitSCM - after git rev-list (sporadic)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Duplicate
    • core, git-plugin
    • None

    Description

      Sometimes a pipeline job hangs around checkout. 

      Lately this seems to be after git rev-list. 

       

       > git rev-list --no-walk db25209b84b12cca5214227a7999fe280c82017e # timeout=10

      It will hang here until job is killed. A restart is fine - and it only happens occasionally. 

      This has been more commonly reported since 2018 February/January, a recent update. 

       

      A support bundle of this when it was happening: support_2018-02-07_22.54.25.zip

      IT was happening on slave-05 at the time (can see in its thread dump a few waiting things). Timeout does not work, it will wait indefinitely. 

       

       


      My pipeline job is getting sporadically hung on checkout step.

      Our git repo is relatively small and clone should not take more than 1 min.

      The problem occurring really often once a couple builds. Once it's occurring, the only way that I know to get rid of this state so far is, is disconnect the slave and re-connect it back.

      Build log:

      Started by user <USER>
      Checking out git <REPO> to read Jenkinsfile.bootstrap
      > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
      > git config remote.origin.url <REPO> # timeout=10
      Fetching upstream changes from <REPO>
      > git --version # timeout=10
      using GIT_SSH to set credentials jenkinsslave SSH key
      > git fetch --tags --progress <REPO> +refs/heads/:refs/remotes/origin/
      > git rev-parse refs/remotes/origin/devops_jenkins^{commit} # timeout=10
      > git rev-parse refs/remotes/origin/origin/devops_jenkins^{commit} # timeout=10
      Checking out Revision 2fd998fbe8b23f814b57d8c6eb7b46004268eedc (refs/remotes/origin/devops_jenkins)
      > git config core.sparsecheckout # timeout=10
      > git checkout -f 2fd998fbe8b23f814b57d8c6eb7b46004268eedc
      > git rev-list 2fd998fbe8b23f814b57d8c6eb7b46004268eedc # timeout=10
      [Pipeline] timestamps
      [Pipeline] {
      [Pipeline] node
      00:00:01.249 Running on <SLAVE> in <WORKSPACE>
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Load Jenkinsfile)
      [Pipeline] echo
      00:00:01.267 Branch: origin/devops_jenkins Repo: <REPO>
      [Pipeline] checkout
      00:03:27.565 > git rev-parse -is-inside-work-tree # timeout=10 <- This and further lines are being printed only after slave disconnect.
      00:03:27.566 Fetching changes from the remote Git repository
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] End of Pipeline
      hudson.remoting.RemotingSystemException: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:272)
      at com.sun.proxy.$Proxy98.setRemoteUrl(Unknown Source)
      at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:794)
      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
      at hudson.security.ACL.impersonate(ACL.java:221)
      at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
      at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:191)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:256)
      ... 16 more
      Caused by: hudson.remoting.Channel$OrderlyShutdown
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1121)
      at hudson.remoting.Channel$1.handle(Channel.java:526)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
      Caused by: Command close created at
      at hudson.remoting.Command.<init>(Command.java:59)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1115)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1113)
      at hudson.remoting.Channel.close(Channel.java:1273)
      at hudson.remoting.Channel.close(Channel.java:1255)
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1120)
      ... 2 more
      Finished: FAILURE

      Attachments

        1. Capture.PNG
          Capture.PNG
          96 kB
        2. config.xml
          2 kB
        3. config.xml
          5 kB
        4. Jenkinsfile
          2 kB
        5. support_2017-10-06_04.20.32.zip
          4.47 MB
        6. support_2017-10-11_21.17.12.zip
          2.65 MB
        7. support_2018-02-07_22.54.25.zip
          1.12 MB
        8. threadDump.txt
          68 kB

        Issue Links

          Activity

            People

              vrobert78 Vincent Robert
              levt Lev Tartakovsky
              Votes:
              8 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: