Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43106

pipeline job hangs forever at checkout GitSCM - after git rev-list (sporadic)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • core, git-plugin
    • None

      Sometimes a pipeline job hangs around checkout. 

      Lately this seems to be after git rev-list. 

       

       > git rev-list --no-walk db25209b84b12cca5214227a7999fe280c82017e # timeout=10

      It will hang here until job is killed. A restart is fine - and it only happens occasionally. 

      This has been more commonly reported since 2018 February/January, a recent update. 

       

      A support bundle of this when it was happening: support_2018-02-07_22.54.25.zip

      IT was happening on slave-05 at the time (can see in its thread dump a few waiting things). Timeout does not work, it will wait indefinitely. 

       

       

      My pipeline job is getting sporadically hung on checkout step.

      Our git repo is relatively small and clone should not take more than 1 min.

      The problem occurring really often once a couple builds. Once it's occurring, the only way that I know to get rid of this state so far is, is disconnect the slave and re-connect it back.

      Build log:

      Started by user <USER>
      Checking out git <REPO> to read Jenkinsfile.bootstrap
      > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
      > git config remote.origin.url <REPO> # timeout=10
      Fetching upstream changes from <REPO>
      > git --version # timeout=10
      using GIT_SSH to set credentials jenkinsslave SSH key
      > git fetch --tags --progress <REPO> +refs/heads/:refs/remotes/origin/
      > git rev-parse refs/remotes/origin/devops_jenkins^{commit} # timeout=10
      > git rev-parse refs/remotes/origin/origin/devops_jenkins^{commit} # timeout=10
      Checking out Revision 2fd998fbe8b23f814b57d8c6eb7b46004268eedc (refs/remotes/origin/devops_jenkins)
      > git config core.sparsecheckout # timeout=10
      > git checkout -f 2fd998fbe8b23f814b57d8c6eb7b46004268eedc
      > git rev-list 2fd998fbe8b23f814b57d8c6eb7b46004268eedc # timeout=10
      [Pipeline] timestamps
      [Pipeline] {
      [Pipeline] node
      00:00:01.249 Running on <SLAVE> in <WORKSPACE>
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Load Jenkinsfile)
      [Pipeline] echo
      00:00:01.267 Branch: origin/devops_jenkins Repo: <REPO>
      [Pipeline] checkout
      00:03:27.565 > git rev-parse -is-inside-work-tree # timeout=10 <- This and further lines are being printed only after slave disconnect.
      00:03:27.566 Fetching changes from the remote Git repository
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] End of Pipeline
      hudson.remoting.RemotingSystemException: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:272)
      at com.sun.proxy.$Proxy98.setRemoteUrl(Unknown Source)
      at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:794)
      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
      at hudson.security.ACL.impersonate(ACL.java:221)
      at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
      at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:191)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:256)
      ... 16 more
      Caused by: hudson.remoting.Channel$OrderlyShutdown
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1121)
      at hudson.remoting.Channel$1.handle(Channel.java:526)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
      Caused by: Command close created at
      at hudson.remoting.Command.<init>(Command.java:59)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1115)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1113)
      at hudson.remoting.Channel.close(Channel.java:1273)
      at hudson.remoting.Channel.close(Channel.java:1255)
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1120)
      ... 2 more
      Finished: FAILURE

        1. support_2018-02-07_22.54.25.zip
          1.12 MB
        2. support_2017-10-11_21.17.12.zip
          2.65 MB
        3. support_2017-10-06_04.20.32.zip
          4.47 MB
        4. config.xml
          2 kB
        5. Capture.PNG
          Capture.PNG
          96 kB
        6. config.xml
          5 kB
        7. Jenkinsfile
          2 kB
        8. threadDump.txt
          68 kB

          [JENKINS-43106] pipeline job hangs forever at checkout GitSCM - after git rev-list (sporadic)

          Lev Tartakovsky added a comment -

          My environment is also affected by https://issues.jenkins-ci.org/browse/JENKINS-43038 

          I suspect the both problem have similar root cause that could be related to communication between master and slave.

          Lev Tartakovsky added a comment - My environment is also affected by https://issues.jenkins-ci.org/browse/JENKINS-43038   I suspect the both problem have similar root cause that could be related to communication between master and slave.

          James Dumay added a comment -

          levt are you still having issues, even upgrading to a new Jenkins core and updating all plugins?

          James Dumay added a comment - levt are you still having issues, even upgrading to a new Jenkins core and updating all plugins?

          Brandon Parker added a comment -

          This is a problem that I see on a daily basis. It just hangs at the git cloning step. 

          We are running Ubuntu 14.04.5, Jenkins 2.70, JRE 1.8.0_131-b11 and git 2.11.0 

          We are using the following plugins:

          PAM Authentication plugin (pam-auth): 1.3
          Branch API Plugin (branch-api): 2.0.11
          TextFinder plugin (text-finder): 1.10
          Gitlab Merge Request Builder (gitlab-merge-request-jenkins): 2.0.0
          Docker Commons Plugin (docker-commons): 1.8
          skip-certificate-check (skip-certificate-check): 1.0
          JIRA Pipeline Steps (jira-steps): 1.2.3
          Docker Pipeline (docker-workflow): 1.12
          Subversion Plug-in (subversion): 2.9
          Next Build Number Plugin (next-build-number): 1.4
          Lockable Resources plugin (lockable-resources): 2.0
          Pipeline: Multibranch (workflow-multibranch): 2.16
          GitLab Plugin (gitlab-plugin): 1.4.7
          Git Parameter Plug-In (git-parameter): 0.8.0
          Changes since last successfull build Plugin (changes-since-last-success): 0.5
          Amazon EC2 plugin (ec2): 1.36
          Pipeline: Supporting APIs (workflow-support): 2.14
          LDAP Plugin (ldap): 1.16
          Pipeline: API (workflow-api): 2.20
          JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
          ruby-runtime (ruby-runtime): 0.12
          SSH Slaves plugin (ssh-slaves): 1.21
          Pipeline: Basic Steps (workflow-basic-steps): 2.6
          Script Security Plugin (script-security): 1.33
          Structs Plugin (structs): 1.10
          Windows Slaves Plugin (windows-slaves): 1.3.1
          Translation Assistance plugin (translation): 1.15
          Pipeline: Stage Step (pipeline-stage-step): 2.2
          Credentials Plugin (credentials): 2.1.14
          Authentication Tokens API Plugin (authentication-tokens): 1.3
          JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
          Timestamper (timestamper): 1.8.8
          Credentials Binding Plugin (credentials-binding): 1.13
          GIT server Plugin (git-server): 1.7
          Extended Choice Parameter Plug-In (extended-choice-parameter): 0.76
          Plain Credentials Plugin (plain-credentials): 1.4
          jQuery plugin (jquery): 1.11.2-0
          CloudBees Amazon Web Services Credentials Plugin (aws-credentials): 1.21
          Pipeline: Groovy (workflow-cps): 2.39
          Durable Task Plugin (durable-task): 1.14
          MapDB API Plugin (mapdb-api): 1.0.9.0
          JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
          Run Condition Plugin (run-condition): 1.0
          Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1
          Scriptler (scriptler): 2.9
          bouncycastle API Plugin (bouncycastle-api): 2.16.2
          Git Changelog (git-changelog): 1.51
          Pipeline: Step API (workflow-step-api): 2.12
          Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.1.9
          Pipeline: Model Definition (pipeline-model-definition): 1.1.9
          Node Iterator API Plugin (node-iterator-api): 1.5
          Job Configuration History Plugin (jobConfigHistory): 2.17
          Display URL API (display-url-api): 2.0
          Icon Shim Plugin (icon-shim): 2.0.3
          Token Macro Plugin (token-macro): 2.2
          Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.1.9
          disk-usage plugin (disk-usage): 0.28
          Pipeline: Job (workflow-job): 2.14.1
          Environment Injector Plugin (envinject): 2.1.3
          JUnit Plugin (junit): 1.21
          Gerrit Trigger (gerrit-trigger): 2.25.0
          Job DSL (job-dsl): 1.64
          Ant Plugin (ant): 1.7
          Matrix Authorization Strategy Plugin (matrix-auth): 1.7
          Cobertura Plugin (cobertura): 1.11
          Active Choices Plug-in (uno-choice): 1.5.3
          Maven Integration plugin (maven-plugin): 2.17
          SCM API Plugin (scm-api): 2.2.1
          build timeout plugin (build-timeout): 1.18
          HipChat Plugin (hipchat): 2.1.1
          Jabber (XMPP) notifier and control plugin (jabber): 1.36
          Mercurial plugin (mercurial): 2.0
          Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.5
          Hidden Parameter plugin (hidden-parameter): 0.0.4
          Build Pipeline Plugin (build-pipeline-plugin): 1.5.7.1
          Pipeline: Build Step (pipeline-build-step): 2.5.1
          Discard Old Build plugin (discard-old-build): 1.05
          CVS Plug-in (cvs): 2.13
          Pipeline: Stage View Plugin (pipeline-stage-view): 2.8
          Workspace Cleanup Plugin (ws-cleanup): 0.34
          Mailer Plugin (mailer): 1.20
          Resource Disposer Plugin (resource-disposer): 0.7
          Matrix Project Plugin (matrix-project): 1.11
          Pipeline: Input Step (pipeline-input-step): 2.8
          Groovy Postbuild (groovy-postbuild): 2.3.1
          Git client plugin (git-client): 2.5.0
          Jackson 2 API Plugin (jackson2-api): 2.7.3
          JaCoCo plugin (jacoco): 2.2.1
          Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1
          External Monitor Job Type Plugin (external-monitor-job): 1.7
          Multiple SCMs plugin (multiple-scms): 0.6
          Pipeline: Nodes and Processes (workflow-durable-task-step): 2.14
          instant-messaging plugin (instant-messaging): 1.35
          Copy Artifact Plugin (copyartifact): 1.38.1
          Python Plugin (python): 1.3
          Conditional BuildStep (conditional-buildstep): 1.3.6
          SSH Credentials Plugin (ssh-credentials): 1.13
          Pipeline: Model API (pipeline-model-api): 1.1.9
          Configuration Slicing plugin (configurationslicing): 1.47
          Hudson Post build task (postbuild-task): 1.8
          Pipeline (workflow-aggregator): 2.5
          Pipeline: SCM Step (workflow-scm-step): 2.6
          Parameterized Trigger plugin (parameterized-trigger): 2.35.1
          Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.8
          OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5
          Pipeline: REST API Plugin (pipeline-rest-api): 2.8
          Role-based Authorization Strategy (role-strategy): 2.5.1
          Naginator (naginator): 1.17.2
          Amazon Web Services SDK (aws-java-sdk): 1.11.119
          EnvInject API Plugin (envinject-api): 1.2
          Folders Plugin (cloudbees-folder): 6.1.2
          Mask Passwords Plugin (mask-passwords): 2.10.1
          Javadoc Plugin (javadoc): 1.4
          JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
          Git plugin (git): 3.5.1
          Email Extension Plugin (email-ext): 2.58
          Gitlab Hook Plugin (gitlab-hook): 1.4.2

           

          Here is the config.xml

           

          Here is the output from the console log after killing the job.

           
          Started by timer
          Obtained src/Jenkinsfile.CoconutPipeline.groovy from git git@git_server:namespace/job_repo.git
          [Pipeline] node
          Running on testops-w2 in /home/jenkins/workspace/Coconut/test-pipeline
          [Pipeline] {
          [Pipeline] timestamps
          [Pipeline] {
          [Pipeline] hipchatSend*00:00:09.027* [INFO] HipChat notification sent to the following rooms: QA FW, Firmware, QA Automation
          [Pipeline] stage
          [Pipeline] { (Checkout)
          [Pipeline] checkout*00:00:09.139* Cloning the remote Git repository*00:00:09.144* Cloning repository git@git_server:namespace/repo.git
          Aborted by Username
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] hipchatSend*01:26:16.377* [INFO] HipChat notification sent to the following rooms: QA FW, Firmware, QA Automation
          [Pipeline] }
          [Pipeline] // timestamps
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at hudson.remoting.Request.call(Request.java:147)
          at hudson.remoting.Channel.call(Channel.java:838)
          at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
          at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
          at com.sun.proxy.$Proxy101.execute(Unknown Source)
          at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1075)
          at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
          at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
          at hudson.security.ACL.impersonate(ACL.java:260)
          at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          Finished: ABORTED
           

          We try to keep Jenkins fairly up to date, and it's really annoying when this happens. It only happens on Pipeline jobs.

          Brandon Parker added a comment - This is a problem that I see on a daily basis. It just hangs at the git cloning step.  We are running Ubuntu 14.04.5, Jenkins 2.70, JRE 1.8.0_131-b11 and git 2.11.0  We are using the following plugins: PAM Authentication plugin (pam-auth): 1.3 Branch API Plugin (branch-api): 2.0.11 TextFinder plugin (text-finder): 1.10 Gitlab Merge Request Builder (gitlab-merge-request-jenkins): 2.0.0 Docker Commons Plugin (docker-commons): 1.8 skip-certificate-check (skip-certificate-check): 1.0 JIRA Pipeline Steps (jira-steps): 1.2.3 Docker Pipeline (docker-workflow): 1.12 Subversion Plug-in (subversion): 2.9 Next Build Number Plugin (next-build-number): 1.4 Lockable Resources plugin (lockable-resources): 2.0 Pipeline: Multibranch (workflow-multibranch): 2.16 GitLab Plugin (gitlab-plugin): 1.4.7 Git Parameter Plug-In (git-parameter): 0.8.0 Changes since last successfull build Plugin (changes-since-last-success): 0.5 Amazon EC2 plugin (ec2): 1.36 Pipeline: Supporting APIs (workflow-support): 2.14 LDAP Plugin (ldap): 1.16 Pipeline: API (workflow-api): 2.20 JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1 ruby-runtime (ruby-runtime): 0.12 SSH Slaves plugin (ssh-slaves): 1.21 Pipeline: Basic Steps (workflow-basic-steps): 2.6 Script Security Plugin (script-security): 1.33 Structs Plugin (structs): 1.10 Windows Slaves Plugin (windows-slaves): 1.3.1 Translation Assistance plugin (translation): 1.15 Pipeline: Stage Step (pipeline-stage-step): 2.2 Credentials Plugin (credentials): 2.1.14 Authentication Tokens API Plugin (authentication-tokens): 1.3 JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1 Timestamper (timestamper): 1.8.8 Credentials Binding Plugin (credentials-binding): 1.13 GIT server Plugin (git-server): 1.7 Extended Choice Parameter Plug-In (extended-choice-parameter): 0.76 Plain Credentials Plugin (plain-credentials): 1.4 jQuery plugin (jquery): 1.11.2-0 CloudBees Amazon Web Services Credentials Plugin (aws-credentials): 1.21 Pipeline: Groovy (workflow-cps): 2.39 Durable Task Plugin (durable-task): 1.14 MapDB API Plugin (mapdb-api): 1.0.9.0 JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1 Run Condition Plugin (run-condition): 1.0 Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1 Scriptler (scriptler): 2.9 bouncycastle API Plugin (bouncycastle-api): 2.16.2 Git Changelog (git-changelog): 1.51 Pipeline: Step API (workflow-step-api): 2.12 Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.1.9 Pipeline: Model Definition (pipeline-model-definition): 1.1.9 Node Iterator API Plugin (node-iterator-api): 1.5 Job Configuration History Plugin (jobConfigHistory): 2.17 Display URL API (display-url-api): 2.0 Icon Shim Plugin (icon-shim): 2.0.3 Token Macro Plugin (token-macro): 2.2 Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.1.9 disk-usage plugin (disk-usage): 0.28 Pipeline: Job (workflow-job): 2.14.1 Environment Injector Plugin (envinject): 2.1.3 JUnit Plugin (junit): 1.21 Gerrit Trigger (gerrit-trigger): 2.25.0 Job DSL (job-dsl): 1.64 Ant Plugin (ant): 1.7 Matrix Authorization Strategy Plugin (matrix-auth): 1.7 Cobertura Plugin (cobertura): 1.11 Active Choices Plug-in (uno-choice): 1.5.3 Maven Integration plugin (maven-plugin): 2.17 SCM API Plugin (scm-api): 2.2.1 build timeout plugin (build-timeout): 1.18 HipChat Plugin (hipchat): 2.1.1 Jabber (XMPP) notifier and control plugin (jabber): 1.36 Mercurial plugin (mercurial): 2.0 Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.5 Hidden Parameter plugin (hidden-parameter): 0.0.4 Build Pipeline Plugin (build-pipeline-plugin): 1.5.7.1 Pipeline: Build Step (pipeline-build-step): 2.5.1 Discard Old Build plugin (discard-old-build): 1.05 CVS Plug-in (cvs): 2.13 Pipeline: Stage View Plugin (pipeline-stage-view): 2.8 Workspace Cleanup Plugin (ws-cleanup): 0.34 Mailer Plugin (mailer): 1.20 Resource Disposer Plugin (resource-disposer): 0.7 Matrix Project Plugin (matrix-project): 1.11 Pipeline: Input Step (pipeline-input-step): 2.8 Groovy Postbuild (groovy-postbuild): 2.3.1 Git client plugin (git-client): 2.5.0 Jackson 2 API Plugin (jackson2-api): 2.7.3 JaCoCo plugin (jacoco): 2.2.1 Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1 External Monitor Job Type Plugin (external-monitor-job): 1.7 Multiple SCMs plugin (multiple-scms): 0.6 Pipeline: Nodes and Processes (workflow-durable-task-step): 2.14 instant-messaging plugin (instant-messaging): 1.35 Copy Artifact Plugin (copyartifact): 1.38.1 Python Plugin (python): 1.3 Conditional BuildStep (conditional-buildstep): 1.3.6 SSH Credentials Plugin (ssh-credentials): 1.13 Pipeline: Model API (pipeline-model-api): 1.1.9 Configuration Slicing plugin (configurationslicing): 1.47 Hudson Post build task (postbuild-task): 1.8 Pipeline (workflow-aggregator): 2.5 Pipeline: SCM Step (workflow-scm-step): 2.6 Parameterized Trigger plugin (parameterized-trigger): 2.35.1 Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.8 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5 Pipeline: REST API Plugin (pipeline-rest-api): 2.8 Role-based Authorization Strategy (role-strategy): 2.5.1 Naginator (naginator): 1.17.2 Amazon Web Services SDK (aws-java-sdk): 1.11.119 EnvInject API Plugin (envinject-api): 1.2 Folders Plugin (cloudbees-folder): 6.1.2 Mask Passwords Plugin (mask-passwords): 2.10.1 Javadoc Plugin (javadoc): 1.4 JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1 Git plugin (git): 3.5.1 Email Extension Plugin (email-ext): 2.58 Gitlab Hook Plugin (gitlab-hook): 1.4.2   Here is the  config.xml   Here is the output from the console log after killing the job.   Started by timer Obtained src/Jenkinsfile.CoconutPipeline.groovy from git git@git_server:namespace/job_repo.git [Pipeline] node Running on testops-w2 in /home/jenkins/workspace/Coconut/test-pipeline [Pipeline] { [Pipeline] timestamps [Pipeline] { [Pipeline] hipchatSend*00:00:09.027* [INFO] HipChat notification sent to the following rooms: QA FW, Firmware, QA Automation [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] checkout*00:00:09.139* Cloning the remote Git repository*00:00:09.144* Cloning repository git@git_server:namespace/repo.git Aborted by Username [Pipeline] } [Pipeline] // stage [Pipeline] hipchatSend*01:26:16.377* [INFO] HipChat notification sent to the following rooms: QA FW, Firmware, QA Automation [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:147) at hudson.remoting.Channel.call(Channel.java:838) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) at com.sun.proxy.$Proxy101.execute(Unknown Source) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1075) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Finished: ABORTED   We try to keep Jenkins fairly up to date, and it's really annoying when this happens. It only happens on Pipeline jobs.

          Tapio Reijonen added a comment -

          This happen also in our CI environment:

          Jenkins ver. 2.53

          and plugins are:

          <plugins><shortName>cloudbees-folder</shortName><version>6.1.2</version><shortName>junit</shortName><version>1.21</version><shortName>ssh-credentials</shortName><version>1.13</version><shortName>script-security</shortName><version>1.34</version><shortName>matrix-project</shortName><version>1.11</version><shortName>antisamy-markup-formatter</shortName><version>1.5</version><shortName>github</shortName><version>1.28.0</version><shortName>pam-auth</shortName><version>1.3</version><shortName>git-client</shortName><version>2.5.0</version><shortName>windows-slaves</shortName><version>1.3.1</version><shortName>mailer</shortName><version>1.20</version><shortName>ldap</shortName><version>1.17</version><shortName>token-macro</shortName><version>2.3</version><shortName>mapdb-api</shortName><version>1.0.9.0</version><shortName>icon-shim</shortName><version>2.0.3</version><shortName>matrix-auth</shortName><version>1.5</version><shortName>pipeline-stage-step</shortName><version>2.2</version><shortName>external-monitor-job</shortName><version>1.7</version><shortName>subversion</shortName><version>2.9</version><shortName>build-timeout</shortName><version>1.18</version><shortName>credentials</shortName><version>2.1.15</version><shortName>ssh-slaves</shortName><version>1.15</version><shortName>structs</shortName><version>1.10</version><shortName>workflow-step-api</shortName><version>2.13</version><shortName>workflow-basic-steps</shortName><version>2.6</version><shortName>plain-credentials</shortName><version>1.4</version><shortName>credentials-binding</shortName><version>1.13</version><shortName>timestamper</shortName><version>1.8.8</version><shortName>email-ext</shortName><version>2.59</version><shortName>ws-cleanup</shortName><version>0.34</version><shortName>jira</shortName><version>2.3</version><shortName>ant</shortName><version>1.7</version><shortName>gradle</shortName><version>1.27.1</version><shortName>workflow-aggregator</shortName><version>2.5</version><shortName>jquery-detached</shortName><version>1.2.1</version><shortName>crowd2</shortName><version>1.8</version><shortName>durable-task</shortName><version>1.14</version><shortName>workflow-api</shortName><version>2.20</version><shortName>workflow-support</shortName><version>2.14</version><shortName>monitoring</shortName><version>1.69.1</version><shortName>workflow-job</shortName><version>2.10</version><shortName>javadoc</shortName><version>1.4</version><shortName>pipeline-rest-api</shortName><version>2.9</version><shortName>handlebars</shortName><version>1.1.1</version><shortName>maven-plugin</shortName><version>2.15.1</version><shortName>momentjs</shortName><version>1.1.1</version><shortName>pipeline-stage-view</shortName><version>2.9</version><shortName>github-api</shortName><version>1.86</version><shortName>pipeline-build-step</shortName><version>2.5.1</version><shortName>jobConfigHistory</shortName><version>2.18</version><shortName>ace-editor</shortName><version>1.1</version><shortName>ssh-agent</shortName><version>1.15</version><shortName>workflow-scm-step</shortName><version>2.4</version><shortName>scm-api</shortName><version>2.2.2</version><shortName>workflow-cps</shortName><version>2.40</version><shortName>git-server</shortName><version>1.7</version><shortName>workflow-cps-global-lib</shortName><version>2.9</version><shortName>thinBackup</shortName><version>1.9</version><shortName>branch-api</shortName><version>2.0.11</version><shortName>git</shortName><version>3.5.1</version><shortName>workflow-multibranch</shortName><version>2.16</version><shortName>workflow-durable-task-step</shortName><version>2.15</version><shortName>pipeline-input-step</shortName><version>2.8</version><shortName>urltrigger</shortName><version>0.41</version><shortName>shelve-project-plugin</shortName><version>1.5</version><shortName>jquery</shortName><version>1.11.2-1</version><shortName>analysis-core</shortName><version>1.92</version><shortName>audit-trail</shortName><version>2.2</version><shortName>StashBranchParameter</shortName><version>0.3.0</version><shortName>job-import-plugin</shortName><version>1.5</version><shortName>parameterized-trigger</shortName><version>2.35.2</version><shortName>robot</shortName><version>1.6.4</version><shortName>rebuild</shortName><version>1.25</version><shortName>nested-view</shortName><version>1.14</version><shortName>claim</shortName><version>2.9</version><shortName>stashNotifier</shortName><version>1.11.6</version><shortName>envinject</shortName><version>1.93.1</version><shortName>mask-passwords</shortName><version>2.10.1</version><shortName>github-branch-source</shortName><version>2.0.5</version><shortName>radiatorviewplugin</shortName><version>1.29</version><shortName>display-url-api</shortName><version>2.0</version><shortName>analysis-collector</shortName><version>1.52</version><shortName>publish-over-ssh</shortName><version>1.17</version><shortName>xunit</shortName><version>1.102</version><shortName>job-node-stalker</shortName><version>1.0.5</version><shortName>run-condition</shortName><version>1.0</version><shortName>scp</shortName><version>1.8</version><shortName>msbuild</shortName><version>1.27</version><shortName>bouncycastle-api</shortName><version>2.16.2</version><shortName>git-parameter</shortName><version>0.8.1</version><shortName>cppcheck</shortName><version>1.21</version><shortName>config-file-provider</shortName><version>2.16.4</version><shortName>copyartifact</shortName><version>1.38.1</version><shortName>conditional-buildstep</shortName><version>1.3.6</version><shortName>jacoco</shortName><version>2.2.1</version><shortName>github-organization-folder</shortName><version>1.6</version><shortName>ivy</shortName><version>1.27.1</version><shortName>emma</shortName><version>1.29</version><shortName>build-user-vars-plugin</shortName><version>1.5</version><shortName>sonar</shortName><version>2.6.1</version><shortName>warnings</shortName><version>4.63</version><shortName>xvfb</shortName><version>1.1.3</version><shortName>embeddable-build-status</shortName><version>1.9</version><shortName>text-finder</shortName><version>1.10</version><shortName>pipeline-stage-tags-metadata</shortName><version>1.1.9</version><shortName>docker-commons</shortName><version>1.8</version><shortName>resource-disposer</shortName><version>0.7</version><shortName>docker-workflow</shortName><version>1.13</version><shortName>jenkins-flowdock-plugin</shortName><version>1.1.8</version><shortName>jackson2-api</shortName><version>2.7.3</version><shortName>pipeline-github-lib</shortName><version>1.0</version><shortName>stash-pullrequest-builder</shortName><version>1.7.0</version><shortName>pipeline-model-declarative-agent</shortName><version>1.1.1</version><shortName>pipeline-model-extensions</shortName><version>1.1.4</version><shortName>pipeline-graph-analysis</shortName><version>1.5</version><shortName>checkstyle</shortName><version>3.49</version><shortName>pipeline-milestone-step</shortName><version>1.3.1</version><shortName>pipeline-model-api</shortName><version>1.1.9</version><shortName>pipeline-model-definition</shortName><version>1.1.4</version><shortName>promoted-builds</shortName><version>2.29.1</version><shortName>build-name-setter</shortName><version>1.6.7</version><shortName>multiple-scms</shortName><version>0.6</version><shortName>valgrind</shortName><version>0.27</version><shortName>build-monitor-plugin</shortName><version>1.12+build.201704111018</version><shortName>authentication-tokens</shortName><version>1.3</version><shortName>sitemonitor</shortName><version>0.5</version><shortName>next-build-number</shortName><version>1.4</version><shortName>htmlpublisher</shortName><version>1.14</version><shortName>description-setter</shortName><version>1.10</version></plugins>

          Tapio Reijonen added a comment - This happen also in our CI environment: Jenkins ver. 2.53 and plugins are: <plugins><shortName>cloudbees-folder</shortName><version>6.1.2</version><shortName>junit</shortName><version>1.21</version><shortName>ssh-credentials</shortName><version>1.13</version><shortName>script-security</shortName><version>1.34</version><shortName>matrix-project</shortName><version>1.11</version><shortName>antisamy-markup-formatter</shortName><version>1.5</version><shortName>github</shortName><version>1.28.0</version><shortName>pam-auth</shortName><version>1.3</version><shortName>git-client</shortName><version>2.5.0</version><shortName>windows-slaves</shortName><version>1.3.1</version><shortName>mailer</shortName><version>1.20</version><shortName>ldap</shortName><version>1.17</version><shortName>token-macro</shortName><version>2.3</version><shortName>mapdb-api</shortName><version>1.0.9.0</version><shortName>icon-shim</shortName><version>2.0.3</version><shortName>matrix-auth</shortName><version>1.5</version><shortName>pipeline-stage-step</shortName><version>2.2</version><shortName>external-monitor-job</shortName><version>1.7</version><shortName>subversion</shortName><version>2.9</version><shortName>build-timeout</shortName><version>1.18</version><shortName>credentials</shortName><version>2.1.15</version><shortName>ssh-slaves</shortName><version>1.15</version><shortName>structs</shortName><version>1.10</version><shortName>workflow-step-api</shortName><version>2.13</version><shortName>workflow-basic-steps</shortName><version>2.6</version><shortName>plain-credentials</shortName><version>1.4</version><shortName>credentials-binding</shortName><version>1.13</version><shortName>timestamper</shortName><version>1.8.8</version><shortName>email-ext</shortName><version>2.59</version><shortName>ws-cleanup</shortName><version>0.34</version><shortName>jira</shortName><version>2.3</version><shortName>ant</shortName><version>1.7</version><shortName>gradle</shortName><version>1.27.1</version><shortName>workflow-aggregator</shortName><version>2.5</version><shortName>jquery-detached</shortName><version>1.2.1</version><shortName>crowd2</shortName><version>1.8</version><shortName>durable-task</shortName><version>1.14</version><shortName>workflow-api</shortName><version>2.20</version><shortName>workflow-support</shortName><version>2.14</version><shortName>monitoring</shortName><version>1.69.1</version><shortName>workflow-job</shortName><version>2.10</version><shortName>javadoc</shortName><version>1.4</version><shortName>pipeline-rest-api</shortName><version>2.9</version><shortName>handlebars</shortName><version>1.1.1</version><shortName>maven-plugin</shortName><version>2.15.1</version><shortName>momentjs</shortName><version>1.1.1</version><shortName>pipeline-stage-view</shortName><version>2.9</version><shortName>github-api</shortName><version>1.86</version><shortName>pipeline-build-step</shortName><version>2.5.1</version><shortName>jobConfigHistory</shortName><version>2.18</version><shortName>ace-editor</shortName><version>1.1</version><shortName>ssh-agent</shortName><version>1.15</version><shortName>workflow-scm-step</shortName><version>2.4</version><shortName>scm-api</shortName><version>2.2.2</version><shortName>workflow-cps</shortName><version>2.40</version><shortName>git-server</shortName><version>1.7</version><shortName>workflow-cps-global-lib</shortName><version>2.9</version><shortName>thinBackup</shortName><version>1.9</version><shortName>branch-api</shortName><version>2.0.11</version><shortName>git</shortName><version>3.5.1</version><shortName>workflow-multibranch</shortName><version>2.16</version><shortName>workflow-durable-task-step</shortName><version>2.15</version><shortName>pipeline-input-step</shortName><version>2.8</version><shortName>urltrigger</shortName><version>0.41</version><shortName>shelve-project-plugin</shortName><version>1.5</version><shortName>jquery</shortName><version>1.11.2-1</version><shortName>analysis-core</shortName><version>1.92</version><shortName>audit-trail</shortName><version>2.2</version><shortName>StashBranchParameter</shortName><version>0.3.0</version><shortName>job-import-plugin</shortName><version>1.5</version><shortName>parameterized-trigger</shortName><version>2.35.2</version><shortName>robot</shortName><version>1.6.4</version><shortName>rebuild</shortName><version>1.25</version><shortName>nested-view</shortName><version>1.14</version><shortName>claim</shortName><version>2.9</version><shortName>stashNotifier</shortName><version>1.11.6</version><shortName>envinject</shortName><version>1.93.1</version><shortName>mask-passwords</shortName><version>2.10.1</version><shortName>github-branch-source</shortName><version>2.0.5</version><shortName>radiatorviewplugin</shortName><version>1.29</version><shortName>display-url-api</shortName><version>2.0</version><shortName>analysis-collector</shortName><version>1.52</version><shortName>publish-over-ssh</shortName><version>1.17</version><shortName>xunit</shortName><version>1.102</version><shortName>job-node-stalker</shortName><version>1.0.5</version><shortName>run-condition</shortName><version>1.0</version><shortName>scp</shortName><version>1.8</version><shortName>msbuild</shortName><version>1.27</version><shortName>bouncycastle-api</shortName><version>2.16.2</version><shortName>git-parameter</shortName><version>0.8.1</version><shortName>cppcheck</shortName><version>1.21</version><shortName>config-file-provider</shortName><version>2.16.4</version><shortName>copyartifact</shortName><version>1.38.1</version><shortName>conditional-buildstep</shortName><version>1.3.6</version><shortName>jacoco</shortName><version>2.2.1</version><shortName>github-organization-folder</shortName><version>1.6</version><shortName>ivy</shortName><version>1.27.1</version><shortName>emma</shortName><version>1.29</version><shortName>build-user-vars-plugin</shortName><version>1.5</version><shortName>sonar</shortName><version>2.6.1</version><shortName>warnings</shortName><version>4.63</version><shortName>xvfb</shortName><version>1.1.3</version><shortName>embeddable-build-status</shortName><version>1.9</version><shortName>text-finder</shortName><version>1.10</version><shortName>pipeline-stage-tags-metadata</shortName><version>1.1.9</version><shortName>docker-commons</shortName><version>1.8</version><shortName>resource-disposer</shortName><version>0.7</version><shortName>docker-workflow</shortName><version>1.13</version><shortName>jenkins-flowdock-plugin</shortName><version>1.1.8</version><shortName>jackson2-api</shortName><version>2.7.3</version><shortName>pipeline-github-lib</shortName><version>1.0</version><shortName>stash-pullrequest-builder</shortName><version>1.7.0</version><shortName>pipeline-model-declarative-agent</shortName><version>1.1.1</version><shortName>pipeline-model-extensions</shortName><version>1.1.4</version><shortName>pipeline-graph-analysis</shortName><version>1.5</version><shortName>checkstyle</shortName><version>3.49</version><shortName>pipeline-milestone-step</shortName><version>1.3.1</version><shortName>pipeline-model-api</shortName><version>1.1.9</version><shortName>pipeline-model-definition</shortName><version>1.1.4</version><shortName>promoted-builds</shortName><version>2.29.1</version><shortName>build-name-setter</shortName><version>1.6.7</version><shortName>multiple-scms</shortName><version>0.6</version><shortName>valgrind</shortName><version>0.27</version><shortName>build-monitor-plugin</shortName><version>1.12+build.201704111018</version><shortName>authentication-tokens</shortName><version>1.3</version><shortName>sitemonitor</shortName><version>0.5</version><shortName>next-build-number</shortName><version>1.4</version><shortName>htmlpublisher</shortName><version>1.14</version><shortName>description-setter</shortName><version>1.10</version></plugins>

          James Dumay added a comment -

          Hi there,

          Could you please generate and upload a support bundle using the support core plugin to provide further information here?

          Thanks,
          James

          James Dumay added a comment - Hi there, Could you please generate and upload a support bundle using the support core plugin to provide further information here? Thanks, James

          Michael Neale added a comment -

          A support bundle https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin would be more useful - not sure anything can be reproduced from the above. 

          Michael Neale added a comment - A support bundle https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin  would be more useful - not sure anything can be reproduced from the above. 

          Tapio Reijonen added a comment -

          Add a support bundle support_2017-10-06_04.20.32.zip while several jobs are waiting SCM

          Tapio Reijonen added a comment - Add a support bundle support_2017-10-06_04.20.32.zip while several jobs are waiting SCM

          Brandon Parker added a comment - - edited

          Adding a support bundle for this issue. It is happening a lot today for some reason. 

          support_2017-10-11_21.17.12.zip

           

          Also, here is a stack trace from a hung checkout
          Thread #4
          at DSL.checkout(running in thread: org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution 4262)
          at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:50)
          at DSL.stage(Native Method)
          at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:49)
          at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.doCheckout(jar:file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:40)
          at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy:45)
          at DSL.node(running on jenkins-w11 (jenkinsw-vm5))
          at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy:44)
          at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:372)
          at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:74)
          at WorkflowScript.run(WorkflowScript:13)

          Brandon Parker added a comment - - edited Adding a support bundle for this issue. It is happening a lot today for some reason.  support_2017-10-11_21.17.12.zip   Also, here is a stack trace from a hung checkout Thread #4 at DSL.checkout(running in thread: org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution 4262 ) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar: file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:50 ) at DSL.stage(Native Method) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar: file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:49 ) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.doCheckout(jar: file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:40 ) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(jar: file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy:45 ) at DSL.node(running on jenkins-w11 (jenkinsw-vm5)) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(jar: file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy:44 ) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(jar: file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:372 ) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(jar: file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:74 ) at WorkflowScript.run(WorkflowScript:13)

          Vincent Robert added a comment - - edited

          Hi,

          we have the same problem with a very updated version: 

          Jenkins 2.89.3 (jenkins/jenkins:2.89.3-alpine)

          GKE (k8s 1.8.7 - COS image)

           

          ace-editor 1.1 true
          antisamy-markup-formatter 1.5 true
          apache-httpcomponents-client-4-api 4.5.3-2.1 true
          authentication-tokens 1.3 true
          blueocean 1.4.0 true
          blueocean-autofavorite 1.2.1 true
          blueocean-bitbucket-pipeline 1.4.0 true
          blueocean-commons 1.4.0 true
          blueocean-config 1.4.0 true
          blueocean-core-js 1.4.0 true
          blueocean-dashboard 1.4.0 true
          blueocean-display-url 2.2.0 true
          blueocean-events 1.4.0 true
          blueocean-git-pipeline 1.4.0 true
          blueocean-github-pipeline 1.4.0 true
          blueocean-i18n 1.4.0 true
          blueocean-jira 1.4.0 true
          blueocean-jwt 1.4.0 true
          blueocean-personalization 1.4.0 true
          blueocean-pipeline-api-impl 1.4.0 true
          blueocean-pipeline-editor 1.4.0 true
          blueocean-pipeline-scm-api 1.4.0 true
          blueocean-rest 1.4.0 true
          blueocean-rest-impl 1.4.0 true
          blueocean-web 1.4.0 true
          bouncycastle-api 2.16.2 true
          branch-api 2.0.18 true
          build-user-vars-plugin 1.5 true
          cloudbees-bitbucket-branch-source 2.2.9 true
          cloudbees-folder 6.3 true
          command-launcher 1.2 true
          credentials 2.1.16 true
          credentials-binding 1.15 true
          display-url-api 2.2.0 true
          docker-commons 1.11 true
          docker-workflow 1.15 true
          durable-task 1.17 true
          favorite 2.3.1 true
          git 3.7.0 true
          git-client 2.7.1 true
          git-server 1.7 true
          github 1.29.0 true
          github-api 1.90 true
          github-branch-source 2.3.2 true
          github-oauth 0.29 true
          google-metadata-plugin 0.2 true
          google-oauth-plugin 0.5 true
          handlebars 1.1.1 true
          handy-uri-templates-2-api 2.1.6-1.0 true
          htmlpublisher 1.14 true
          icon-shim 2.0.3 true
          jackson2-api 2.8.10.1 true
          javadoc 1.4 true
          jenkins-design-language 1.4.0 true
          jira 2.5.1 true
          jquery 1.12.4-0 true
          jquery-detached 1.2.1 true
          jsch 0.1.54.1 true
          junit 1.24 true
          kubernetes 1.2 true
          kubernetes-credentials 0.3.0 true
          lockable-resources 2.1 true
          mailer 1.20 true
          matrix-auth 2.2 true
          matrix-project 1.12 true
          maven-plugin 3.1 true
          mercurial 2.2 true
          momentjs 1.1.1 true
          oauth-credentials 0.3 true
          pipeline-build-step 2.7 true
          pipeline-github-lib 1.0 true
          pipeline-graph-analysis 1.6 true
          pipeline-input-step 2.8 true
          pipeline-milestone-step 1.3.1 true
          pipeline-model-api 1.2.7 true
          pipeline-model-declarative-agent 1.1.1 true
          pipeline-model-definition 1.2.7 true
          pipeline-model-extensions 1.2.7 true
          pipeline-rest-api 2.9 true
          pipeline-stage-step 2.3 true
          pipeline-stage-tags-metadata 1.2.7 true
          pipeline-stage-view 2.9 true
          plain-credentials 1.4 true
          pubsub-light 1.12 true
          scm-api 2.2.6 true
          script-security 1.40 true
          slack 2.3 true
          sonar 2.6.1 true
          sse-gateway 1.15 true
          ssh-agent 1.15 true
          ssh-credentials 1.13 true
          ssh-slaves 1.25.1 true
          structs 1.13 true
          token-macro 2.3 true
          variant 1.1 true
          windows-slaves 1.3.1 true
          workflow-aggregator 2.5 true
          workflow-api 2.25 true
          workflow-basic-steps 2.6 true
          workflow-cps 2.44 true
          workflow-cps-global-lib 2.9 true
          workflow-durable-task-step 2.18 true
          workflow-job 2.17 true
          workflow-multibranch 2.17 true
          workflow-scm-step 2.6 true
          workflow-step-api 2.14 true
          workflow-support 2.18 true

           

          The build come in a infinite loop just after the "git rev-list".

          We have to kill the build to stop it.

          The problem appears sporadically, in general a replay successed.

          The problem has appeared in the last 3 weeks.

           
          Push event to branch dev15:11:08 Connecting to https://api.github.com using xxxx/******Obtained Jenkinsfile from xxxx
          Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] lock
          Trying to acquire lock on [xxxx]
          Lock acquired on [xxxx]
          [Pipeline] {
          [Pipeline] podTemplate
          [Pipeline] {
          [Pipeline] node
          Still waiting to schedule task
          Waiting for next available executor on jenkins-slave-w9k9c-5dghv
          Running on
          jenkins-slave-w9k9c-zh83m
          in /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ
          [Pipeline] {
          [Pipeline] checkout
          Cloning the remote Git repository
          Cloning with configured refspecs honoured and without tags
          Cloning repository
          https://github.com/xxx/xxx.git
          > git init /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ # timeout=10
          Fetching upstream changes from
          https://github.com/ xxx/xxx .git
          > git --version # timeout=10
          using GIT_ASKPASS to set credentials
          > git fetch --no-tags --progress
          https://github.com/ xxx/xxx .git
          +refs/heads/dev:refs/remotes/origin/dev
          > git config remote.origin.url
          https://github.com/ xxx/xxx .git

          1. timeout=10
            > git config --add remote.origin.fetch +refs/heads/dev:refs/remotes/origin/dev # timeout=10
            > git config remote.origin.url
            https://github.com/ xxx/xxx .git
          2. timeout=10
            Fetching without tags
            Fetching upstream changes from
            https://github.com/ xxx/xxx .git
            using GIT_ASKPASS to set credentials
            > git fetch --no-tags --progress
            https://github.com/ xxx/xxx .git
            +refs/heads/dev:refs/remotes/origin/dev
            Checking out Revision xxxx (dev)
            > git config core.sparsecheckout # timeout=10
            > git checkout -f xxxx
            Commit message: "Merge tag 'xxxx' into dev"
            > git rev-list --no-walk xxxx # timeout=10

          Vincent Robert added a comment - - edited Hi, we have the same problem with a very updated version:  Jenkins 2.89.3 (jenkins/jenkins:2.89.3-alpine) GKE (k8s 1.8.7 - COS image)   ace-editor 1.1 true antisamy-markup-formatter 1.5 true apache-httpcomponents-client-4-api 4.5.3-2.1 true authentication-tokens 1.3 true blueocean 1.4.0 true blueocean-autofavorite 1.2.1 true blueocean-bitbucket-pipeline 1.4.0 true blueocean-commons 1.4.0 true blueocean-config 1.4.0 true blueocean-core-js 1.4.0 true blueocean-dashboard 1.4.0 true blueocean-display-url 2.2.0 true blueocean-events 1.4.0 true blueocean-git-pipeline 1.4.0 true blueocean-github-pipeline 1.4.0 true blueocean-i18n 1.4.0 true blueocean-jira 1.4.0 true blueocean-jwt 1.4.0 true blueocean-personalization 1.4.0 true blueocean-pipeline-api-impl 1.4.0 true blueocean-pipeline-editor 1.4.0 true blueocean-pipeline-scm-api 1.4.0 true blueocean-rest 1.4.0 true blueocean-rest-impl 1.4.0 true blueocean-web 1.4.0 true bouncycastle-api 2.16.2 true branch-api 2.0.18 true build-user-vars-plugin 1.5 true cloudbees-bitbucket-branch-source 2.2.9 true cloudbees-folder 6.3 true command-launcher 1.2 true credentials 2.1.16 true credentials-binding 1.15 true display-url-api 2.2.0 true docker-commons 1.11 true docker-workflow 1.15 true durable-task 1.17 true favorite 2.3.1 true git 3.7.0 true git-client 2.7.1 true git-server 1.7 true github 1.29.0 true github-api 1.90 true github-branch-source 2.3.2 true github-oauth 0.29 true google-metadata-plugin 0.2 true google-oauth-plugin 0.5 true handlebars 1.1.1 true handy-uri-templates-2-api 2.1.6-1.0 true htmlpublisher 1.14 true icon-shim 2.0.3 true jackson2-api 2.8.10.1 true javadoc 1.4 true jenkins-design-language 1.4.0 true jira 2.5.1 true jquery 1.12.4-0 true jquery-detached 1.2.1 true jsch 0.1.54.1 true junit 1.24 true kubernetes 1.2 true kubernetes-credentials 0.3.0 true lockable-resources 2.1 true mailer 1.20 true matrix-auth 2.2 true matrix-project 1.12 true maven-plugin 3.1 true mercurial 2.2 true momentjs 1.1.1 true oauth-credentials 0.3 true pipeline-build-step 2.7 true pipeline-github-lib 1.0 true pipeline-graph-analysis 1.6 true pipeline-input-step 2.8 true pipeline-milestone-step 1.3.1 true pipeline-model-api 1.2.7 true pipeline-model-declarative-agent 1.1.1 true pipeline-model-definition 1.2.7 true pipeline-model-extensions 1.2.7 true pipeline-rest-api 2.9 true pipeline-stage-step 2.3 true pipeline-stage-tags-metadata 1.2.7 true pipeline-stage-view 2.9 true plain-credentials 1.4 true pubsub-light 1.12 true scm-api 2.2.6 true script-security 1.40 true slack 2.3 true sonar 2.6.1 true sse-gateway 1.15 true ssh-agent 1.15 true ssh-credentials 1.13 true ssh-slaves 1.25.1 true structs 1.13 true token-macro 2.3 true variant 1.1 true windows-slaves 1.3.1 true workflow-aggregator 2.5 true workflow-api 2.25 true workflow-basic-steps 2.6 true workflow-cps 2.44 true workflow-cps-global-lib 2.9 true workflow-durable-task-step 2.18 true workflow-job 2.17 true workflow-multibranch 2.17 true workflow-scm-step 2.6 true workflow-step-api 2.14 true workflow-support 2.18 true   The build come in a infinite loop just after the "git rev-list". We have to kill the build to stop it. The problem appears sporadically, in general a replay successed. The problem has appeared in the last 3 weeks.   Push event to branch dev15:11:08 Connecting to https://api.github.com using xxxx/******Obtained Jenkinsfile from xxxx Running in Durability level: MAX_SURVIVABILITY [Pipeline] lock Trying to acquire lock on [xxxx] Lock acquired on [xxxx] [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] node Still waiting to schedule task Waiting for next available executor on jenkins-slave-w9k9c-5dghv Running on jenkins-slave-w9k9c-zh83m in /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ [Pipeline] { [Pipeline] checkout Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository https://github.com/xxx/xxx.git > git init /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ # timeout=10 Fetching upstream changes from https://github.com/ xxx/xxx .git > git --version # timeout=10 using GIT_ASKPASS to set credentials > git fetch --no-tags --progress https://github.com/ xxx/xxx .git +refs/heads/dev:refs/remotes/origin/dev > git config remote.origin.url https://github.com/ xxx/xxx .git timeout=10 > git config --add remote.origin.fetch +refs/heads/dev:refs/remotes/origin/dev # timeout=10 > git config remote.origin.url https://github.com/ xxx/xxx .git timeout=10 Fetching without tags Fetching upstream changes from https://github.com/ xxx/xxx .git using GIT_ASKPASS to set credentials > git fetch --no-tags --progress https://github.com/ xxx/xxx .git +refs/heads/dev:refs/remotes/origin/dev Checking out Revision xxxx (dev) > git config core.sparsecheckout # timeout=10 > git checkout -f xxxx Commit message: "Merge tag 'xxxx' into dev" > git rev-list --no-walk xxxx # timeout=10

          Michael Neale added a comment -

          vrobert78 I see the same thing - git rev-list too... I haev a support core bundle for it too - taking a look

          Michael Neale added a comment - vrobert78 I see the same thing - git rev-list too... I haev a support core bundle for it too - taking a look

          Michael Neale added a comment - - edited

          My support bundle support_2018-02-07_22.54.25.zip

           Slave-05 is the one that was hanging at the time if that helps. 

          Hey markwaite any ideas on this? This seems to be a recently popped up thing, and it is biting more people (perhaps I should close and re-open this ticket, as the original one doesn't seem to be quite the same). 

          Michael Neale added a comment - - edited My support bundle  support_2018-02-07_22.54.25.zip  Slave-05 is the one that was hanging at the time if that helps.  Hey markwaite any ideas on this? This seems to be a recently popped up thing, and it is biting more people (perhaps I should close and re-open this ticket, as the original one doesn't seem to be quite the same). 

          Michael Neale added a comment -

          could this be the same as https://issues.jenkins-ci.org/browse/JENKINS-46503 - in which case the solution is to use JIRA plugin 2.5.0 (not 2.5.1) - still not totally resolved. 

          Michael Neale added a comment - could this be the same as https://issues.jenkins-ci.org/browse/JENKINS-46503  - in which case the solution is to use JIRA plugin 2.5.0 (not 2.5.1) - still not totally resolved. 

          Vincent Robert added a comment -

          Hi michaelneale, 

          i just disabled JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean. 

          We will see during the day if the problem persists or not.

          We will keep you in touch.

           

          Best regards,

          Vincent Robert added a comment - Hi michaelneale ,  i just disabled JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean.  We will see during the day if the problem persists or not. We will keep you in touch.   Best regards,

          Mark Waite added a comment - - edited

          vrobert78 I don't think it is enough to disable the Jira plugin. I believe that you need to downgrade the Jira plugin from 2.5.1 to 2.5.0 or earlier. There was a discussion of "blacklisting" Jira plugin 2.5.1 and that has been implemented about a week ago in the Jenkins update center.

          Refer to JENKINS-48357 for the details of the types of damage done to critical libraries by the 2.5.1 release.

          Mark Waite added a comment - - edited vrobert78 I don't think it is enough to disable the Jira plugin. I believe that you need to downgrade the Jira plugin from 2.5.1 to 2.5.0 or earlier. There was a discussion of "blacklisting" Jira plugin 2.5.1 and that has been implemented about a week ago in the Jenkins update center . Refer to JENKINS-48357 for the details of the types of damage done to critical libraries by the 2.5.1 release.

          Vincent Robert added a comment -

          markewaite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday.

          Vincent Robert added a comment - markewaite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday.

          Vincent Robert added a comment -

          We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them.

          Indeed, versions has changed:

          • Blue Ocean 1.4.1
          • JIRA Integration for Blue Ocean 1.4.1
          • JIRA Plugin 2.5 (was 2.5.1)

          We hope this will be as fine as yesterday.

          Vincent Robert added a comment - We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them. Indeed, versions has changed: Blue Ocean 1.4.1 JIRA Integration for Blue Ocean 1.4.1 JIRA Plugin 2.5 (was 2.5.1) We hope this will be as fine as yesterday.

          Mark Waite added a comment - - edited

          vrobert78 thanks very much for your investigation and for sharing your results. I'd like to close this bug as a duplicate of JENKINS-48357. JENKINS-48357 will remain open until the fix is released in the Jira plugin. Is that OK?

          Mark Waite added a comment - - edited vrobert78 thanks very much for your investigation and for sharing your results. I'd like to close this bug as a duplicate of JENKINS-48357 . JENKINS-48357 will remain open until the fix is released in the Jira plugin. Is that OK?

          Veaceslav Gaidarji added a comment -

          svanoort ping

          Veaceslav Gaidarji added a comment - svanoort ping

            vrobert78 Vincent Robert
            levt Lev Tartakovsky
            Votes:
            8 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: