Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43106

pipeline job hangs forever at checkout GitSCM - after git rev-list (sporadic)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Duplicate
    • core, git-plugin
    • None

    Description

      Sometimes a pipeline job hangs around checkout. 

      Lately this seems to be after git rev-list. 

       

       > git rev-list --no-walk db25209b84b12cca5214227a7999fe280c82017e # timeout=10

      It will hang here until job is killed. A restart is fine - and it only happens occasionally. 

      This has been more commonly reported since 2018 February/January, a recent update. 

       

      A support bundle of this when it was happening: support_2018-02-07_22.54.25.zip

      IT was happening on slave-05 at the time (can see in its thread dump a few waiting things). Timeout does not work, it will wait indefinitely. 

       

       


      My pipeline job is getting sporadically hung on checkout step.

      Our git repo is relatively small and clone should not take more than 1 min.

      The problem occurring really often once a couple builds. Once it's occurring, the only way that I know to get rid of this state so far is, is disconnect the slave and re-connect it back.

      Build log:

      Started by user <USER>
      Checking out git <REPO> to read Jenkinsfile.bootstrap
      > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
      > git config remote.origin.url <REPO> # timeout=10
      Fetching upstream changes from <REPO>
      > git --version # timeout=10
      using GIT_SSH to set credentials jenkinsslave SSH key
      > git fetch --tags --progress <REPO> +refs/heads/:refs/remotes/origin/
      > git rev-parse refs/remotes/origin/devops_jenkins^{commit} # timeout=10
      > git rev-parse refs/remotes/origin/origin/devops_jenkins^{commit} # timeout=10
      Checking out Revision 2fd998fbe8b23f814b57d8c6eb7b46004268eedc (refs/remotes/origin/devops_jenkins)
      > git config core.sparsecheckout # timeout=10
      > git checkout -f 2fd998fbe8b23f814b57d8c6eb7b46004268eedc
      > git rev-list 2fd998fbe8b23f814b57d8c6eb7b46004268eedc # timeout=10
      [Pipeline] timestamps
      [Pipeline] {
      [Pipeline] node
      00:00:01.249 Running on <SLAVE> in <WORKSPACE>
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Load Jenkinsfile)
      [Pipeline] echo
      00:00:01.267 Branch: origin/devops_jenkins Repo: <REPO>
      [Pipeline] checkout
      00:03:27.565 > git rev-parse -is-inside-work-tree # timeout=10 <- This and further lines are being printed only after slave disconnect.
      00:03:27.566 Fetching changes from the remote Git repository
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] End of Pipeline
      hudson.remoting.RemotingSystemException: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:272)
      at com.sun.proxy.$Proxy98.setRemoteUrl(Unknown Source)
      at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:794)
      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
      at hudson.security.ACL.impersonate(ACL.java:221)
      at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
      at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:191)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:256)
      ... 16 more
      Caused by: hudson.remoting.Channel$OrderlyShutdown
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1121)
      at hudson.remoting.Channel$1.handle(Channel.java:526)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
      Caused by: Command close created at
      at hudson.remoting.Command.<init>(Command.java:59)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1115)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1113)
      at hudson.remoting.Channel.close(Channel.java:1273)
      at hudson.remoting.Channel.close(Channel.java:1255)
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1120)
      ... 2 more
      Finished: FAILURE

      Attachments

        1. Capture.PNG
          Capture.PNG
          96 kB
        2. config.xml
          2 kB
        3. config.xml
          5 kB
        4. Jenkinsfile
          2 kB
        5. support_2017-10-06_04.20.32.zip
          4.47 MB
        6. support_2017-10-11_21.17.12.zip
          2.65 MB
        7. support_2018-02-07_22.54.25.zip
          1.12 MB
        8. threadDump.txt
          68 kB

        Issue Links

          Activity

            My environment is also affected by https://issues.jenkins-ci.org/browse/JENKINS-43038 

            I suspect the both problem have similar root cause that could be related to communication between master and slave.

            levt Lev Tartakovsky added a comment - My environment is also affected by https://issues.jenkins-ci.org/browse/JENKINS-43038   I suspect the both problem have similar root cause that could be related to communication between master and slave.
            jamesdumay James Dumay added a comment -

            levt are you still having issues, even upgrading to a new Jenkins core and updating all plugins?

            jamesdumay James Dumay added a comment - levt are you still having issues, even upgrading to a new Jenkins core and updating all plugins?

            This is a problem that I see on a daily basis. It just hangs at the git cloning step. 

            We are running Ubuntu 14.04.5, Jenkins 2.70, JRE 1.8.0_131-b11 and git 2.11.0 

            We are using the following plugins:

            PAM Authentication plugin (pam-auth): 1.3
            Branch API Plugin (branch-api): 2.0.11
            TextFinder plugin (text-finder): 1.10
            Gitlab Merge Request Builder (gitlab-merge-request-jenkins): 2.0.0
            Docker Commons Plugin (docker-commons): 1.8
            skip-certificate-check (skip-certificate-check): 1.0
            JIRA Pipeline Steps (jira-steps): 1.2.3
            Docker Pipeline (docker-workflow): 1.12
            Subversion Plug-in (subversion): 2.9
            Next Build Number Plugin (next-build-number): 1.4
            Lockable Resources plugin (lockable-resources): 2.0
            Pipeline: Multibranch (workflow-multibranch): 2.16
            GitLab Plugin (gitlab-plugin): 1.4.7
            Git Parameter Plug-In (git-parameter): 0.8.0
            Changes since last successfull build Plugin (changes-since-last-success): 0.5
            Amazon EC2 plugin (ec2): 1.36
            Pipeline: Supporting APIs (workflow-support): 2.14
            LDAP Plugin (ldap): 1.16
            Pipeline: API (workflow-api): 2.20
            JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
            ruby-runtime (ruby-runtime): 0.12
            SSH Slaves plugin (ssh-slaves): 1.21
            Pipeline: Basic Steps (workflow-basic-steps): 2.6
            Script Security Plugin (script-security): 1.33
            Structs Plugin (structs): 1.10
            Windows Slaves Plugin (windows-slaves): 1.3.1
            Translation Assistance plugin (translation): 1.15
            Pipeline: Stage Step (pipeline-stage-step): 2.2
            Credentials Plugin (credentials): 2.1.14
            Authentication Tokens API Plugin (authentication-tokens): 1.3
            JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
            Timestamper (timestamper): 1.8.8
            Credentials Binding Plugin (credentials-binding): 1.13
            GIT server Plugin (git-server): 1.7
            Extended Choice Parameter Plug-In (extended-choice-parameter): 0.76
            Plain Credentials Plugin (plain-credentials): 1.4
            jQuery plugin (jquery): 1.11.2-0
            CloudBees Amazon Web Services Credentials Plugin (aws-credentials): 1.21
            Pipeline: Groovy (workflow-cps): 2.39
            Durable Task Plugin (durable-task): 1.14
            MapDB API Plugin (mapdb-api): 1.0.9.0
            JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
            Run Condition Plugin (run-condition): 1.0
            Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1
            Scriptler (scriptler): 2.9
            bouncycastle API Plugin (bouncycastle-api): 2.16.2
            Git Changelog (git-changelog): 1.51
            Pipeline: Step API (workflow-step-api): 2.12
            Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.1.9
            Pipeline: Model Definition (pipeline-model-definition): 1.1.9
            Node Iterator API Plugin (node-iterator-api): 1.5
            Job Configuration History Plugin (jobConfigHistory): 2.17
            Display URL API (display-url-api): 2.0
            Icon Shim Plugin (icon-shim): 2.0.3
            Token Macro Plugin (token-macro): 2.2
            Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.1.9
            disk-usage plugin (disk-usage): 0.28
            Pipeline: Job (workflow-job): 2.14.1
            Environment Injector Plugin (envinject): 2.1.3
            JUnit Plugin (junit): 1.21
            Gerrit Trigger (gerrit-trigger): 2.25.0
            Job DSL (job-dsl): 1.64
            Ant Plugin (ant): 1.7
            Matrix Authorization Strategy Plugin (matrix-auth): 1.7
            Cobertura Plugin (cobertura): 1.11
            Active Choices Plug-in (uno-choice): 1.5.3
            Maven Integration plugin (maven-plugin): 2.17
            SCM API Plugin (scm-api): 2.2.1
            build timeout plugin (build-timeout): 1.18
            HipChat Plugin (hipchat): 2.1.1
            Jabber (XMPP) notifier and control plugin (jabber): 1.36
            Mercurial plugin (mercurial): 2.0
            Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.5
            Hidden Parameter plugin (hidden-parameter): 0.0.4
            Build Pipeline Plugin (build-pipeline-plugin): 1.5.7.1
            Pipeline: Build Step (pipeline-build-step): 2.5.1
            Discard Old Build plugin (discard-old-build): 1.05
            CVS Plug-in (cvs): 2.13
            Pipeline: Stage View Plugin (pipeline-stage-view): 2.8
            Workspace Cleanup Plugin (ws-cleanup): 0.34
            Mailer Plugin (mailer): 1.20
            Resource Disposer Plugin (resource-disposer): 0.7
            Matrix Project Plugin (matrix-project): 1.11
            Pipeline: Input Step (pipeline-input-step): 2.8
            Groovy Postbuild (groovy-postbuild): 2.3.1
            Git client plugin (git-client): 2.5.0
            Jackson 2 API Plugin (jackson2-api): 2.7.3
            JaCoCo plugin (jacoco): 2.2.1
            Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1
            External Monitor Job Type Plugin (external-monitor-job): 1.7
            Multiple SCMs plugin (multiple-scms): 0.6
            Pipeline: Nodes and Processes (workflow-durable-task-step): 2.14
            instant-messaging plugin (instant-messaging): 1.35
            Copy Artifact Plugin (copyartifact): 1.38.1
            Python Plugin (python): 1.3
            Conditional BuildStep (conditional-buildstep): 1.3.6
            SSH Credentials Plugin (ssh-credentials): 1.13
            Pipeline: Model API (pipeline-model-api): 1.1.9
            Configuration Slicing plugin (configurationslicing): 1.47
            Hudson Post build task (postbuild-task): 1.8
            Pipeline (workflow-aggregator): 2.5
            Pipeline: SCM Step (workflow-scm-step): 2.6
            Parameterized Trigger plugin (parameterized-trigger): 2.35.1
            Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.8
            OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5
            Pipeline: REST API Plugin (pipeline-rest-api): 2.8
            Role-based Authorization Strategy (role-strategy): 2.5.1
            Naginator (naginator): 1.17.2
            Amazon Web Services SDK (aws-java-sdk): 1.11.119
            EnvInject API Plugin (envinject-api): 1.2
            Folders Plugin (cloudbees-folder): 6.1.2
            Mask Passwords Plugin (mask-passwords): 2.10.1
            Javadoc Plugin (javadoc): 1.4
            JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
            Git plugin (git): 3.5.1
            Email Extension Plugin (email-ext): 2.58
            Gitlab Hook Plugin (gitlab-hook): 1.4.2

             

            Here is the config.xml

             

            Here is the output from the console log after killing the job.

             
            Started by timer
            Obtained src/Jenkinsfile.CoconutPipeline.groovy from git git@git_server:namespace/job_repo.git
            [Pipeline] node
            Running on testops-w2 in /home/jenkins/workspace/Coconut/test-pipeline
            [Pipeline] {
            [Pipeline] timestamps
            [Pipeline] {
            [Pipeline] hipchatSend*00:00:09.027* [INFO] HipChat notification sent to the following rooms: QA FW, Firmware, QA Automation
            [Pipeline] stage
            [Pipeline] { (Checkout)
            [Pipeline] checkout*00:00:09.139* Cloning the remote Git repository*00:00:09.144* Cloning repository git@git_server:namespace/repo.git
            Aborted by Username
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] hipchatSend*01:26:16.377* [INFO] HipChat notification sent to the following rooms: QA FW, Firmware, QA Automation
            [Pipeline] }
            [Pipeline] // timestamps
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] End of Pipeline
            java.lang.InterruptedException
            at java.lang.Object.wait(Native Method)
            at hudson.remoting.Request.call(Request.java:147)
            at hudson.remoting.Channel.call(Channel.java:838)
            at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
            at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
            at com.sun.proxy.$Proxy101.execute(Unknown Source)
            at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1075)
            at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
            at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
            at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
            at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
            at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
            at hudson.security.ACL.impersonate(ACL.java:260)
            at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:748)
            Finished: ABORTED
             

            We try to keep Jenkins fairly up to date, and it's really annoying when this happens. It only happens on Pipeline jobs.

            bparker Brandon Parker added a comment - This is a problem that I see on a daily basis. It just hangs at the git cloning step.  We are running Ubuntu 14.04.5, Jenkins 2.70, JRE 1.8.0_131-b11 and git 2.11.0  We are using the following plugins: PAM Authentication plugin (pam-auth): 1.3 Branch API Plugin (branch-api): 2.0.11 TextFinder plugin (text-finder): 1.10 Gitlab Merge Request Builder (gitlab-merge-request-jenkins): 2.0.0 Docker Commons Plugin (docker-commons): 1.8 skip-certificate-check (skip-certificate-check): 1.0 JIRA Pipeline Steps (jira-steps): 1.2.3 Docker Pipeline (docker-workflow): 1.12 Subversion Plug-in (subversion): 2.9 Next Build Number Plugin (next-build-number): 1.4 Lockable Resources plugin (lockable-resources): 2.0 Pipeline: Multibranch (workflow-multibranch): 2.16 GitLab Plugin (gitlab-plugin): 1.4.7 Git Parameter Plug-In (git-parameter): 0.8.0 Changes since last successfull build Plugin (changes-since-last-success): 0.5 Amazon EC2 plugin (ec2): 1.36 Pipeline: Supporting APIs (workflow-support): 2.14 LDAP Plugin (ldap): 1.16 Pipeline: API (workflow-api): 2.20 JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1 ruby-runtime (ruby-runtime): 0.12 SSH Slaves plugin (ssh-slaves): 1.21 Pipeline: Basic Steps (workflow-basic-steps): 2.6 Script Security Plugin (script-security): 1.33 Structs Plugin (structs): 1.10 Windows Slaves Plugin (windows-slaves): 1.3.1 Translation Assistance plugin (translation): 1.15 Pipeline: Stage Step (pipeline-stage-step): 2.2 Credentials Plugin (credentials): 2.1.14 Authentication Tokens API Plugin (authentication-tokens): 1.3 JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1 Timestamper (timestamper): 1.8.8 Credentials Binding Plugin (credentials-binding): 1.13 GIT server Plugin (git-server): 1.7 Extended Choice Parameter Plug-In (extended-choice-parameter): 0.76 Plain Credentials Plugin (plain-credentials): 1.4 jQuery plugin (jquery): 1.11.2-0 CloudBees Amazon Web Services Credentials Plugin (aws-credentials): 1.21 Pipeline: Groovy (workflow-cps): 2.39 Durable Task Plugin (durable-task): 1.14 MapDB API Plugin (mapdb-api): 1.0.9.0 JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1 Run Condition Plugin (run-condition): 1.0 Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1 Scriptler (scriptler): 2.9 bouncycastle API Plugin (bouncycastle-api): 2.16.2 Git Changelog (git-changelog): 1.51 Pipeline: Step API (workflow-step-api): 2.12 Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.1.9 Pipeline: Model Definition (pipeline-model-definition): 1.1.9 Node Iterator API Plugin (node-iterator-api): 1.5 Job Configuration History Plugin (jobConfigHistory): 2.17 Display URL API (display-url-api): 2.0 Icon Shim Plugin (icon-shim): 2.0.3 Token Macro Plugin (token-macro): 2.2 Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.1.9 disk-usage plugin (disk-usage): 0.28 Pipeline: Job (workflow-job): 2.14.1 Environment Injector Plugin (envinject): 2.1.3 JUnit Plugin (junit): 1.21 Gerrit Trigger (gerrit-trigger): 2.25.0 Job DSL (job-dsl): 1.64 Ant Plugin (ant): 1.7 Matrix Authorization Strategy Plugin (matrix-auth): 1.7 Cobertura Plugin (cobertura): 1.11 Active Choices Plug-in (uno-choice): 1.5.3 Maven Integration plugin (maven-plugin): 2.17 SCM API Plugin (scm-api): 2.2.1 build timeout plugin (build-timeout): 1.18 HipChat Plugin (hipchat): 2.1.1 Jabber (XMPP) notifier and control plugin (jabber): 1.36 Mercurial plugin (mercurial): 2.0 Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.5 Hidden Parameter plugin (hidden-parameter): 0.0.4 Build Pipeline Plugin (build-pipeline-plugin): 1.5.7.1 Pipeline: Build Step (pipeline-build-step): 2.5.1 Discard Old Build plugin (discard-old-build): 1.05 CVS Plug-in (cvs): 2.13 Pipeline: Stage View Plugin (pipeline-stage-view): 2.8 Workspace Cleanup Plugin (ws-cleanup): 0.34 Mailer Plugin (mailer): 1.20 Resource Disposer Plugin (resource-disposer): 0.7 Matrix Project Plugin (matrix-project): 1.11 Pipeline: Input Step (pipeline-input-step): 2.8 Groovy Postbuild (groovy-postbuild): 2.3.1 Git client plugin (git-client): 2.5.0 Jackson 2 API Plugin (jackson2-api): 2.7.3 JaCoCo plugin (jacoco): 2.2.1 Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1 External Monitor Job Type Plugin (external-monitor-job): 1.7 Multiple SCMs plugin (multiple-scms): 0.6 Pipeline: Nodes and Processes (workflow-durable-task-step): 2.14 instant-messaging plugin (instant-messaging): 1.35 Copy Artifact Plugin (copyartifact): 1.38.1 Python Plugin (python): 1.3 Conditional BuildStep (conditional-buildstep): 1.3.6 SSH Credentials Plugin (ssh-credentials): 1.13 Pipeline: Model API (pipeline-model-api): 1.1.9 Configuration Slicing plugin (configurationslicing): 1.47 Hudson Post build task (postbuild-task): 1.8 Pipeline (workflow-aggregator): 2.5 Pipeline: SCM Step (workflow-scm-step): 2.6 Parameterized Trigger plugin (parameterized-trigger): 2.35.1 Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.8 OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5 Pipeline: REST API Plugin (pipeline-rest-api): 2.8 Role-based Authorization Strategy (role-strategy): 2.5.1 Naginator (naginator): 1.17.2 Amazon Web Services SDK (aws-java-sdk): 1.11.119 EnvInject API Plugin (envinject-api): 1.2 Folders Plugin (cloudbees-folder): 6.1.2 Mask Passwords Plugin (mask-passwords): 2.10.1 Javadoc Plugin (javadoc): 1.4 JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1 Git plugin (git): 3.5.1 Email Extension Plugin (email-ext): 2.58 Gitlab Hook Plugin (gitlab-hook): 1.4.2   Here is the  config.xml   Here is the output from the console log after killing the job.   Started by timer Obtained src/Jenkinsfile.CoconutPipeline.groovy from git git@git_server:namespace/job_repo.git [Pipeline] node Running on testops-w2 in /home/jenkins/workspace/Coconut/test-pipeline [Pipeline] { [Pipeline] timestamps [Pipeline] { [Pipeline] hipchatSend*00:00:09.027* [INFO] HipChat notification sent to the following rooms: QA FW, Firmware, QA Automation [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] checkout*00:00:09.139* Cloning the remote Git repository*00:00:09.144* Cloning repository git@git_server:namespace/repo.git Aborted by Username [Pipeline] } [Pipeline] // stage [Pipeline] hipchatSend*01:26:16.377* [INFO] HipChat notification sent to the following rooms: QA FW, Firmware, QA Automation [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:147) at hudson.remoting.Channel.call(Channel.java:838) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) at com.sun.proxy.$Proxy101.execute(Unknown Source) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1075) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:260) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Finished: ABORTED   We try to keep Jenkins fairly up to date, and it's really annoying when this happens. It only happens on Pipeline jobs.

            This happen also in our CI environment:

            Jenkins ver. 2.53

            and plugins are:

            <plugins><shortName>cloudbees-folder</shortName><version>6.1.2</version><shortName>junit</shortName><version>1.21</version><shortName>ssh-credentials</shortName><version>1.13</version><shortName>script-security</shortName><version>1.34</version><shortName>matrix-project</shortName><version>1.11</version><shortName>antisamy-markup-formatter</shortName><version>1.5</version><shortName>github</shortName><version>1.28.0</version><shortName>pam-auth</shortName><version>1.3</version><shortName>git-client</shortName><version>2.5.0</version><shortName>windows-slaves</shortName><version>1.3.1</version><shortName>mailer</shortName><version>1.20</version><shortName>ldap</shortName><version>1.17</version><shortName>token-macro</shortName><version>2.3</version><shortName>mapdb-api</shortName><version>1.0.9.0</version><shortName>icon-shim</shortName><version>2.0.3</version><shortName>matrix-auth</shortName><version>1.5</version><shortName>pipeline-stage-step</shortName><version>2.2</version><shortName>external-monitor-job</shortName><version>1.7</version><shortName>subversion</shortName><version>2.9</version><shortName>build-timeout</shortName><version>1.18</version><shortName>credentials</shortName><version>2.1.15</version><shortName>ssh-slaves</shortName><version>1.15</version><shortName>structs</shortName><version>1.10</version><shortName>workflow-step-api</shortName><version>2.13</version><shortName>workflow-basic-steps</shortName><version>2.6</version><shortName>plain-credentials</shortName><version>1.4</version><shortName>credentials-binding</shortName><version>1.13</version><shortName>timestamper</shortName><version>1.8.8</version><shortName>email-ext</shortName><version>2.59</version><shortName>ws-cleanup</shortName><version>0.34</version><shortName>jira</shortName><version>2.3</version><shortName>ant</shortName><version>1.7</version><shortName>gradle</shortName><version>1.27.1</version><shortName>workflow-aggregator</shortName><version>2.5</version><shortName>jquery-detached</shortName><version>1.2.1</version><shortName>crowd2</shortName><version>1.8</version><shortName>durable-task</shortName><version>1.14</version><shortName>workflow-api</shortName><version>2.20</version><shortName>workflow-support</shortName><version>2.14</version><shortName>monitoring</shortName><version>1.69.1</version><shortName>workflow-job</shortName><version>2.10</version><shortName>javadoc</shortName><version>1.4</version><shortName>pipeline-rest-api</shortName><version>2.9</version><shortName>handlebars</shortName><version>1.1.1</version><shortName>maven-plugin</shortName><version>2.15.1</version><shortName>momentjs</shortName><version>1.1.1</version><shortName>pipeline-stage-view</shortName><version>2.9</version><shortName>github-api</shortName><version>1.86</version><shortName>pipeline-build-step</shortName><version>2.5.1</version><shortName>jobConfigHistory</shortName><version>2.18</version><shortName>ace-editor</shortName><version>1.1</version><shortName>ssh-agent</shortName><version>1.15</version><shortName>workflow-scm-step</shortName><version>2.4</version><shortName>scm-api</shortName><version>2.2.2</version><shortName>workflow-cps</shortName><version>2.40</version><shortName>git-server</shortName><version>1.7</version><shortName>workflow-cps-global-lib</shortName><version>2.9</version><shortName>thinBackup</shortName><version>1.9</version><shortName>branch-api</shortName><version>2.0.11</version><shortName>git</shortName><version>3.5.1</version><shortName>workflow-multibranch</shortName><version>2.16</version><shortName>workflow-durable-task-step</shortName><version>2.15</version><shortName>pipeline-input-step</shortName><version>2.8</version><shortName>urltrigger</shortName><version>0.41</version><shortName>shelve-project-plugin</shortName><version>1.5</version><shortName>jquery</shortName><version>1.11.2-1</version><shortName>analysis-core</shortName><version>1.92</version><shortName>audit-trail</shortName><version>2.2</version><shortName>StashBranchParameter</shortName><version>0.3.0</version><shortName>job-import-plugin</shortName><version>1.5</version><shortName>parameterized-trigger</shortName><version>2.35.2</version><shortName>robot</shortName><version>1.6.4</version><shortName>rebuild</shortName><version>1.25</version><shortName>nested-view</shortName><version>1.14</version><shortName>claim</shortName><version>2.9</version><shortName>stashNotifier</shortName><version>1.11.6</version><shortName>envinject</shortName><version>1.93.1</version><shortName>mask-passwords</shortName><version>2.10.1</version><shortName>github-branch-source</shortName><version>2.0.5</version><shortName>radiatorviewplugin</shortName><version>1.29</version><shortName>display-url-api</shortName><version>2.0</version><shortName>analysis-collector</shortName><version>1.52</version><shortName>publish-over-ssh</shortName><version>1.17</version><shortName>xunit</shortName><version>1.102</version><shortName>job-node-stalker</shortName><version>1.0.5</version><shortName>run-condition</shortName><version>1.0</version><shortName>scp</shortName><version>1.8</version><shortName>msbuild</shortName><version>1.27</version><shortName>bouncycastle-api</shortName><version>2.16.2</version><shortName>git-parameter</shortName><version>0.8.1</version><shortName>cppcheck</shortName><version>1.21</version><shortName>config-file-provider</shortName><version>2.16.4</version><shortName>copyartifact</shortName><version>1.38.1</version><shortName>conditional-buildstep</shortName><version>1.3.6</version><shortName>jacoco</shortName><version>2.2.1</version><shortName>github-organization-folder</shortName><version>1.6</version><shortName>ivy</shortName><version>1.27.1</version><shortName>emma</shortName><version>1.29</version><shortName>build-user-vars-plugin</shortName><version>1.5</version><shortName>sonar</shortName><version>2.6.1</version><shortName>warnings</shortName><version>4.63</version><shortName>xvfb</shortName><version>1.1.3</version><shortName>embeddable-build-status</shortName><version>1.9</version><shortName>text-finder</shortName><version>1.10</version><shortName>pipeline-stage-tags-metadata</shortName><version>1.1.9</version><shortName>docker-commons</shortName><version>1.8</version><shortName>resource-disposer</shortName><version>0.7</version><shortName>docker-workflow</shortName><version>1.13</version><shortName>jenkins-flowdock-plugin</shortName><version>1.1.8</version><shortName>jackson2-api</shortName><version>2.7.3</version><shortName>pipeline-github-lib</shortName><version>1.0</version><shortName>stash-pullrequest-builder</shortName><version>1.7.0</version><shortName>pipeline-model-declarative-agent</shortName><version>1.1.1</version><shortName>pipeline-model-extensions</shortName><version>1.1.4</version><shortName>pipeline-graph-analysis</shortName><version>1.5</version><shortName>checkstyle</shortName><version>3.49</version><shortName>pipeline-milestone-step</shortName><version>1.3.1</version><shortName>pipeline-model-api</shortName><version>1.1.9</version><shortName>pipeline-model-definition</shortName><version>1.1.4</version><shortName>promoted-builds</shortName><version>2.29.1</version><shortName>build-name-setter</shortName><version>1.6.7</version><shortName>multiple-scms</shortName><version>0.6</version><shortName>valgrind</shortName><version>0.27</version><shortName>build-monitor-plugin</shortName><version>1.12+build.201704111018</version><shortName>authentication-tokens</shortName><version>1.3</version><shortName>sitemonitor</shortName><version>0.5</version><shortName>next-build-number</shortName><version>1.4</version><shortName>htmlpublisher</shortName><version>1.14</version><shortName>description-setter</shortName><version>1.10</version></plugins>

            tapsa Tapio Reijonen added a comment - This happen also in our CI environment: Jenkins ver. 2.53 and plugins are: <plugins><shortName>cloudbees-folder</shortName><version>6.1.2</version><shortName>junit</shortName><version>1.21</version><shortName>ssh-credentials</shortName><version>1.13</version><shortName>script-security</shortName><version>1.34</version><shortName>matrix-project</shortName><version>1.11</version><shortName>antisamy-markup-formatter</shortName><version>1.5</version><shortName>github</shortName><version>1.28.0</version><shortName>pam-auth</shortName><version>1.3</version><shortName>git-client</shortName><version>2.5.0</version><shortName>windows-slaves</shortName><version>1.3.1</version><shortName>mailer</shortName><version>1.20</version><shortName>ldap</shortName><version>1.17</version><shortName>token-macro</shortName><version>2.3</version><shortName>mapdb-api</shortName><version>1.0.9.0</version><shortName>icon-shim</shortName><version>2.0.3</version><shortName>matrix-auth</shortName><version>1.5</version><shortName>pipeline-stage-step</shortName><version>2.2</version><shortName>external-monitor-job</shortName><version>1.7</version><shortName>subversion</shortName><version>2.9</version><shortName>build-timeout</shortName><version>1.18</version><shortName>credentials</shortName><version>2.1.15</version><shortName>ssh-slaves</shortName><version>1.15</version><shortName>structs</shortName><version>1.10</version><shortName>workflow-step-api</shortName><version>2.13</version><shortName>workflow-basic-steps</shortName><version>2.6</version><shortName>plain-credentials</shortName><version>1.4</version><shortName>credentials-binding</shortName><version>1.13</version><shortName>timestamper</shortName><version>1.8.8</version><shortName>email-ext</shortName><version>2.59</version><shortName>ws-cleanup</shortName><version>0.34</version><shortName>jira</shortName><version>2.3</version><shortName>ant</shortName><version>1.7</version><shortName>gradle</shortName><version>1.27.1</version><shortName>workflow-aggregator</shortName><version>2.5</version><shortName>jquery-detached</shortName><version>1.2.1</version><shortName>crowd2</shortName><version>1.8</version><shortName>durable-task</shortName><version>1.14</version><shortName>workflow-api</shortName><version>2.20</version><shortName>workflow-support</shortName><version>2.14</version><shortName>monitoring</shortName><version>1.69.1</version><shortName>workflow-job</shortName><version>2.10</version><shortName>javadoc</shortName><version>1.4</version><shortName>pipeline-rest-api</shortName><version>2.9</version><shortName>handlebars</shortName><version>1.1.1</version><shortName>maven-plugin</shortName><version>2.15.1</version><shortName>momentjs</shortName><version>1.1.1</version><shortName>pipeline-stage-view</shortName><version>2.9</version><shortName>github-api</shortName><version>1.86</version><shortName>pipeline-build-step</shortName><version>2.5.1</version><shortName>jobConfigHistory</shortName><version>2.18</version><shortName>ace-editor</shortName><version>1.1</version><shortName>ssh-agent</shortName><version>1.15</version><shortName>workflow-scm-step</shortName><version>2.4</version><shortName>scm-api</shortName><version>2.2.2</version><shortName>workflow-cps</shortName><version>2.40</version><shortName>git-server</shortName><version>1.7</version><shortName>workflow-cps-global-lib</shortName><version>2.9</version><shortName>thinBackup</shortName><version>1.9</version><shortName>branch-api</shortName><version>2.0.11</version><shortName>git</shortName><version>3.5.1</version><shortName>workflow-multibranch</shortName><version>2.16</version><shortName>workflow-durable-task-step</shortName><version>2.15</version><shortName>pipeline-input-step</shortName><version>2.8</version><shortName>urltrigger</shortName><version>0.41</version><shortName>shelve-project-plugin</shortName><version>1.5</version><shortName>jquery</shortName><version>1.11.2-1</version><shortName>analysis-core</shortName><version>1.92</version><shortName>audit-trail</shortName><version>2.2</version><shortName>StashBranchParameter</shortName><version>0.3.0</version><shortName>job-import-plugin</shortName><version>1.5</version><shortName>parameterized-trigger</shortName><version>2.35.2</version><shortName>robot</shortName><version>1.6.4</version><shortName>rebuild</shortName><version>1.25</version><shortName>nested-view</shortName><version>1.14</version><shortName>claim</shortName><version>2.9</version><shortName>stashNotifier</shortName><version>1.11.6</version><shortName>envinject</shortName><version>1.93.1</version><shortName>mask-passwords</shortName><version>2.10.1</version><shortName>github-branch-source</shortName><version>2.0.5</version><shortName>radiatorviewplugin</shortName><version>1.29</version><shortName>display-url-api</shortName><version>2.0</version><shortName>analysis-collector</shortName><version>1.52</version><shortName>publish-over-ssh</shortName><version>1.17</version><shortName>xunit</shortName><version>1.102</version><shortName>job-node-stalker</shortName><version>1.0.5</version><shortName>run-condition</shortName><version>1.0</version><shortName>scp</shortName><version>1.8</version><shortName>msbuild</shortName><version>1.27</version><shortName>bouncycastle-api</shortName><version>2.16.2</version><shortName>git-parameter</shortName><version>0.8.1</version><shortName>cppcheck</shortName><version>1.21</version><shortName>config-file-provider</shortName><version>2.16.4</version><shortName>copyartifact</shortName><version>1.38.1</version><shortName>conditional-buildstep</shortName><version>1.3.6</version><shortName>jacoco</shortName><version>2.2.1</version><shortName>github-organization-folder</shortName><version>1.6</version><shortName>ivy</shortName><version>1.27.1</version><shortName>emma</shortName><version>1.29</version><shortName>build-user-vars-plugin</shortName><version>1.5</version><shortName>sonar</shortName><version>2.6.1</version><shortName>warnings</shortName><version>4.63</version><shortName>xvfb</shortName><version>1.1.3</version><shortName>embeddable-build-status</shortName><version>1.9</version><shortName>text-finder</shortName><version>1.10</version><shortName>pipeline-stage-tags-metadata</shortName><version>1.1.9</version><shortName>docker-commons</shortName><version>1.8</version><shortName>resource-disposer</shortName><version>0.7</version><shortName>docker-workflow</shortName><version>1.13</version><shortName>jenkins-flowdock-plugin</shortName><version>1.1.8</version><shortName>jackson2-api</shortName><version>2.7.3</version><shortName>pipeline-github-lib</shortName><version>1.0</version><shortName>stash-pullrequest-builder</shortName><version>1.7.0</version><shortName>pipeline-model-declarative-agent</shortName><version>1.1.1</version><shortName>pipeline-model-extensions</shortName><version>1.1.4</version><shortName>pipeline-graph-analysis</shortName><version>1.5</version><shortName>checkstyle</shortName><version>3.49</version><shortName>pipeline-milestone-step</shortName><version>1.3.1</version><shortName>pipeline-model-api</shortName><version>1.1.9</version><shortName>pipeline-model-definition</shortName><version>1.1.4</version><shortName>promoted-builds</shortName><version>2.29.1</version><shortName>build-name-setter</shortName><version>1.6.7</version><shortName>multiple-scms</shortName><version>0.6</version><shortName>valgrind</shortName><version>0.27</version><shortName>build-monitor-plugin</shortName><version>1.12+build.201704111018</version><shortName>authentication-tokens</shortName><version>1.3</version><shortName>sitemonitor</shortName><version>0.5</version><shortName>next-build-number</shortName><version>1.4</version><shortName>htmlpublisher</shortName><version>1.14</version><shortName>description-setter</shortName><version>1.10</version></plugins>
            jamesdumay James Dumay added a comment -

            Hi there,

            Could you please generate and upload a support bundle using the support core plugin to provide further information here?

            Thanks,
            James

            jamesdumay James Dumay added a comment - Hi there, Could you please generate and upload a support bundle using the support core plugin to provide further information here? Thanks, James
            michaelneale Michael Neale added a comment -

            A support bundle https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin would be more useful - not sure anything can be reproduced from the above. 

            michaelneale Michael Neale added a comment - A support bundle https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin  would be more useful - not sure anything can be reproduced from the above. 

            Add a support bundle support_2017-10-06_04.20.32.zip while several jobs are waiting SCM

            tapsa Tapio Reijonen added a comment - Add a support bundle support_2017-10-06_04.20.32.zip while several jobs are waiting SCM
            bparker Brandon Parker added a comment - - edited

            Adding a support bundle for this issue. It is happening a lot today for some reason. 

            support_2017-10-11_21.17.12.zip

             

            Also, here is a stack trace from a hung checkout
            Thread #4
            at DSL.checkout(running in thread: org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution 4262)
            at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:50)
            at DSL.stage(Native Method)
            at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:49)
            at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.doCheckout(jar:file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:40)
            at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy:45)
            at DSL.node(running on jenkins-w11 (jenkinsw-vm5))
            at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy:44)
            at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:372)
            at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:74)
            at WorkflowScript.run(WorkflowScript:13)

            bparker Brandon Parker added a comment - - edited Adding a support bundle for this issue. It is happening a lot today for some reason.  support_2017-10-11_21.17.12.zip   Also, here is a stack trace from a hung checkout Thread #4 at DSL.checkout(running in thread: org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution 4262 ) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar: file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:50 ) at DSL.stage(Native Method) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar: file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:49 ) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.doCheckout(jar: file:/var/lib/jenkins/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:40 ) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(jar: file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy:45 ) at DSL.node(running on jenkins-w11 (jenkinsw-vm5)) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(jar: file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/LabelScript.groovy:44 ) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(jar: file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:372 ) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(jar: file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:74 ) at WorkflowScript.run(WorkflowScript:13)
            vrobert78 Vincent Robert added a comment - - edited

            Hi,

            we have the same problem with a very updated version: 

            Jenkins 2.89.3 (jenkins/jenkins:2.89.3-alpine)

            GKE (k8s 1.8.7 - COS image)

             

            ace-editor 1.1 true
            antisamy-markup-formatter 1.5 true
            apache-httpcomponents-client-4-api 4.5.3-2.1 true
            authentication-tokens 1.3 true
            blueocean 1.4.0 true
            blueocean-autofavorite 1.2.1 true
            blueocean-bitbucket-pipeline 1.4.0 true
            blueocean-commons 1.4.0 true
            blueocean-config 1.4.0 true
            blueocean-core-js 1.4.0 true
            blueocean-dashboard 1.4.0 true
            blueocean-display-url 2.2.0 true
            blueocean-events 1.4.0 true
            blueocean-git-pipeline 1.4.0 true
            blueocean-github-pipeline 1.4.0 true
            blueocean-i18n 1.4.0 true
            blueocean-jira 1.4.0 true
            blueocean-jwt 1.4.0 true
            blueocean-personalization 1.4.0 true
            blueocean-pipeline-api-impl 1.4.0 true
            blueocean-pipeline-editor 1.4.0 true
            blueocean-pipeline-scm-api 1.4.0 true
            blueocean-rest 1.4.0 true
            blueocean-rest-impl 1.4.0 true
            blueocean-web 1.4.0 true
            bouncycastle-api 2.16.2 true
            branch-api 2.0.18 true
            build-user-vars-plugin 1.5 true
            cloudbees-bitbucket-branch-source 2.2.9 true
            cloudbees-folder 6.3 true
            command-launcher 1.2 true
            credentials 2.1.16 true
            credentials-binding 1.15 true
            display-url-api 2.2.0 true
            docker-commons 1.11 true
            docker-workflow 1.15 true
            durable-task 1.17 true
            favorite 2.3.1 true
            git 3.7.0 true
            git-client 2.7.1 true
            git-server 1.7 true
            github 1.29.0 true
            github-api 1.90 true
            github-branch-source 2.3.2 true
            github-oauth 0.29 true
            google-metadata-plugin 0.2 true
            google-oauth-plugin 0.5 true
            handlebars 1.1.1 true
            handy-uri-templates-2-api 2.1.6-1.0 true
            htmlpublisher 1.14 true
            icon-shim 2.0.3 true
            jackson2-api 2.8.10.1 true
            javadoc 1.4 true
            jenkins-design-language 1.4.0 true
            jira 2.5.1 true
            jquery 1.12.4-0 true
            jquery-detached 1.2.1 true
            jsch 0.1.54.1 true
            junit 1.24 true
            kubernetes 1.2 true
            kubernetes-credentials 0.3.0 true
            lockable-resources 2.1 true
            mailer 1.20 true
            matrix-auth 2.2 true
            matrix-project 1.12 true
            maven-plugin 3.1 true
            mercurial 2.2 true
            momentjs 1.1.1 true
            oauth-credentials 0.3 true
            pipeline-build-step 2.7 true
            pipeline-github-lib 1.0 true
            pipeline-graph-analysis 1.6 true
            pipeline-input-step 2.8 true
            pipeline-milestone-step 1.3.1 true
            pipeline-model-api 1.2.7 true
            pipeline-model-declarative-agent 1.1.1 true
            pipeline-model-definition 1.2.7 true
            pipeline-model-extensions 1.2.7 true
            pipeline-rest-api 2.9 true
            pipeline-stage-step 2.3 true
            pipeline-stage-tags-metadata 1.2.7 true
            pipeline-stage-view 2.9 true
            plain-credentials 1.4 true
            pubsub-light 1.12 true
            scm-api 2.2.6 true
            script-security 1.40 true
            slack 2.3 true
            sonar 2.6.1 true
            sse-gateway 1.15 true
            ssh-agent 1.15 true
            ssh-credentials 1.13 true
            ssh-slaves 1.25.1 true
            structs 1.13 true
            token-macro 2.3 true
            variant 1.1 true
            windows-slaves 1.3.1 true
            workflow-aggregator 2.5 true
            workflow-api 2.25 true
            workflow-basic-steps 2.6 true
            workflow-cps 2.44 true
            workflow-cps-global-lib 2.9 true
            workflow-durable-task-step 2.18 true
            workflow-job 2.17 true
            workflow-multibranch 2.17 true
            workflow-scm-step 2.6 true
            workflow-step-api 2.14 true
            workflow-support 2.18 true

             

            The build come in a infinite loop just after the "git rev-list".

            We have to kill the build to stop it.

            The problem appears sporadically, in general a replay successed.

            The problem has appeared in the last 3 weeks.

             
            Push event to branch dev15:11:08 Connecting to https://api.github.com using xxxx/******Obtained Jenkinsfile from xxxx
            Running in Durability level: MAX_SURVIVABILITY
            [Pipeline] lock
            Trying to acquire lock on [xxxx]
            Lock acquired on [xxxx]
            [Pipeline] {
            [Pipeline] podTemplate
            [Pipeline] {
            [Pipeline] node
            Still waiting to schedule task
            Waiting for next available executor on jenkins-slave-w9k9c-5dghv
            Running on
            jenkins-slave-w9k9c-zh83m
            in /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ
            [Pipeline] {
            [Pipeline] checkout
            Cloning the remote Git repository
            Cloning with configured refspecs honoured and without tags
            Cloning repository
            https://github.com/xxx/xxx.git
            > git init /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ # timeout=10
            Fetching upstream changes from
            https://github.com/ xxx/xxx .git
            > git --version # timeout=10
            using GIT_ASKPASS to set credentials
            > git fetch --no-tags --progress
            https://github.com/ xxx/xxx .git
            +refs/heads/dev:refs/remotes/origin/dev
            > git config remote.origin.url
            https://github.com/ xxx/xxx .git

            1. timeout=10
              > git config --add remote.origin.fetch +refs/heads/dev:refs/remotes/origin/dev # timeout=10
              > git config remote.origin.url
              https://github.com/ xxx/xxx .git
            2. timeout=10
              Fetching without tags
              Fetching upstream changes from
              https://github.com/ xxx/xxx .git
              using GIT_ASKPASS to set credentials
              > git fetch --no-tags --progress
              https://github.com/ xxx/xxx .git
              +refs/heads/dev:refs/remotes/origin/dev
              Checking out Revision xxxx (dev)
              > git config core.sparsecheckout # timeout=10
              > git checkout -f xxxx
              Commit message: "Merge tag 'xxxx' into dev"
              > git rev-list --no-walk xxxx # timeout=10
            vrobert78 Vincent Robert added a comment - - edited Hi, we have the same problem with a very updated version:  Jenkins 2.89.3 (jenkins/jenkins:2.89.3-alpine) GKE (k8s 1.8.7 - COS image)   ace-editor 1.1 true antisamy-markup-formatter 1.5 true apache-httpcomponents-client-4-api 4.5.3-2.1 true authentication-tokens 1.3 true blueocean 1.4.0 true blueocean-autofavorite 1.2.1 true blueocean-bitbucket-pipeline 1.4.0 true blueocean-commons 1.4.0 true blueocean-config 1.4.0 true blueocean-core-js 1.4.0 true blueocean-dashboard 1.4.0 true blueocean-display-url 2.2.0 true blueocean-events 1.4.0 true blueocean-git-pipeline 1.4.0 true blueocean-github-pipeline 1.4.0 true blueocean-i18n 1.4.0 true blueocean-jira 1.4.0 true blueocean-jwt 1.4.0 true blueocean-personalization 1.4.0 true blueocean-pipeline-api-impl 1.4.0 true blueocean-pipeline-editor 1.4.0 true blueocean-pipeline-scm-api 1.4.0 true blueocean-rest 1.4.0 true blueocean-rest-impl 1.4.0 true blueocean-web 1.4.0 true bouncycastle-api 2.16.2 true branch-api 2.0.18 true build-user-vars-plugin 1.5 true cloudbees-bitbucket-branch-source 2.2.9 true cloudbees-folder 6.3 true command-launcher 1.2 true credentials 2.1.16 true credentials-binding 1.15 true display-url-api 2.2.0 true docker-commons 1.11 true docker-workflow 1.15 true durable-task 1.17 true favorite 2.3.1 true git 3.7.0 true git-client 2.7.1 true git-server 1.7 true github 1.29.0 true github-api 1.90 true github-branch-source 2.3.2 true github-oauth 0.29 true google-metadata-plugin 0.2 true google-oauth-plugin 0.5 true handlebars 1.1.1 true handy-uri-templates-2-api 2.1.6-1.0 true htmlpublisher 1.14 true icon-shim 2.0.3 true jackson2-api 2.8.10.1 true javadoc 1.4 true jenkins-design-language 1.4.0 true jira 2.5.1 true jquery 1.12.4-0 true jquery-detached 1.2.1 true jsch 0.1.54.1 true junit 1.24 true kubernetes 1.2 true kubernetes-credentials 0.3.0 true lockable-resources 2.1 true mailer 1.20 true matrix-auth 2.2 true matrix-project 1.12 true maven-plugin 3.1 true mercurial 2.2 true momentjs 1.1.1 true oauth-credentials 0.3 true pipeline-build-step 2.7 true pipeline-github-lib 1.0 true pipeline-graph-analysis 1.6 true pipeline-input-step 2.8 true pipeline-milestone-step 1.3.1 true pipeline-model-api 1.2.7 true pipeline-model-declarative-agent 1.1.1 true pipeline-model-definition 1.2.7 true pipeline-model-extensions 1.2.7 true pipeline-rest-api 2.9 true pipeline-stage-step 2.3 true pipeline-stage-tags-metadata 1.2.7 true pipeline-stage-view 2.9 true plain-credentials 1.4 true pubsub-light 1.12 true scm-api 2.2.6 true script-security 1.40 true slack 2.3 true sonar 2.6.1 true sse-gateway 1.15 true ssh-agent 1.15 true ssh-credentials 1.13 true ssh-slaves 1.25.1 true structs 1.13 true token-macro 2.3 true variant 1.1 true windows-slaves 1.3.1 true workflow-aggregator 2.5 true workflow-api 2.25 true workflow-basic-steps 2.6 true workflow-cps 2.44 true workflow-cps-global-lib 2.9 true workflow-durable-task-step 2.18 true workflow-job 2.17 true workflow-multibranch 2.17 true workflow-scm-step 2.6 true workflow-step-api 2.14 true workflow-support 2.18 true   The build come in a infinite loop just after the "git rev-list". We have to kill the build to stop it. The problem appears sporadically, in general a replay successed. The problem has appeared in the last 3 weeks.   Push event to branch dev15:11:08 Connecting to https://api.github.com using xxxx/******Obtained Jenkinsfile from xxxx Running in Durability level: MAX_SURVIVABILITY [Pipeline] lock Trying to acquire lock on [xxxx] Lock acquired on [xxxx] [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] node Still waiting to schedule task Waiting for next available executor on jenkins-slave-w9k9c-5dghv Running on jenkins-slave-w9k9c-zh83m in /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ [Pipeline] { [Pipeline] checkout Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository https://github.com/xxx/xxx.git > git init /home/jenkins/workspace/xxxx-L44IHDFACBXJ3M2RTOZ4YDAKTOFVJNRK4LJKN7AORR6T4YS5VNEQ # timeout=10 Fetching upstream changes from https://github.com/ xxx/xxx .git > git --version # timeout=10 using GIT_ASKPASS to set credentials > git fetch --no-tags --progress https://github.com/ xxx/xxx .git +refs/heads/dev:refs/remotes/origin/dev > git config remote.origin.url https://github.com/ xxx/xxx .git timeout=10 > git config --add remote.origin.fetch +refs/heads/dev:refs/remotes/origin/dev # timeout=10 > git config remote.origin.url https://github.com/ xxx/xxx .git timeout=10 Fetching without tags Fetching upstream changes from https://github.com/ xxx/xxx .git using GIT_ASKPASS to set credentials > git fetch --no-tags --progress https://github.com/ xxx/xxx .git +refs/heads/dev:refs/remotes/origin/dev Checking out Revision xxxx (dev) > git config core.sparsecheckout # timeout=10 > git checkout -f xxxx Commit message: "Merge tag 'xxxx' into dev" > git rev-list --no-walk xxxx # timeout=10
            michaelneale Michael Neale added a comment -

            vrobert78 I see the same thing - git rev-list too... I haev a support core bundle for it too - taking a look

            michaelneale Michael Neale added a comment - vrobert78 I see the same thing - git rev-list too... I haev a support core bundle for it too - taking a look
            michaelneale Michael Neale added a comment - - edited

            My support bundle support_2018-02-07_22.54.25.zip

             Slave-05 is the one that was hanging at the time if that helps. 

            Hey markwaite any ideas on this? This seems to be a recently popped up thing, and it is biting more people (perhaps I should close and re-open this ticket, as the original one doesn't seem to be quite the same). 

            michaelneale Michael Neale added a comment - - edited My support bundle  support_2018-02-07_22.54.25.zip  Slave-05 is the one that was hanging at the time if that helps.  Hey markwaite any ideas on this? This seems to be a recently popped up thing, and it is biting more people (perhaps I should close and re-open this ticket, as the original one doesn't seem to be quite the same). 
            michaelneale Michael Neale added a comment -

            could this be the same as https://issues.jenkins-ci.org/browse/JENKINS-46503 - in which case the solution is to use JIRA plugin 2.5.0 (not 2.5.1) - still not totally resolved. 

            michaelneale Michael Neale added a comment - could this be the same as https://issues.jenkins-ci.org/browse/JENKINS-46503  - in which case the solution is to use JIRA plugin 2.5.0 (not 2.5.1) - still not totally resolved. 

            Hi michaelneale

            i just disabled JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean. 

            We will see during the day if the problem persists or not.

            We will keep you in touch.

             

            Best regards,

            vrobert78 Vincent Robert added a comment - Hi michaelneale ,  i just disabled JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean.  We will see during the day if the problem persists or not. We will keep you in touch.   Best regards,
            markewaite Mark Waite added a comment - - edited

            vrobert78 I don't think it is enough to disable the Jira plugin. I believe that you need to downgrade the Jira plugin from 2.5.1 to 2.5.0 or earlier. There was a discussion of "blacklisting" Jira plugin 2.5.1 and that has been implemented about a week ago in the Jenkins update center.

            Refer to JENKINS-48357 for the details of the types of damage done to critical libraries by the 2.5.1 release.

            markewaite Mark Waite added a comment - - edited vrobert78 I don't think it is enough to disable the Jira plugin. I believe that you need to downgrade the Jira plugin from 2.5.1 to 2.5.0 or earlier. There was a discussion of "blacklisting" Jira plugin 2.5.1 and that has been implemented about a week ago in the Jenkins update center . Refer to JENKINS-48357 for the details of the types of damage done to critical libraries by the 2.5.1 release.

            markewaite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday.

            vrobert78 Vincent Robert added a comment - markewaite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday.

            We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them.

            Indeed, versions has changed:

            • Blue Ocean 1.4.1
            • JIRA Integration for Blue Ocean 1.4.1
            • JIRA Plugin 2.5 (was 2.5.1)

            We hope this will be as fine as yesterday.

            vrobert78 Vincent Robert added a comment - We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them. Indeed, versions has changed: Blue Ocean 1.4.1 JIRA Integration for Blue Ocean 1.4.1 JIRA Plugin 2.5 (was 2.5.1) We hope this will be as fine as yesterday.
            markewaite Mark Waite added a comment - - edited

            vrobert78 thanks very much for your investigation and for sharing your results. I'd like to close this bug as a duplicate of JENKINS-48357. JENKINS-48357 will remain open until the fix is released in the Jira plugin. Is that OK?

            markewaite Mark Waite added a comment - - edited vrobert78 thanks very much for your investigation and for sharing your results. I'd like to close this bug as a duplicate of JENKINS-48357 . JENKINS-48357 will remain open until the fix is released in the Jira plugin. Is that OK?
            vgaidarji Veaceslav Gaidarji added a comment - svanoort ping

            People

              vrobert78 Vincent Robert
              levt Lev Tartakovsky
              Votes:
              8 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: