Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52362

Jenkins hangs due to "Running CpsFlowExecution unresponsive"

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Blocker
    • Resolution: Unresolved

    Description

      Three times in the last two weeks, we've had our Jenkins server stop responding to requests. When I check syslog, I see errors like this:

      Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
      Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/263:project #263] unresponsive for 5 sec
      Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
      Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/368:project #368] unresponsive for 5 sec
      Jun 30 16:07:18 jenkins [jenkins]: Jun 30, 2018 4:07:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
      Jun 30 16:07:18 jenkins [jenkins]: INFO: Running CpsFlowExecutionOwner[project/318:project #318] unresponsive for 5 sec

      These seem to persist indefinitely and there don't seem to be any other relevant messages in the log. The Web UI just hangs until nginx times out.

      The Java process will then refuse to stop when I try to restart the service and I have to kill it with kill -9.

       

      Attachments

        Activity

          girishkg Girish KG added a comment - - edited

          Unfortunately, this issue exists in 2.3.0.3 as well.
          We are now relying on restarting the Jenkins between 4-8 hours.

          Killing the threads will be quickly loaded with another deadlock, and this goes on until it is restarted.
          We have also observed that this deadlock will not appear for FREE-STYLE jobs.

          This appears only when we have PIPELINES running at around 25 to 40 concurrent JOBS.

          We have 72 Gigs of RAM, 32-CORE CPU, and fast lane disk at 15K RPM, but nothing is saving us other than just a restart.

          It's been 15-days and we are struggling to keep the Jenkins up for at least 12 continuous hours to facilitate long-run needed builds but hitting deadlock.

          So, extremely painful, badly hurt  and now we are migrating long-run jobs to individual Jenkins.

          These are the the documentation and guide I tried.

          2021-11-30 16:44:06.415+0000 [id=87538] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,736ms to load/not load java.io.Serializable$groovy$lang$com$cloudbees$groovy$cps$env$HTTPS_PROXY from classLoader hudson.PluginManager$UberClassLoader
          2021-11-30 16:44:06.413+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/VTY/VTY~vm-provisioning~git/216:SEPG/VTY/VTY~vm-provisioning~git #216]] loading java.io.Serializable$com$cloudbees$groovy$cps$hudson$model$Hudson unresponsive for 13 min
          2021-11-30 16:45:27.977+0000 [id=87816] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 57,825ms to load/not load java/util/groovy/json/env$SEPG_OVERRIDE_NO_PROXY.groovy from classLoader hudson.PluginManager$UberClassLoader
          2021-11-30 16:44:59.173+0000 [id=87338] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,805ms to load/not load java/io/Serializable$com$cloudbees$groovy$cps$hudson$model$Hudson.groovy from classLoader hudson.PluginManager$UberClassLoader
          2021-11-30 16:45:33.645+0000 [id=87801] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,664ms to load/not load groovy.lang.GroovyObject$java$net$jenkins$env$JOB_NAME from classLoader hudson.PluginManager$UberClassLoader
          2021-11-30 16:45:27.980+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/IDSolutions/IDSolutions~eurus-ui~ISOL-883~CI/1:SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI #1]] unresponsive for 12 min
          2021-11-30 16:45:39.405+0000 [id=87338] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 11,423ms to load/not load groovy.lang.GroovyObject$com$cloudbees$groovy$cps$hudson$model$Hudson from classLoader hudson.PluginManager$UberClassLoader
          2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/VTY/VTY~ftp~master~CI/1012:SEPG/VTY/VTY~ftp~master~CI #1012]] unresponsive for 13 min
          2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep 447: checking /tmp/ws on sepg-nix03-DH unresponsive for 13 min
          2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/sepg/sepg~sample-dotnet-6~git/5:SEPG/sepg/sepg~sample-dotnet-6~git #5]] unresponsive for 14 min
          2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/sepg/sepg~SEPGrelBasic~git/99:SEPG/sepg/sepg~SEPGrelBasic~git #99]] unresponsive for 19 min
          2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/VTY/VTY~deploy-helm~git/651:SEPG/VTY/VTY~deploy-helm~git #651]] loading groovy/lang/GroovyObject$com$cloudbees$groovy$cps$hudson$model$Hudson.groovy unresponsive for 15 min
          2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/VTY/VTY~vm-provisioning~git/216:SEPG/VTY/VTY~vm-provisioning~git #216]] unresponsive for 15 min
          2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI/1:SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI #1]] unresponsive for 12 min
          2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecutionOwner[SEPG/VTY/VTY~ftp~master~CI/1012:SEPG/VTY/VTY~ftp~master~CI #1012]] unresponsive for 14 min
          2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep 447: checking /tmp/ws on sepg-nix03-DH unresponsive for 14 min

           

          JENKINS_JAVA_OPTIONS="-Xms1024m \
          -Xmx30720m \
          -XX:+UseG1GC \
          -XX:MaxMetaspaceSize=2048m \
          -XX:MaxGCPauseMillis=200 \
          -XX:+ParallelRefProcEnabled \
          -XX:+AlwaysPreTouch \
          -XX:+UseStringDeduplication \
          -XX:+DisableExplicitGC \
          -XX:+ExplicitGCInvokesConcurrent \
          -Djava.awt.headless=true \"

           

           

          girishkg Girish KG added a comment - - edited Unfortunately, this issue exists in 2.3.0.3 as well. We are now relying on restarting the Jenkins between 4-8 hours. Killing the threads will be quickly loaded with another deadlock, and this goes on until it is restarted. We have also observed that this deadlock will not appear for FREE-STYLE jobs. This appears only when we have PIPELINES running at around 25 to 40 concurrent JOBS. We have 72 Gigs of RAM, 32-CORE CPU, and fast lane disk at 15K RPM, but nothing is saving us other than just a restart. It's been 15-days and we are struggling to keep the Jenkins up for at least 12 continuous hours to facilitate long-run needed builds but hitting deadlock. So, extremely painful, badly hurt  and now we are migrating long-run jobs to individual Jenkins. These are the the documentation and guide I tried. https://www.jenkins.io/blog/2016/11/21/gc-tuning/ https://www.cloudbees.com/blog/joining-big-leagues-tuning-jenkins-gc-responsiveness-and-stability https://www.oracle.com/java/technologies/javase/vmoptions-jsp.html https://wiki.jenkins.io/JENKINS/Consideration-for-Large-Scale-Jenkins-Deployment.html https://www.jenkins.io/blog/2016/11/21/gc-tuning/ https://issues.jenkins.io/browse/JENKINS-29664 https://issues.jenkins.io/browse/JENKINS-59850 https://issues.jenkins.io/browse/JENKINS-62222 2021-11-30 16:44:06.415+0000 [id=87538] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,736ms to load/not load java.io.Serializable$groovy$lang$com$cloudbees$groovy$cps$env$HTTPS_PROXY from classLoader hudson.PluginManager$UberClassLoader 2021-11-30 16:44:06.413+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecution Owner[SEPG/VTY/VTY~vm-provisioning~git/216:SEPG/VTY/VTY~vm-provisioning~git #216] ] loading java.io.Serializable$com$cloudbees$groovy$cps$hudson$model$Hudson unresponsive for 13 min 2021-11-30 16:45:27.977+0000 [id=87816] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 57,825ms to load/not load java/util/groovy/json/env$SEPG_OVERRIDE_NO_PROXY.groovy from classLoader hudson.PluginManager$UberClassLoader 2021-11-30 16:44:59.173+0000 [id=87338] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,805ms to load/not load java/io /Serializable$com$cloudbees$groovy$cps$hudson$model$Hudson.groovy from classLoader hudson.PluginManager$UberClassLoader 2021-11-30 16:45:33.645+0000 [id=87801] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 5,664ms to load/not load groovy.lang.GroovyObject$java$net$jenkins$env$JOB_NAME from classLoader hudson.PluginManager$UberClassLoader 2021-11-30 16:45:27.980+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/IDSolutions/IDSolutions~eurus-ui~ISOL-883~CI/1:SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI #1] ] unresponsive for 12 min 2021-11-30 16:45:39.405+0000 [id=87338] INFO o.j.p.s.s.g.SandboxResolvingClassLoader#lambda$load$2: took 11,423ms to load/not load groovy.lang.GroovyObject$com$cloudbees$groovy$cps$hudson$model$Hudson from classLoader hudson.PluginManager$UberClassLoader 2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/VTY/VTY~ftp~master~CI/1012:SEPG/VTY/VTY~ftp~master~CI #1012] ] unresponsive for 13 min 2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep 447 : checking /tmp/ws on sepg-nix03-DH unresponsive for 13 min 2021-11-30 16:45:39.405+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecution Owner[SEPG/sepg/sepg~sample-dotnet-6~git/5:SEPG/sepg/sepg~sample-dotnet-6~git #5] ] unresponsive for 14 min 2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/sepg/sepg~SEPGrelBasic~git/99:SEPG/sepg/sepg~SEPGrelBasic~git #99] ] unresponsive for 19 min 2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/VTY/VTY~deploy-helm~git/651:SEPG/VTY/VTY~deploy-helm~git #651] ] loading groovy/lang/GroovyObject$com$cloudbees$groovy$cps$hudson$model$Hudson.groovy unresponsive for 15 min 2021-11-30 16:46:02.449+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout#lambda$ping$0: Running CpsFlowExecution Owner[SEPG/VTY/VTY~vm-provisioning~git/216:SEPG/VTY/VTY~vm-provisioning~git #216] ] unresponsive for 15 min 2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI/1:SEPG/IDLSolutions/IDLSolutions~eurus-ui~ISOL-883~CI #1] ] unresponsive for 12 min 2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: Running CpsFlowExecution Owner[SEPG/VTY/VTY~ftp~master~CI/1012:SEPG/VTY/VTY~ftp~master~CI #1012] ] unresponsive for 14 min 2021-11-30 16:46:02.450+0000 [id=1596] INFO o.j.p.w.s.concurrent.Timeout #lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep 447 : checking /tmp/ws on sepg-nix03-DH unresponsive for 14 mi n   JENKINS_JAVA_OPTIONS="-Xms1024m \ -Xmx30720m \ -XX:+UseG1GC \ -XX:MaxMetaspaceSize=2048m \ -XX:MaxGCPauseMillis=200 \ -XX:+ParallelRefProcEnabled \ -XX:+AlwaysPreTouch \ -XX:+UseStringDeduplication \ -XX:+DisableExplicitGC \ -XX:+ExplicitGCInvokesConcurrent \ -Djava.awt.headless=true \"    
          shoelzle Stefan Hölzle added a comment - - edited

          We encounter this issue on a daily basis. Each time Jenkins does not continue any job, uses only one CPU core to about 100% and logs the following INFO message for any running job in the Jenkins log:

          INFO: Running CpsFlowExecution[Owner[FOLDER/repo/branch/306:FOLDER/repo/branch #306]] unresponsive for 45 min
          

          At the time I copied the log message above, Jenkins was already in this state for 45 min. After some more waiting, Jenkins seems to be able to break out of this seemlessly endless state.
          Sometimes this state last for only a few minutes, sometime it lasts for about 1 hour.

          I can provide more information if necessary, please just tell me what you need.

          Jenkins version 2.334 running with Temurin Java 11.
          Installed Plugins:
          com/coravy/hudson/plugins/github/github/1.29.3/github-1.29.3.hpi
          com/datapipe/jenkins/plugins/hashicorp-vault-plugin/336.v182c0fbaaeb7/hashicorp-vault-plugin-336.v182c0fbaaeb7.hpi
          io/jenkins/blueocean/blueocean-bitbucket-pipeline/1.25.2/blueocean-bitbucket-pipeline-1.25.2.hpi
          io/jenkins/blueocean/blueocean-commons/1.25.2/blueocean-commons-1.25.2.hpi
          io/jenkins/blueocean/blueocean-config/1.25.2/blueocean-config-1.25.2.hpi
          io/jenkins/blueocean/blueocean-core-js/1.25.2/blueocean-core-js-1.25.2.hpi
          io/jenkins/blueocean/blueocean-dashboard/1.25.2/blueocean-dashboard-1.25.2.hpi
          io/jenkins/blueocean/blueocean-events/1.25.2/blueocean-events-1.25.2.hpi
          io/jenkins/blueocean/blueocean-git-pipeline/1.25.2/blueocean-git-pipeline-1.25.2.hpi
          io/jenkins/blueocean/blueocean-github-pipeline/1.25.2/blueocean-github-pipeline-1.25.2.hpi
          io/jenkins/blueocean/blueocean-i18n/1.25.2/blueocean-i18n-1.25.2.hpi
          io/jenkins/blueocean/blueocean-jwt/1.25.2/blueocean-jwt-1.25.2.hpi
          io/jenkins/blueocean/blueocean-personalization/1.25.2/blueocean-personalization-1.25.2.hpi
          io/jenkins/blueocean/blueocean-pipeline-api-impl/1.25.2/blueocean-pipeline-api-impl-1.25.2.hpi
          io/jenkins/blueocean/blueocean-pipeline-editor/1.25.2/blueocean-pipeline-editor-1.25.2.hpi
          io/jenkins/blueocean/blueocean-pipeline-scm-api/1.25.2/blueocean-pipeline-scm-api-1.25.2.hpi
          io/jenkins/blueocean/blueocean-rest-impl/1.25.2/blueocean-rest-impl-1.25.2.hpi
          io/jenkins/blueocean/blueocean-rest/1.25.2/blueocean-rest-1.25.2.hpi
          io/jenkins/blueocean/blueocean-web/1.25.2/blueocean-web-1.25.2.hpi
          io/jenkins/blueocean/blueocean/1.25.2/blueocean-1.25.2.hpi
          io/jenkins/blueocean/jenkins-design-language/1.25.2/jenkins-design-language-1.25.2.hpi
          io/jenkins/configuration-as-code/1346.ve8cfa_3473c94/configuration-as-code-1346.ve8cfa_3473c94.hpi
          io/jenkins/plugins/bootstrap4-api/4.6.0-3/bootstrap4-api-4.6.0-3.hpi
          io/jenkins/plugins/bootstrap5-api/5.1.1-1/bootstrap5-api-5.1.1-1.hpi
          io/jenkins/plugins/caffeine-api/2.9.2-29.v717aac953ff3/caffeine-api-2.9.2-29.v717aac953ff3.hpi
          io/jenkins/plugins/checks-api/1.7.2/checks-api-1.7.2.hpi
          io/jenkins/plugins/code-coverage-api/2.0.4/code-coverage-api-2.0.4.hpi
          io/jenkins/plugins/configuration-as-code-groovy/1.1/configuration-as-code-groovy-1.1.hpi
          io/jenkins/plugins/data-tables-api/1.11.3-1/data-tables-api-1.11.3-1.hpi
          io/jenkins/plugins/echarts-api/5.2.1-2/echarts-api-5.2.1-2.hpi
          io/jenkins/plugins/font-awesome-api/5.15.4-1/font-awesome-api-5.15.4-1.hpi
          io/jenkins/plugins/forensics-api/1.5.0/forensics-api-1.5.0.hpi
          io/jenkins/plugins/h2-api/1.4.199/h2-api-1.4.199.hpi
          io/jenkins/plugins/javax-activation-api/1.2.0-2/javax-activation-api-1.2.0-2.hpi
          io/jenkins/plugins/javax-mail-api/1.6.2-2/javax-mail-api-1.6.2-2.hpi
          io/jenkins/plugins/jquery3-api/3.6.0-2/jquery3-api-3.6.0-2.hpi
          io/jenkins/plugins/okhttp-api/3.12.12.2/okhttp-api-3.12.12.2.hpi
          io/jenkins/plugins/plugin-util-api/2.5.1/plugin-util-api-2.5.1.hpi
          io/jenkins/plugins/popper-api<popper-api-1.16.1-2.hpi
          io/jenkins/plugins/popper2-api/2.10.1-1/popper2-api-2.10.1-1.hpi
          io/jenkins/plugins/snakeyaml-api/1.29.1/snakeyaml-api-1.29.1.hpi
          org/6wind/jenkins/lockable-resources/2.3/lockable-resources-2.3.hpi
          org/csanchez/jenkins/plugins/kubernetes/1.31.3/kubernetes-1.31.3.hpi
          org/jenkins-ci/main/maven-plugin/3.17/maven-plugin-3.17.hpi
          org/jenkins-ci/modules/sshd/3.0.4/sshd-3.0.4.hpi
          org/jenkins-ci/plugins/antisamy-markup-formatter/2.7/antisamy-markup-formatter-2.7.hpi
          org/jenkins-ci/plugins/apache-httpcomponents-client-4-api/4.5.13-1.0/apache-httpcomponents-client-4-api-4.5.13-1.0.hpi
          org/jenkins-ci/plugins/authentication-tokens/1.4/authentication-tokens-1.4.hpi
          org/jenkins-ci/plugins/authorize-project/1.4.0/authorize-project-1.4.0.hpi
          org/jenkins-ci/plugins/blueocean-autofavorite/1.2.4/blueocean-autofavorite-1.2.4.hpi
          org/jenkins-ci/plugins/blueocean-display-url/2.4.1/blueocean-display-url-2.4.1.hpi
          org/jenkins-ci/plugins/bouncycastle-api/2.25/bouncycastle-api-2.25.hpi
          org/jenkins-ci/plugins/branch-api/2.7.0/branch-api-2.7.0.hpi
          org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/751.vda_24678a_f781/cloudbees-bitbucket-branch-source-751.vda_24678a_f781.hpi
          org/jenkins-ci/plugins/cloudbees-folder/6.17/cloudbees-folder-6.17.hpi
          org/jenkins-ci/plugins/conditional-buildstep/1.4.1/conditional-buildstep-1.4.1.hpi
          org/jenkins-ci/plugins/config-file-provider/3.9.0/config-file-provider-3.9.0.hpi
          org/jenkins-ci/plugins/credentials-binding/1.27.1/credentials-binding-1.27.1.hpi
          org/jenkins-ci/plugins/credentials/1074.v60e6c29b_b_44b_/credentials-1074.v60e6c29b_b_44b_.hpi
          org/jenkins-ci/plugins/display-url-api/2.3.5/display-url-api-2.3.5.hpi
          org/jenkins-ci/plugins/durable-task/493.v195aefbb0ff2/durable-task-493.v195aefbb0ff2.hpi
          org/jenkins-ci/plugins/git-client/3.11.0/git-client-3.11.0.hpi
          org/jenkins-ci/plugins/git-server/1.10/git-server-1.10.hpi
          org/jenkins-ci/plugins/git/4.10.1/git-4.10.1.hpi
          org/jenkins-ci/plugins/github-api/1.114.2/github-api-1.114.2.hpi
          org/jenkins-ci/plugins/github-branch-source/2.6.0/github-branch-source-2.6.0.hpi
          org/jenkins-ci/plugins/gradle/1.38/gradle-1.38.hpi
          org/jenkins-ci/plugins/handy-uri-templates-2-api/2.1.8-1.0/handy-uri-templates-2-api-2.1.8-1.0.hpi
          org/jenkins-ci/plugins/htmlpublisher/1.25/htmlpublisher-1.25.hpi
          org/jenkins-ci/plugins/http_request/1.14/http_request-1.14.hpi
          org/jenkins-ci/plugins/jackson2-api/2.13.1-246.va8a9f3eaf46a/jackson2-api-2.13.1-246.va8a9f3eaf46a.hpi
          org/jenkins-ci/plugins/javadoc/1.6/javadoc-1.6.hpi
          org/jenkins-ci/plugins/job-dsl/1.78.3/job-dsl-1.78.3.hpi
          org/jenkins-ci/plugins/jsch/0.1.55.2/jsch-0.1.55.2.hpi
          org/jenkins-ci/plugins/junit/1.53/junit-1.53.hpi
          org/jenkins-ci/plugins/kubernetes-client-api/5.11.2-182.v0f1cf4c5904e/kubernetes-client-api-5.11.2-182.v0f1cf4c5904e.hpi
          org/jenkins-ci/plugins/mailer/408.vd726a_1130320/mailer-408.vd726a_1130320.hpi
          org/jenkins-ci/plugins/matrix-auth/3.0.1/matrix-auth-3.0.1.hpi
          org/jenkins-ci/plugins/matrix-project/1.19/matrix-project-1.19.hpi
          org/jenkins-ci/plugins/metrics/4.1.6.1/metrics-4.1.6.1.hpi
          org/jenkins-ci/plugins/nodejs/1.5.1/nodejs-1.5.1.hpi
          org/jenkins-ci/plugins/parameterized-trigger/2.43/parameterized-trigger-2.43.hpi
          org/jenkins-ci/plugins/pipeline-build-step/2.15/pipeline-build-step-2.15.hpi
          org/jenkins-ci/plugins/pipeline-graph-analysis/1.11/pipeline-graph-analysis-1.11.hpi
          org/jenkins-ci/plugins/pipeline-input-step/446.vf27b_0b_83500e/pipeline-input-step-446.vf27b_0b_83500e.hpi
          org/jenkins-ci/plugins/pipeline-maven/3.10.0/pipeline-maven-3.10.0.hpi
          org/jenkins-ci/plugins/pipeline-milestone-step/1.3.2/pipeline-milestone-step-1.3.2.hpi
          org/jenkins-ci/plugins/pipeline-stage-step/291.vf0a8a7aeeb50/pipeline-stage-step-291.vf0a8a7aeeb50.hpi
          org/jenkins-ci/plugins/pipeline-stage-view/pipeline-rest-api/2.15/pipeline-rest-api-2.15.hpi
          org/jenkins-ci/plugins/pipeline-stage-view/pipeline-stage-view/2.10/pipeline-stage-view-2.10.hpi
          org/jenkins-ci/plugins/pipeline-utility-steps/2.12.0/pipeline-utility-steps-2.12.0.hpi
          org/jenkins-ci/plugins/plain-credentials/1.8/plain-credentials-1.8.hpi
          org/jenkins-ci/plugins/prometheus/2.0.10/prometheus-2.0.10.hpi
          org/jenkins-ci/plugins/pubsub-light/1.16/pubsub-light-1.16.hpi
          org/jenkins-ci/plugins/resource-disposer/0.16/resource-disposer-0.16.hpi
          org/jenkins-ci/plugins/run-condition/1.3/run-condition-1.3.hpi
          org/jenkins-ci/plugins/scm-api/595.vd5a_df5eb_0e39/scm-api-595.vd5a_df5eb_0e39.hpi
          org/jenkins-ci/plugins/script-security/1131.v8b_b_5eda_c328e/script-security-1131.v8b_b_5eda_c328e.hpi
          org/jenkins-ci/plugins/simple-theme-plugin/103.va_161d09c38c7/simple-theme-plugin-103.va_161d09c38c7.hpi
          org/jenkins-ci/plugins/sonar/2.14/sonar-2.14.hpi
          org/jenkins-ci/plugins/sse-gateway/1.24/sse-gateway-1.24.hpi
          org/jenkins-ci/plugins/ssh-agent/1.24.1/ssh-agent-1.24.1.hpi
          org/jenkins-ci/plugins/ssh-credentials/1.19/ssh-credentials-1.19.hpi
          org/jenkins-ci/plugins/ssh-steps/2.0.0/ssh-steps-2.0.0.hpi
          org/jenkins-ci/plugins/structs/308.v852b473a2b8c/structs-308.v852b473a2b8c.hpi
          org/jenkins-ci/plugins/timestamper/1.17/timestamper-1.17.hpi
          org/jenkins-ci/plugins/token-macro/277.v7c8f82a_d66b_3/token-macro-277.v7c8f82a_d66b_3.hpi
          org/jenkins-ci/plugins/trilead-api/1.0.13/trilead-api-1.0.13.hpi
          org/jenkins-ci/plugins/variant/1.4/variant-1.4.hpi
          org/jenkins-ci/plugins/webhook-step/89.vfa4b9e961ebf/webhook-step-89.vfa4b9e961ebf.hpi
          org/jenkins-ci/plugins/workflow/workflow-aggregator/2.6/workflow-aggregator-2.6.hpi
          org/jenkins-ci/plugins/workflow/workflow-api/1136.v7f5f1759dc16/workflow-api-1136.v7f5f1759dc16.hpi
          org/jenkins-ci/plugins/workflow/workflow-basic-steps/2.24/workflow-basic-steps-2.24.hpi
          org/jenkins-ci/plugins/workflow/workflow-cps-global-lib/552.vd9cc05b8a2e1/workflow-cps-global-lib-552.vd9cc05b8a2e1.hpi
          org/jenkins-ci/plugins/workflow/workflow-cps/2648.va9433432b33c/workflow-cps-2648.va9433432b33c.hpi
          org/jenkins-ci/plugins/workflow/workflow-durable-task-step/1121.va_65b_d2701486/workflow-durable-task-step-1121.va_65b_d2701486.hpi
          org/jenkins-ci/plugins/workflow/workflow-job/1145.v7f2433caa07f/workflow-job-1145.v7f2433caa07f.hpi
          org/jenkins-ci/plugins/workflow/workflow-multibranch/706.vd43c65dec013/workflow-multibranch-706.vd43c65dec013.hpi
          org/jenkins-ci/plugins/workflow/workflow-scm-step/2.13/workflow-scm-step-2.13.hpi
          org/jenkins-ci/plugins/workflow/workflow-step-api/622.vb_8e7c15b_c95a_/workflow-step-api-622.vb_8e7c15b_c95a_.hpi
          org/jenkins-ci/plugins/workflow/workflow-support/813.vb_d7c3d2984a_0/workflow-support-813.vb_d7c3d2984a_0.hpi
          org/jenkins-ci/plugins/ws-cleanup/0.40/ws-cleanup-0.40.hpi
          org/jenkins-ci/ui/ace-editor/1.1/ace-editor-1.1.hpi
          org/jenkins-ci/ui/handlebars/1.1/handlebars-1.1.hpi
          org/jenkins-ci/ui/jquery-detached/1.2.1/jquery-detached-1.2.1.hpi
          org/jenkins-ci/ui/momentjs/1.1/momentjs-1.1.hpi
          org/jenkinsci/plugins/kubernetes-credentials/0.9.0/kubernetes-credentials-0.9.0.hpi
          org/jenkinsci/plugins/pipeline-model-api/2.2064.v5eef7d0982b_e/pipeline-model-api-2.2064.v5eef7d0982b_e.hpi
          org/jenkinsci/plugins/pipeline-model-definition/2.2064.v5eef7d0982b_e/pipeline-model-definition-2.2064.v5eef7d0982b_e.hpi
          org/jenkinsci/plugins/pipeline-model-extensions/2.2064.v5eef7d0982b_e/pipeline-model-extensions-2.2064.v5eef7d0982b_e.hpi
          org/jenkinsci/plugins/pipeline-stage-tags-metadata/2.2064.v5eef7d0982b_e/pipeline-stage-tags-metadata-2.2064.v5eef7d0982b_e.hpi
          org/jvnet/hudson/plugins/favorite/2.3.1/favorite-2.3.1.hpi
          com/oneandone/jenkins/intranet-login-plugin/2.7/intranet-login-plugin-2.7.hpi

          shoelzle Stefan Hölzle added a comment - - edited We encounter this issue on a daily basis. Each time Jenkins does not continue any job, uses only one CPU core to about 100% and logs the following INFO message for any running job in the Jenkins log: INFO: Running CpsFlowExecution[Owner[FOLDER/repo/branch/306:FOLDER/repo/branch #306]] unresponsive for 45 min At the time I copied the log message above, Jenkins was already in this state for 45 min. After some more waiting, Jenkins seems to be able to break out of this seemlessly endless state. Sometimes this state last for only a few minutes, sometime it lasts for about 1 hour. I can provide more information if necessary, please just tell me what you need. Jenkins version 2.334 running with Temurin Java 11. Installed Plugins: com/coravy/hudson/plugins/github/github/1.29.3/github-1.29.3.hpi com/datapipe/jenkins/plugins/hashicorp-vault-plugin/336.v182c0fbaaeb7/hashicorp-vault-plugin-336.v182c0fbaaeb7.hpi io/jenkins/blueocean/blueocean-bitbucket-pipeline/1.25.2/blueocean-bitbucket-pipeline-1.25.2.hpi io/jenkins/blueocean/blueocean-commons/1.25.2/blueocean-commons-1.25.2.hpi io/jenkins/blueocean/blueocean-config/1.25.2/blueocean-config-1.25.2.hpi io/jenkins/blueocean/blueocean-core-js/1.25.2/blueocean-core-js-1.25.2.hpi io/jenkins/blueocean/blueocean-dashboard/1.25.2/blueocean-dashboard-1.25.2.hpi io/jenkins/blueocean/blueocean-events/1.25.2/blueocean-events-1.25.2.hpi io/jenkins/blueocean/blueocean-git-pipeline/1.25.2/blueocean-git-pipeline-1.25.2.hpi io/jenkins/blueocean/blueocean-github-pipeline/1.25.2/blueocean-github-pipeline-1.25.2.hpi io/jenkins/blueocean/blueocean-i18n/1.25.2/blueocean-i18n-1.25.2.hpi io/jenkins/blueocean/blueocean-jwt/1.25.2/blueocean-jwt-1.25.2.hpi io/jenkins/blueocean/blueocean-personalization/1.25.2/blueocean-personalization-1.25.2.hpi io/jenkins/blueocean/blueocean-pipeline-api-impl/1.25.2/blueocean-pipeline-api-impl-1.25.2.hpi io/jenkins/blueocean/blueocean-pipeline-editor/1.25.2/blueocean-pipeline-editor-1.25.2.hpi io/jenkins/blueocean/blueocean-pipeline-scm-api/1.25.2/blueocean-pipeline-scm-api-1.25.2.hpi io/jenkins/blueocean/blueocean-rest-impl/1.25.2/blueocean-rest-impl-1.25.2.hpi io/jenkins/blueocean/blueocean-rest/1.25.2/blueocean-rest-1.25.2.hpi io/jenkins/blueocean/blueocean-web/1.25.2/blueocean-web-1.25.2.hpi io/jenkins/blueocean/blueocean/1.25.2/blueocean-1.25.2.hpi io/jenkins/blueocean/jenkins-design-language/1.25.2/jenkins-design-language-1.25.2.hpi io/jenkins/configuration-as-code/1346.ve8cfa_3473c94/configuration-as-code-1346.ve8cfa_3473c94.hpi io/jenkins/plugins/bootstrap4-api/4.6.0-3/bootstrap4-api-4.6.0-3.hpi io/jenkins/plugins/bootstrap5-api/5.1.1-1/bootstrap5-api-5.1.1-1.hpi io/jenkins/plugins/caffeine-api/2.9.2-29.v717aac953ff3/caffeine-api-2.9.2-29.v717aac953ff3.hpi io/jenkins/plugins/checks-api/1.7.2/checks-api-1.7.2.hpi io/jenkins/plugins/code-coverage-api/2.0.4/code-coverage-api-2.0.4.hpi io/jenkins/plugins/configuration-as-code-groovy/1.1/configuration-as-code-groovy-1.1.hpi io/jenkins/plugins/data-tables-api/1.11.3-1/data-tables-api-1.11.3-1.hpi io/jenkins/plugins/echarts-api/5.2.1-2/echarts-api-5.2.1-2.hpi io/jenkins/plugins/font-awesome-api/5.15.4-1/font-awesome-api-5.15.4-1.hpi io/jenkins/plugins/forensics-api/1.5.0/forensics-api-1.5.0.hpi io/jenkins/plugins/h2-api/1.4.199/h2-api-1.4.199.hpi io/jenkins/plugins/javax-activation-api/1.2.0-2/javax-activation-api-1.2.0-2.hpi io/jenkins/plugins/javax-mail-api/1.6.2-2/javax-mail-api-1.6.2-2.hpi io/jenkins/plugins/jquery3-api/3.6.0-2/jquery3-api-3.6.0-2.hpi io/jenkins/plugins/okhttp-api/3.12.12.2/okhttp-api-3.12.12.2.hpi io/jenkins/plugins/plugin-util-api/2.5.1/plugin-util-api-2.5.1.hpi io/jenkins/plugins/popper-api<popper-api-1.16.1-2.hpi io/jenkins/plugins/popper2-api/2.10.1-1/popper2-api-2.10.1-1.hpi io/jenkins/plugins/snakeyaml-api/1.29.1/snakeyaml-api-1.29.1.hpi org/6wind/jenkins/lockable-resources/2.3/lockable-resources-2.3.hpi org/csanchez/jenkins/plugins/kubernetes/1.31.3/kubernetes-1.31.3.hpi org/jenkins-ci/main/maven-plugin/3.17/maven-plugin-3.17.hpi org/jenkins-ci/modules/sshd/3.0.4/sshd-3.0.4.hpi org/jenkins-ci/plugins/antisamy-markup-formatter/2.7/antisamy-markup-formatter-2.7.hpi org/jenkins-ci/plugins/apache-httpcomponents-client-4-api/4.5.13-1.0/apache-httpcomponents-client-4-api-4.5.13-1.0.hpi org/jenkins-ci/plugins/authentication-tokens/1.4/authentication-tokens-1.4.hpi org/jenkins-ci/plugins/authorize-project/1.4.0/authorize-project-1.4.0.hpi org/jenkins-ci/plugins/blueocean-autofavorite/1.2.4/blueocean-autofavorite-1.2.4.hpi org/jenkins-ci/plugins/blueocean-display-url/2.4.1/blueocean-display-url-2.4.1.hpi org/jenkins-ci/plugins/bouncycastle-api/2.25/bouncycastle-api-2.25.hpi org/jenkins-ci/plugins/branch-api/2.7.0/branch-api-2.7.0.hpi org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/751.vda_24678a_f781/cloudbees-bitbucket-branch-source-751.vda_24678a_f781.hpi org/jenkins-ci/plugins/cloudbees-folder/6.17/cloudbees-folder-6.17.hpi org/jenkins-ci/plugins/conditional-buildstep/1.4.1/conditional-buildstep-1.4.1.hpi org/jenkins-ci/plugins/config-file-provider/3.9.0/config-file-provider-3.9.0.hpi org/jenkins-ci/plugins/credentials-binding/1.27.1/credentials-binding-1.27.1.hpi org/jenkins-ci/plugins/credentials/1074.v60e6c29b_b_44b_/credentials-1074.v60e6c29b_b_44b_.hpi org/jenkins-ci/plugins/display-url-api/2.3.5/display-url-api-2.3.5.hpi org/jenkins-ci/plugins/durable-task/493.v195aefbb0ff2/durable-task-493.v195aefbb0ff2.hpi org/jenkins-ci/plugins/git-client/3.11.0/git-client-3.11.0.hpi org/jenkins-ci/plugins/git-server/1.10/git-server-1.10.hpi org/jenkins-ci/plugins/git/4.10.1/git-4.10.1.hpi org/jenkins-ci/plugins/github-api/1.114.2/github-api-1.114.2.hpi org/jenkins-ci/plugins/github-branch-source/2.6.0/github-branch-source-2.6.0.hpi org/jenkins-ci/plugins/gradle/1.38/gradle-1.38.hpi org/jenkins-ci/plugins/handy-uri-templates-2-api/2.1.8-1.0/handy-uri-templates-2-api-2.1.8-1.0.hpi org/jenkins-ci/plugins/htmlpublisher/1.25/htmlpublisher-1.25.hpi org/jenkins-ci/plugins/http_request/1.14/http_request-1.14.hpi org/jenkins-ci/plugins/jackson2-api/2.13.1-246.va8a9f3eaf46a/jackson2-api-2.13.1-246.va8a9f3eaf46a.hpi org/jenkins-ci/plugins/javadoc/1.6/javadoc-1.6.hpi org/jenkins-ci/plugins/job-dsl/1.78.3/job-dsl-1.78.3.hpi org/jenkins-ci/plugins/jsch/0.1.55.2/jsch-0.1.55.2.hpi org/jenkins-ci/plugins/junit/1.53/junit-1.53.hpi org/jenkins-ci/plugins/kubernetes-client-api/5.11.2-182.v0f1cf4c5904e/kubernetes-client-api-5.11.2-182.v0f1cf4c5904e.hpi org/jenkins-ci/plugins/mailer/408.vd726a_1130320/mailer-408.vd726a_1130320.hpi org/jenkins-ci/plugins/matrix-auth/3.0.1/matrix-auth-3.0.1.hpi org/jenkins-ci/plugins/matrix-project/1.19/matrix-project-1.19.hpi org/jenkins-ci/plugins/metrics/4.1.6.1/metrics-4.1.6.1.hpi org/jenkins-ci/plugins/nodejs/1.5.1/nodejs-1.5.1.hpi org/jenkins-ci/plugins/parameterized-trigger/2.43/parameterized-trigger-2.43.hpi org/jenkins-ci/plugins/pipeline-build-step/2.15/pipeline-build-step-2.15.hpi org/jenkins-ci/plugins/pipeline-graph-analysis/1.11/pipeline-graph-analysis-1.11.hpi org/jenkins-ci/plugins/pipeline-input-step/446.vf27b_0b_83500e/pipeline-input-step-446.vf27b_0b_83500e.hpi org/jenkins-ci/plugins/pipeline-maven/3.10.0/pipeline-maven-3.10.0.hpi org/jenkins-ci/plugins/pipeline-milestone-step/1.3.2/pipeline-milestone-step-1.3.2.hpi org/jenkins-ci/plugins/pipeline-stage-step/291.vf0a8a7aeeb50/pipeline-stage-step-291.vf0a8a7aeeb50.hpi org/jenkins-ci/plugins/pipeline-stage-view/pipeline-rest-api/2.15/pipeline-rest-api-2.15.hpi org/jenkins-ci/plugins/pipeline-stage-view/pipeline-stage-view/2.10/pipeline-stage-view-2.10.hpi org/jenkins-ci/plugins/pipeline-utility-steps/2.12.0/pipeline-utility-steps-2.12.0.hpi org/jenkins-ci/plugins/plain-credentials/1.8/plain-credentials-1.8.hpi org/jenkins-ci/plugins/prometheus/2.0.10/prometheus-2.0.10.hpi org/jenkins-ci/plugins/pubsub-light/1.16/pubsub-light-1.16.hpi org/jenkins-ci/plugins/resource-disposer/0.16/resource-disposer-0.16.hpi org/jenkins-ci/plugins/run-condition/1.3/run-condition-1.3.hpi org/jenkins-ci/plugins/scm-api/595.vd5a_df5eb_0e39/scm-api-595.vd5a_df5eb_0e39.hpi org/jenkins-ci/plugins/script-security/1131.v8b_b_5eda_c328e/script-security-1131.v8b_b_5eda_c328e.hpi org/jenkins-ci/plugins/simple-theme-plugin/103.va_161d09c38c7/simple-theme-plugin-103.va_161d09c38c7.hpi org/jenkins-ci/plugins/sonar/2.14/sonar-2.14.hpi org/jenkins-ci/plugins/sse-gateway/1.24/sse-gateway-1.24.hpi org/jenkins-ci/plugins/ssh-agent/1.24.1/ssh-agent-1.24.1.hpi org/jenkins-ci/plugins/ssh-credentials/1.19/ssh-credentials-1.19.hpi org/jenkins-ci/plugins/ssh-steps/2.0.0/ssh-steps-2.0.0.hpi org/jenkins-ci/plugins/structs/308.v852b473a2b8c/structs-308.v852b473a2b8c.hpi org/jenkins-ci/plugins/timestamper/1.17/timestamper-1.17.hpi org/jenkins-ci/plugins/token-macro/277.v7c8f82a_d66b_3/token-macro-277.v7c8f82a_d66b_3.hpi org/jenkins-ci/plugins/trilead-api/1.0.13/trilead-api-1.0.13.hpi org/jenkins-ci/plugins/variant/1.4/variant-1.4.hpi org/jenkins-ci/plugins/webhook-step/89.vfa4b9e961ebf/webhook-step-89.vfa4b9e961ebf.hpi org/jenkins-ci/plugins/workflow/workflow-aggregator/2.6/workflow-aggregator-2.6.hpi org/jenkins-ci/plugins/workflow/workflow-api/1136.v7f5f1759dc16/workflow-api-1136.v7f5f1759dc16.hpi org/jenkins-ci/plugins/workflow/workflow-basic-steps/2.24/workflow-basic-steps-2.24.hpi org/jenkins-ci/plugins/workflow/workflow-cps-global-lib/552.vd9cc05b8a2e1/workflow-cps-global-lib-552.vd9cc05b8a2e1.hpi org/jenkins-ci/plugins/workflow/workflow-cps/2648.va9433432b33c/workflow-cps-2648.va9433432b33c.hpi org/jenkins-ci/plugins/workflow/workflow-durable-task-step/1121.va_65b_d2701486/workflow-durable-task-step-1121.va_65b_d2701486.hpi org/jenkins-ci/plugins/workflow/workflow-job/1145.v7f2433caa07f/workflow-job-1145.v7f2433caa07f.hpi org/jenkins-ci/plugins/workflow/workflow-multibranch/706.vd43c65dec013/workflow-multibranch-706.vd43c65dec013.hpi org/jenkins-ci/plugins/workflow/workflow-scm-step/2.13/workflow-scm-step-2.13.hpi org/jenkins-ci/plugins/workflow/workflow-step-api/622.vb_8e7c15b_c95a_/workflow-step-api-622.vb_8e7c15b_c95a_.hpi org/jenkins-ci/plugins/workflow/workflow-support/813.vb_d7c3d2984a_0/workflow-support-813.vb_d7c3d2984a_0.hpi org/jenkins-ci/plugins/ws-cleanup/0.40/ws-cleanup-0.40.hpi org/jenkins-ci/ui/ace-editor/1.1/ace-editor-1.1.hpi org/jenkins-ci/ui/handlebars/1.1/handlebars-1.1.hpi org/jenkins-ci/ui/jquery-detached/1.2.1/jquery-detached-1.2.1.hpi org/jenkins-ci/ui/momentjs/1.1/momentjs-1.1.hpi org/jenkinsci/plugins/kubernetes-credentials/0.9.0/kubernetes-credentials-0.9.0.hpi org/jenkinsci/plugins/pipeline-model-api/2.2064.v5eef7d0982b_e/pipeline-model-api-2.2064.v5eef7d0982b_e.hpi org/jenkinsci/plugins/pipeline-model-definition/2.2064.v5eef7d0982b_e/pipeline-model-definition-2.2064.v5eef7d0982b_e.hpi org/jenkinsci/plugins/pipeline-model-extensions/2.2064.v5eef7d0982b_e/pipeline-model-extensions-2.2064.v5eef7d0982b_e.hpi org/jenkinsci/plugins/pipeline-stage-tags-metadata/2.2064.v5eef7d0982b_e/pipeline-stage-tags-metadata-2.2064.v5eef7d0982b_e.hpi org/jvnet/hudson/plugins/favorite/2.3.1/favorite-2.3.1.hpi com/oneandone/jenkins/intranet-login-plugin/2.7/intranet-login-plugin-2.7.hpi
          ganthore Mark Austin added a comment - - edited

          Edit: Root cause was the performance plugin running perfReport on ~240mb worth of test data.

          We're experiencing a similar issue for the past 2 days.

          Attached is a thread dump. ganthore-threads.dump

          We're running on core 2.344 and latest pipelines plugins.

          ganthore Mark Austin added a comment - - edited Edit: Root cause was the performance plugin running perfReport on ~240mb worth of test data. We're experiencing a similar issue for the past 2 days. Attached is a thread dump. ganthore-threads.dump We're running on core 2.344 and latest pipelines plugins.
          imqyh yuhang qiu added a comment -

          Version: Jenkins 2.355 with latest plugins.

          Problem:

          1. Some job cannot finish even if it's actually done, and keeps logging same output
          2. Nodes of those jobs report unresponsive for 5 sec/ 10 sec... ,and the time can reset to 5 sec(means it's not a dead lock)
          3. Other job on those node cannot start or finish
          4. jstack jenkins, jobs are waiting for a thread. And there is tee in this thread backtrace. (I forgot to snapshot/save it...)

          Reason:

          1. We use tee in pipeline
          2. Maybe cmd in tee does not close fd properly, or unstable network(packet loss for EOF), or tee self has bugs
          3. Such huge and infinite outputs occupy the lock for too long.
          4. The node start unresponsive, and the unresponsive time will reset when lock is get.
          5. Jobs on those node cannot start or finish.

          After we remove all tee in all jobs, the problem disappeared. But the reason might be different in other user's reports.

          If you got tee in your backtrace, might be the tee problem, you should try to remove it.

          Report by: Aliyun PolarDB Testing team.

          imqyh yuhang qiu added a comment - Version: Jenkins 2.355 with latest plugins. Problem: Some job cannot finish even if it's actually done, and keeps logging same output Nodes of those jobs report unresponsive for 5 sec/ 10 sec... ,and the time can reset to 5 sec(means it's not a dead lock) Other job on those node cannot start or finish jstack jenkins, jobs are waiting for a thread. And there is tee in this thread backtrace. (I forgot to snapshot/save it...) Reason: We use tee in pipeline Maybe cmd in tee does not close fd properly, or unstable network(packet loss for EOF), or tee self has bugs Such huge and infinite outputs occupy the lock for too long. The node start unresponsive, and the unresponsive time will reset when lock is get. Jobs on those node cannot start or finish. After we remove all tee in all jobs, the problem disappeared. But the reason might be different in other user's reports. If you got tee in your backtrace, might be the tee problem, you should try to remove it. Report by: Aliyun PolarDB Testing team.
          mcascone Max Cascone added a comment -

          We are having this same problem, I'm not sure if it's related to the CPS issue of the original subject or not. Several times a day, jobs will just stop, at various spots in their pipeline. These are all declarative pipelines, Linux controller, windows agent. There is plenty of disk space.

          Jenkins 2.346.2
          mostly all updated plugins

          that -Xmx256m looks suspiciously low

          This is from the jenkins.xml:

          <executable>/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin</executable> -->
            <arguments>-Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djsse.enableSNIExtension=false -jar "%BASE%\jenkins.war" --httpPort=8080</arguments>
          

          memory:

          $ free -h
                        total        used        free      shared  buff/cache   available
          Mem:           7.6G        3.6G        1.2G        256M        2.9G        3.5G
          Swap:          1.0G        345M        678M
          

          Should we bump up the -Xmx to 4g?

          mcascone Max Cascone added a comment - We are having this same problem, I'm not sure if it's related to the CPS issue of the original subject or not. Several times a day, jobs will just stop, at various spots in their pipeline. These are all declarative pipelines, Linux controller, windows agent. There is plenty of disk space. Jenkins 2.346.2 mostly all updated plugins that -Xmx256m looks suspiciously low This is from the jenkins.xml: <executable> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin </executable> --> <arguments> -Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djsse.enableSNIExtension=false -jar "%BASE%\jenkins.war" --httpPort=8080 </arguments> memory: $ free -h total used free shared buff/cache available Mem: 7.6G 3.6G 1.2G 256M 2.9G 3.5G Swap: 1.0G 345M 678M Should we bump up the -Xmx to 4g?

          People

            Unassigned Unassigned
            pdouglas Philip Douglas
            Votes:
            33 Vote for this issue
            Watchers:
            48 Start watching this issue

            Dates

              Created:
              Updated: