Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-74957

Jenkins eventually stops responding properly and running jobs since 2.479.1

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • core
    • None

      Jenkins 2.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

      A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

      This week, the Build Time Trend has stopped loading properly again, and then it started to accumulate a growing build queue and I had to restart it (I downgraded back to 2.462.3 at the same time2, as it's our production instance and we can't afford much downtime).

      I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

      Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

      "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
      	at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
      	at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
      	at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
      	at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
      	at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
      	at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
      	at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
      	at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
      	at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
      	at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
      	at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
      	at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
      	at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
      	at hudson.model.Run.getLogText(Run.java:1505)
      	at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
      	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
      	at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
      	at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
      	at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
      	at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      	at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      	at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)
      
      	Number of locked synchronizers = 1
      	- java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
      

      I also have a job stuck waiting for the previous build to complete, but it already has:

      The previous build "finished" with these logs, but it's still showing the animated dots below that:

      Errors were encountered
      Build step 'Execute shell' marked build as failure
      Sending e-mails to: xxx@xxx.com
      Notifying upstream projects of job completion
      

      And the thread dump shows:

      "Executor #0 for lhr-vexec02-fast : executing nexus_db_replicate_roles #295805" Id=146231 Group=main WAITING on java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4c072eb0
      at java.base@17.0.13/jdk.internal.misc.Unsafe.park(Native Method)

      • waiting on java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4c072eb0
        at java.base@17.0.13/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
        at java.base@17.0.13/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:447)
        at java.base@17.0.13/java.util.concurrent.FutureTask.get(FutureTask.java:190)
        at hudson.tasks.BuildTrigger.execute(BuildTrigger.java:268)
        at hudson.model.AbstractBuild$AbstractBuildExecution.cleanUp(AbstractBuild.java:728)
        at hudson.model.Build$BuildExecution.cleanUp(Build.java:194)
        at hudson.model.Run.execute(Run.java:1874)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
        at hudson.model.ResourceController.execute(ResourceController.java:101)
        at hudson.model.Executor.run(Executor.java:445)

      Many jobs are stuck waiting on the same object, I'm going to have to restart Jenkins soon.

      • waiting on java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4c072eb0

      It seems from the logs that jobs start getting completed but not deleted (that usually are deleted afterwards) and then end up stuck in the build queue:

      Dec 05 09:54:59 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:54:59.420+0000 [id=145920]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: blackops_import_marketaxess_arm_respo>
      Dec 05 09:55:57 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:55:57.102+0000 [id=145912]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: intraday_alloc_barcap_fo #96214
      Dec 05 09:55:57 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:55:57.121+0000 [id=145912]        INFO        o.j.p.l.queue.LockRunListener#onDeleted: intraday_alloc_barcap_fo #95820
      Dec 05 09:55:57 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:55:57.402+0000 [id=145916]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: intraday_recon_ms_futures #436179
      Dec 05 09:55:57 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:55:57.412+0000 [id=145916]        INFO        o.j.p.l.queue.LockRunListener#onDeleted: intraday_recon_ms_futures #435398
      Dec 05 09:56:01 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:56:01.741+0000 [id=145882]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: report_blackops_margin_spot_consolida>
      Dec 05 09:56:01 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:56:01.752+0000 [id=145882]        INFO        o.j.p.l.queue.LockRunListener#onDeleted: report_blackops_margin_spot_consolidate>
      Dec 05 09:56:10 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:56:10.959+0000 [id=140281]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: nexus_db_mirror_checker #171910
      Dec 05 09:56:10 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:56:10.969+0000 [id=140281]        INFO        o.j.p.l.queue.LockRunListener#onDeleted: nexus_db_mirror_checker #171775
      Dec 05 09:57:02 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:57:02.155+0000 [id=145926]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: blackops_import_ubs_pb_cash_positions>
      Dec 05 09:57:02 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:57:02.165+0000 [id=145926]        INFO        o.j.p.l.queue.LockRunListener#onDeleted: blackops_import_ubs_pb_cash_positions_c>
      Dec 05 09:57:12 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:57:12.472+0000 [id=138985]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: nexus_db_replicate_roles #295798
      Dec 05 09:57:12 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:57:12.489+0000 [id=138985]        INFO        o.j.p.l.queue.LockRunListener#onDeleted: nexus_db_replicate_roles #295642
      Dec 05 09:57:20 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:57:20.796+0000 [id=145857]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: intraday_block_ubs_fx #228536
      Dec 05 09:57:20 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:57:20.810+0000 [id=145857]        INFO        o.j.p.l.queue.LockRunListener#onDeleted: intraday_block_ubs_fx #228114
      Dec 05 09:57:21 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:57:21.544+0000 [id=138981]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: intraday_alloc_jpm_lme_futures #81195
      Dec 05 09:57:21 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:57:21.549+0000 [id=138981]        INFO        o.j.p.l.queue.LockRunListener#onDeleted: intraday_alloc_jpm_lme_futures #81015
      Dec 05 09:59:52 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:59:52.735+0000 [id=145802]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: it_cam_make_file_windows-netapp-cifs >
      Dec 05 09:59:52 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 09:59:52.741+0000 [id=145802]        INFO        o.j.p.l.queue.LockRunListener#onDeleted: it_cam_make_file_windows-netapp-cifs #2>
      Dec 05 10:00:02 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 10:00:02.271+0000 [id=145951]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: it_cam_change_file_linux-netapp-nfs #>
      Dec 05 10:00:02 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 10:00:02.288+0000 [id=145951]        INFO        o.j.p.l.queue.LockRunListener#onDeleted: it_cam_change_file_linux-netapp-nfs #29>
      Dec 05 10:00:07 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 10:00:07.582+0000 [id=145977]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: blackops_import_marketaxess_arm_respo>
      Dec 05 10:00:09 jenkins-node-01 jenkins-prod[2672733]: 2024-12-05 10:00:09.267+0000 [id=145990]        INFO        o.j.p.l.queue.LockRunListener#onCompleted: intraday_alloc_ne_lme #146288
      

          [JENKINS-74957] Jenkins eventually stops responding properly and running jobs since 2.479.1

          Chris Wilson created issue -
          Chris Wilson made changes -
          Environment Original: Jenkins: 2.479.1
          OS: Linux - 4.18.0-553.27.1.el8_10.x86_64
          Java: 17.0.13 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
          ---
          Office-365-Connector:5.0.0
          ant:511.v0a_a_1a_334f41b_
          antisamy-markup-formatter:162.v0e6ec0fcfcf6
          apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
          asm-api:9.7.1-97.v4cc844130d97
          audit-trail:382.vf64d6f626060
          authentication-tokens:1.119.v50285141b_7e1
          authorize-project:1.8.1
          bootstrap5-api:5.3.3-1
          bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
          branch-api:2.1200.v4b_a_3da_2eb_db_4
          build-timeout:1.33
          build-user-vars-plugin:182.v378b_9f14b_487
          buildresult-trigger:0.18
          caffeine-api:3.1.8-133.v17b_1ff2e0599
          checks-api:2.2.1
          chucknorris:159.vdfe649cb_9c37
          cloudbees-folder:6.955.v81e2a_35c08d3
          command-launcher:116.vd85919c54a_d6
          commons-httpclient3-api:3.1-3
          commons-lang3-api:3.17.0-84.vb_b_938040b_078
          commons-text-api:1.12.0-129.v99a_50df237f7
          conditional-buildstep:1.4.3
          configurationslicing:648.v57e7f26a_6e49
          credentials:1389.vd7a_b_f5fa_50a_2
          credentials-binding:687.v619cb_15e923f
          cvs:469.v57a_96d4f6886
          data-tables-api:2.1.8-1
          depgraph-view:1.0.5
          display-url-api:2.209.v582ed814ff2f
          docker-commons:445.v6b_646c962a_94
          docker-workflow:580.vc0c340686b_54
          durable-task:577.v2a_8a_4b_7c0247
          echarts-api:5.5.1-4
          eddsa-api:0.3.0-4.v84c6f0f4969e
          elastic-axis:615.vb_f7c20ec040e
          envinject-api:1.199.v3ce31253ed13
          extended-read-permission:53.v6499940139e5
          external-monitor-job:215.v2e88e894db_f8
          font-awesome-api:6.6.0-2
          fstrigger:1.00
          git:5.6.0
          git-client:6.1.0
          git-server:126.v0d945d8d2b_39
          gitlab-plugin:1.9.6
          gson-api:2.11.0-85.v1f4e87273c33
          instance-identity:201.vd2a_b_5a_468a_a_6
          ionicons-api:74.v93d5eb_813d5f
          ivytrigger:1.02
          jackson2-api:2.17.0-379.v02de8ec9f64c
          jakarta-activation-api:2.1.3-1
          jakarta-mail-api:2.1.3-1
          javadoc:280.v050b_5c849f69
          javax-activation-api:1.2.0-7
          javax-mail-api:1.6.2-10
          jaxb:2.3.9-1
          jdk-tool:80.v8a_dee33ed6f0
          jersey2-api:2.44-151.v6df377fff741
          job-dsl:1.90
          job-restrictions:0.8
          jobConfigHistory:1283.veb_dfb_00b_5ec0
          joda-time-api:2.13.0-93.v9934da_29b_a_e9
          join:1.22-SNAPSHOT (private-682cfed6-kanara)
          jquery:1.12.4-3
          jquery3-api:3.7.1-2
          jsch:0.2.16-86.v42e010d9484b_
          json-api:20240303-101.v7a_8666713110
          json-path-api:2.9.0-118.v7f23ed82a_8b_8
          junit:1309.v0078b_fecd6ed
          label-linked-jobs:6.0.1
          ldap:770.vb_455e934581a_
          lockable-resources:1327.ved786b_a_197e0
          logfilesizechecker:1.5
          mailer:489.vd4b_25144138f
          mapdb-api:1.0.9-40.v58107308b_7a_7
          matrix-auth:3.2.3
          matrix-project:840.v812f627cb_578
          maven-plugin:3.24
          mina-sshd-api-common:2.14.0-133.vcc091215a_358
          mina-sshd-api-core:2.14.0-133.vcc091215a_358
          naginator:1.481.vcb_b_384a_3de89
          nested-view:1.36
          nodelabelparameter:1.13.0
          pam-auth:1.11
          parameterized-trigger:806.vf6fff3e28c3e
          pipeline-build-step:540.vb_e8849e1a_b_d8
          pipeline-graph-analysis:216.vfd8b_ece330ca_
          pipeline-groovy-lib:744.v5b_556ee7c253
          pipeline-input-step:495.ve9c153f6067b_
          pipeline-milestone-step:119.vdfdc43fc3b_9a_
          pipeline-model-api:2.2218.v56d0cda_37c72
          pipeline-model-definition:2.2218.v56d0cda_37c72
          pipeline-model-extensions:2.2218.v56d0cda_37c72
          pipeline-rest-api:2.34
          pipeline-stage-step:312.v8cd10304c27a_
          pipeline-stage-tags-metadata:2.2218.v56d0cda_37c72
          pipeline-stage-view:2.34
          plain-credentials:183.va_de8f1dd5a_2b_
          plugin-util-api:5.1.0
          reverse-proxy-auth-plugin:238.v82ceca_8417a_6
          role-strategy:743.v142ea_b_d5f1d3
          run-condition:1.7
          scm-api:698.v8e3b_c788f0a_6
          script-security:1369.v9b_98a_4e95b_2d
          snakeyaml-api:2.3-123.v13484c65210a_
          ssh-credentials:343.v884f71d78167
          ssh-slaves:2.973.v0fa_8c0dea_f9f
          sshd:3.330.vc866a_8389b_58
          status-view:1.0
          structs:338.v848422169819
          subversion:1281.vc8837f91a_07a_
          token-macro:400.v35420b_922dcb_
          trilead-api:2.147.vb_73cc728a_32e
          urltrigger:1.02
          variant:60.v7290fc0eb_b_cd
          versioncolumn:243.vda_c20eea_a_8a_f
          view-job-filters:392.v2c0a_4dd46909
          workflow-aggregator:600.vb_57cdd26fdd7
          workflow-api:1336.vee415d95c521
          workflow-basic-steps:1058.vcb_fc1e3a_21a_9
          workflow-cps:3993.v3e20a_37282f8
          workflow-durable-task-step:1378.v6a_3e903058a_3
          workflow-job:1468.vcf4f5ee92395
          workflow-multibranch:795.ve0cb_1f45ca_9a_
          workflow-scm-step:427.v4ca_6512e7df1
          workflow-step-api:678.v3ee58b_469476
          workflow-support:932.vb_555de1b_a_b_94
          xtrigger-api:1.0
          New: Accessing via Nginx reverse proxy with Duo two-factor authentication.

          Jenkins: 2.479.1
          OS: Linux - 4.18.0-553.27.1.el8_10.x86_64
          Java: 17.0.13 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
          ---
          Office-365-Connector:5.0.0
          ant:511.v0a_a_1a_334f41b_
          antisamy-markup-formatter:162.v0e6ec0fcfcf6
          apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
          asm-api:9.7.1-97.v4cc844130d97
          audit-trail:382.vf64d6f626060
          authentication-tokens:1.119.v50285141b_7e1
          authorize-project:1.8.1
          bootstrap5-api:5.3.3-1
          bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
          branch-api:2.1200.v4b_a_3da_2eb_db_4
          build-timeout:1.33
          build-user-vars-plugin:182.v378b_9f14b_487
          buildresult-trigger:0.18
          caffeine-api:3.1.8-133.v17b_1ff2e0599
          checks-api:2.2.1
          chucknorris:159.vdfe649cb_9c37
          cloudbees-folder:6.955.v81e2a_35c08d3
          command-launcher:116.vd85919c54a_d6
          commons-httpclient3-api:3.1-3
          commons-lang3-api:3.17.0-84.vb_b_938040b_078
          commons-text-api:1.12.0-129.v99a_50df237f7
          conditional-buildstep:1.4.3
          configurationslicing:648.v57e7f26a_6e49
          credentials:1389.vd7a_b_f5fa_50a_2
          credentials-binding:687.v619cb_15e923f
          cvs:469.v57a_96d4f6886
          data-tables-api:2.1.8-1
          depgraph-view:1.0.5
          display-url-api:2.209.v582ed814ff2f
          docker-commons:445.v6b_646c962a_94
          docker-workflow:580.vc0c340686b_54
          durable-task:577.v2a_8a_4b_7c0247
          echarts-api:5.5.1-4
          eddsa-api:0.3.0-4.v84c6f0f4969e
          elastic-axis:615.vb_f7c20ec040e
          envinject-api:1.199.v3ce31253ed13
          extended-read-permission:53.v6499940139e5
          external-monitor-job:215.v2e88e894db_f8
          font-awesome-api:6.6.0-2
          fstrigger:1.00
          git:5.6.0
          git-client:6.1.0
          git-server:126.v0d945d8d2b_39
          gitlab-plugin:1.9.6
          gson-api:2.11.0-85.v1f4e87273c33
          instance-identity:201.vd2a_b_5a_468a_a_6
          ionicons-api:74.v93d5eb_813d5f
          ivytrigger:1.02
          jackson2-api:2.17.0-379.v02de8ec9f64c
          jakarta-activation-api:2.1.3-1
          jakarta-mail-api:2.1.3-1
          javadoc:280.v050b_5c849f69
          javax-activation-api:1.2.0-7
          javax-mail-api:1.6.2-10
          jaxb:2.3.9-1
          jdk-tool:80.v8a_dee33ed6f0
          jersey2-api:2.44-151.v6df377fff741
          job-dsl:1.90
          job-restrictions:0.8
          jobConfigHistory:1283.veb_dfb_00b_5ec0
          joda-time-api:2.13.0-93.v9934da_29b_a_e9
          join:1.22-SNAPSHOT (private-682cfed6-kanara)
          jquery:1.12.4-3
          jquery3-api:3.7.1-2
          jsch:0.2.16-86.v42e010d9484b_
          json-api:20240303-101.v7a_8666713110
          json-path-api:2.9.0-118.v7f23ed82a_8b_8
          junit:1309.v0078b_fecd6ed
          label-linked-jobs:6.0.1
          ldap:770.vb_455e934581a_
          lockable-resources:1327.ved786b_a_197e0
          logfilesizechecker:1.5
          mailer:489.vd4b_25144138f
          mapdb-api:1.0.9-40.v58107308b_7a_7
          matrix-auth:3.2.3
          matrix-project:840.v812f627cb_578
          maven-plugin:3.24
          mina-sshd-api-common:2.14.0-133.vcc091215a_358
          mina-sshd-api-core:2.14.0-133.vcc091215a_358
          naginator:1.481.vcb_b_384a_3de89
          nested-view:1.36
          nodelabelparameter:1.13.0
          pam-auth:1.11
          parameterized-trigger:806.vf6fff3e28c3e
          pipeline-build-step:540.vb_e8849e1a_b_d8
          pipeline-graph-analysis:216.vfd8b_ece330ca_
          pipeline-groovy-lib:744.v5b_556ee7c253
          pipeline-input-step:495.ve9c153f6067b_
          pipeline-milestone-step:119.vdfdc43fc3b_9a_
          pipeline-model-api:2.2218.v56d0cda_37c72
          pipeline-model-definition:2.2218.v56d0cda_37c72
          pipeline-model-extensions:2.2218.v56d0cda_37c72
          pipeline-rest-api:2.34
          pipeline-stage-step:312.v8cd10304c27a_
          pipeline-stage-tags-metadata:2.2218.v56d0cda_37c72
          pipeline-stage-view:2.34
          plain-credentials:183.va_de8f1dd5a_2b_
          plugin-util-api:5.1.0
          reverse-proxy-auth-plugin:238.v82ceca_8417a_6
          role-strategy:743.v142ea_b_d5f1d3
          run-condition:1.7
          scm-api:698.v8e3b_c788f0a_6
          script-security:1369.v9b_98a_4e95b_2d
          snakeyaml-api:2.3-123.v13484c65210a_
          ssh-credentials:343.v884f71d78167
          ssh-slaves:2.973.v0fa_8c0dea_f9f
          sshd:3.330.vc866a_8389b_58
          status-view:1.0
          structs:338.v848422169819
          subversion:1281.vc8837f91a_07a_
          token-macro:400.v35420b_922dcb_
          trilead-api:2.147.vb_73cc728a_32e
          urltrigger:1.02
          variant:60.v7290fc0eb_b_cd
          versioncolumn:243.vda_c20eea_a_8a_f
          view-job-filters:392.v2c0a_4dd46909
          workflow-aggregator:600.vb_57cdd26fdd7
          workflow-api:1336.vee415d95c521
          workflow-basic-steps:1058.vcb_fc1e3a_21a_9
          workflow-cps:3993.v3e20a_37282f8
          workflow-durable-task-step:1378.v6a_3e903058a_3
          workflow-job:1468.vcf4f5ee92395
          workflow-multibranch:795.ve0cb_1f45ca_9a_
          workflow-scm-step:427.v4ca_6512e7df1
          workflow-step-api:678.v3ee58b_469476
          workflow-support:932.vb_555de1b_a_b_94
          xtrigger-api:1.0
          Chris Wilson made changes -
          Description Original: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1.

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.
          New: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.
          Chris Wilson made changes -
          Description Original: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.
          New: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          Chris Wilson made changes -
          Description Original: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          New: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          {code:java}
          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          {code}
          Chris Wilson made changes -
          Attachment New: stuck-job.png [ 63680 ]
          Chris Wilson made changes -
          Description Original: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          {code:java}
          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          {code}
          New: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          {code:java}
          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          {code}

          I also have a job stuck waiting for the previous one to complete, but it already has:

           !stuck-job.png|thumbnail!
          Chris Wilson made changes -
          Description Original: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          {code:java}
          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          {code}

          I also have a job stuck waiting for the previous one to complete, but it already has:

           !stuck-job.png|thumbnail!
          New: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          {code:java}
          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          {code}

          I also have a job stuck waiting for the previous build to complete, but it already has:

           !stuck-job.png|thumbnail!
          Chris Wilson made changes -
          Description Original: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          {code:java}
          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          {code}

          I also have a job stuck waiting for the previous build to complete, but it already has:

           !stuck-job.png|thumbnail!
          New: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          {code:java}
          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          {code}

          I also have a job stuck waiting for the previous build to complete, but it already has:

           !stuck-job.png|thumbnail!

          The previous build "finished" with these logs, but it's still showing the animated dots below that:



          {noformat}
          Errors were encountered
          Build step 'Execute shell' marked build as failure
          Sending e-mails to: xxx@xxx.com
          Notifying upstream projects of job completion

          {noformat}

          Chris Wilson made changes -
          Description Original: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          {code:java}
          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          {code}

          I also have a job stuck waiting for the previous build to complete, but it already has:

           !stuck-job.png|thumbnail!

          The previous build "finished" with these logs, but it's still showing the animated dots below that:



          {noformat}
          Errors were encountered
          Build step 'Execute shell' marked build as failure
          Sending e-mails to: xxx@xxx.com
          Notifying upstream projects of job completion

          {noformat}

          New: Jenkins 4.462.x was running fine for us since May. About two weeks ago we upgraded to 2.479.1 (and all plugins to latest compatible versions).

          A few days later, the Build Time Trend stopped loading properly (says "Computation in progress" forever). The following Saturday it stopped running jobs (parent kicked off downstream but didn't get woken up when downstream completed) and the restart timed out and it was killed by systemd.

          This week, the Build Time Trend has stopped loading properly again. I would not be surprised if it stops running jobs again this weekend. This is our production instance and I think we will have to downgrade very soon if it keeps doing this. But right now it's in a state where it's usable and potentially debuggable.

          I know we are using a lot of plugins and the problem could potentially be there, but I don't know in which one and can't really experiment with disabling random plugins on the production server. I suspect that we wouldn't see the issue on a test instance. Anything that could point to which plugin could be causing the problem would be most helpful.

          Looking at the thread dump, only one thing stands out: we have 10 threads running the logfilesizechecker plugin, all busy trying to check if something is a Gzip stream, which seems like far too many of these threads:

          {code:java}
          "jenkins.util.Timer [#10]" Id=67 Group=main RUNNABLE
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at java.base@17.0.13/sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:68)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:133)
          at java.base@17.0.13/sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:146)
          at java.base@17.0.13/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:216)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:380)
          at java.base@17.0.13/java.nio.file.Files.newByteChannel(Files.java:432)
          at java.base@17.0.13/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
          at java.base@17.0.13/java.nio.file.Files.newInputStream(Files.java:160)
          at org.kohsuke.stapler.framework.io.LargeText$GzipAwareSession.isGzipStream(LargeText.java:542)
          at org.kohsuke.stapler.framework.io.LargeText.<init>(LargeText.java:110)
          at hudson.console.AnnotatedLargeText.<init>(AnnotatedLargeText.java:88)
          at hudson.model.Run.getLogText(Run.java:1505)
          at PluginClassLoader for logfilesizechecker//hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:108)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base@17.0.13/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base@17.0.13/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
          at java.base@17.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.13/java.lang.Thread.run(Thread.java:840)

          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@89ed5b6
          {code}

          I also have a job stuck waiting for the previous build to complete, but it already has:

           !stuck-job.png|thumbnail!

          The previous build "finished" with these logs, but it's still showing the animated dots below that:

          {noformat}
          Errors were encountered
          Build step 'Execute shell' marked build as failure
          Sending e-mails to: xxx@xxx.com
          Notifying upstream projects of job completion
          {noformat}

          And the thread dump shows:

          "Executor #0 for lhr-vexec02-fast : executing nexus_db_replicate_roles #295805" Id=146231 Group=main WAITING on java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4c072eb0
          at java.base@17.0.13/jdk.internal.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4c072eb0
          at java.base@17.0.13/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
          at java.base@17.0.13/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:447)
          at java.base@17.0.13/java.util.concurrent.FutureTask.get(FutureTask.java:190)
          at hudson.tasks.BuildTrigger.execute(BuildTrigger.java:268)
          at hudson.model.AbstractBuild$AbstractBuildExecution.cleanUp(AbstractBuild.java:728)
          at hudson.model.Build$BuildExecution.cleanUp(Build.java:194)
          at hudson.model.Run.execute(Run.java:1874)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
          at hudson.model.ResourceController.execute(ResourceController.java:101)
          at hudson.model.Executor.run(Executor.java:445)

            Unassigned Unassigned
            qris Chris Wilson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: