Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64727

Jenkins 2.263.3 not triggering jobs or pipelines.

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • kubernetes-plugin
    • None

      We just upgraded our jenkins server to the `jenkins/jenkins:2.263.3-lts-jdk11` docker image.

      The upgrade went well and we upgraded to the latest compatible plugins.

      Unfortunately none of the jobs or pipelines are triggering, even on a schedule. If we click 'Build Now' a tooltip shows that it got triggered but nothing happens.

      If we click on a build with parameters then click Build we get redirected to `/build?delay=0sec` and the page get stuck.

      There is nothing in the logs showing the the jobs were triggered.

      This is our list of plugins:

        - ace-editor:1.1
        - ansicolor:0.7.4
        - ant:1.11
        - antisamy-markup-formatter:2.1
        - anything-goes-formatter:1.0
        - apache-httpcomponents-client-4-api:4.5.13-1.0
        - artifact-manager-s3:1.12
        - audit-trail:3.8
        - authentication-tokens:1.4
        - aws-credentials:1.28
        - aws-global-configuration:1.6
        - aws-java-sdk:1.11.930
        - basic-branch-build-strategies:1.3.2
        - blueocean:1.24.4
        - blueocean-autofavorite:1.2.4
        - blueocean-bitbucket-pipeline:1.24.4
        - blueocean-commons:1.24.4
        - blueocean-config:1.24.4
        - blueocean-core-js:1.24.4
        - blueocean-dashboard:1.24.4
        - blueocean-display-url:2.4.0
        - blueocean-events:1.24.4
        - blueocean-git-pipeline:1.24.4
        - blueocean-github-pipeline:1.24.4
        - blueocean-i18n:1.24.4
        - blueocean-jira:1.24.4
        - blueocean-jwt:1.24.4
        - blueocean-personalization:1.24.4
        - blueocean-pipeline-api-impl:1.24.4
        - blueocean-pipeline-editor:1.24.4
        - blueocean-pipeline-scm-api:1.24.4
        - blueocean-rest:1.24.4
        - blueocean-rest-impl:1.24.4
        - blueocean-web:1.24.4
        - bootstrap4-api:4.5.3-2
        - bouncycastle-api:2.18
        - branch-api:2.6.2
        - build-monitor-plugin:1.12+build.201809061734
        - build-time-blame:2.0.3
        - build-timeout:1.20
        - build-timestamp:1.0.3
        - build-user-vars-plugin:1.6
        - build-with-parameters:1.5
        - cctray-xml:1.0
        - checks-api:1.2.0
        - chucknorris:1.4
        - cloudbees-bitbucket-branch-source:2.9.7
        - cloudbees-folder:6.15
        - cobertura:1.16
        - code-coverage-api:1.2.0
        - command-launcher:1.5
        - conditional-buildstep:1.4.1
        - config-file-provider:3.7.0
        - configuration-as-code:1.46
        - configurationslicing:1.51
        - copyartifact:1.46
        - credentials:2.3.14
        - credentials-binding:1.24
        - datadog:2.8.0
        - display-url-api:2.3.4
        - docker-commons:1.17
        - docker-java-api:3.1.5.2
        - docker-workflow:1.25
        - durable-task:1.35
        - ec2:1.56
        - ec2-fleet:2.3.2
        - echarts-api:4.9.0-3
        - email-ext:2.81
        - embeddable-build-status:2.0.3
        - envinject:2.3.0
        - envinject-api:1.7
        - environment-script:1.2.5
        - extended-choice-parameter:0.82
        - extensible-choice-parameter:1.7.0
        - external-monitor-job:1.7
        - favorite:2.3.2
        - folder-auth:1.3
        - font-awesome-api:5.15.1-1
        - ghprb:1.42.1
        - git:4.5.2
        - git-client:3.6.0
        - git-parameter:0.9.13
        - git-server:1.9
        - github:1.32.0
        - github-api:1.122
        - github-branch-source:2.9.3
        - google-oauth-plugin:1.0.2
        - gradle:1.36
        - gravatar:2.1
        - groovy:2.3
        - handlebars:1.1.1
        - handy-uri-templates-2-api:2.1.8-1.0
        - hashicorp-vault-plugin:3.7.0
        - htmlpublisher:1.25
        - ivy:2.1
        - jackson2-api:2.12.1
        - javadoc:1.6
        - jaxb:2.3.0.1
        - jdk-tool:1.4
        - jenkins-design-language:1.24.4
        - jira:3.1.3
        - job-import-plugin:3.4
        - jobConfigHistory:2.26
        - jobcacher:1.0
        - jquery:1.12.4-1
        - jquery-detached:1.2.1
        - jquery3-api:3.5.1-2
        - jsch:0.1.55.2
        - junit:1.48
        - kubernetes:1.28.7
        - kubernetes-client-api:4.11.1
        - kubernetes-credentials:0.8.0
        - kubernetes-pipeline-steps:1.6
        - ldap:1.22
        - lockable-resources:2.10
        - mailer:1.32.1
        - mapdb-api:1.0.9.0
        - mask-passwords:2.13
        - matrix-auth:2.6.5
        - matrix-project:1.18
        - maven-plugin:3.8
        - mercurial:2.12
        - metrics:4.0.2.7
        - momentjs:1.1.1
        - monitoring:1.86.0
        - multi-branch-project-plugin:0.7
        - next-build-number:1.6
        - node-iterator-api:1.5.0
        - oauth-credentials:0.4
        - okhttp-api:3.14.9
        - pam-auth:1.6
        - parameterized-trigger:2.39
        - pipeline-build-step:2.13
        - pipeline-cloudwatch-logs:0.1
        - pipeline-github-lib:1.0
        - pipeline-graph-analysis:1.10
        - pipeline-input-step:2.12
        - pipeline-milestone-step:1.3.1
        - pipeline-model-api:1.7.2
        - pipeline-model-definition:1.7.2
        - pipeline-model-extensions:1.7.2
        - pipeline-rest-api:2.19
        - pipeline-stage-step:2.5
        - pipeline-stage-tags-metadata:1.7.2
        - pipeline-stage-view:2.19
        - pipeline-utility-steps:2.6.1
        - plain-credentials:1.7
        - plugin-util-api:1.6.1
        - popper-api:1.16.0-7
        - publish-to-bitbucket:0.4
        - pubsub-light:1.13
        - rebuild:1.31
        - resource-disposer:0.14
        - run-condition:1.5
        - saferestart:0.3
        - saml:1.1.7
        - scm-api:2.6.4
        - script-security:1.75
        - simple-theme-plugin:0.6
        - skip-notifications-trait:1.0.5
        - snakeyaml-api:1.27.0
        - sse-gateway:1.24
        - ssh-credentials:1.18.1
        - ssh-slaves:1.31.5
        - startup-trigger-plugin:2.9.3
        - structs:1.20
        - subversion:2.14.0
        - test-results-analyzer:0.3.5
        - timestamper:1.11.8
        - token-macro:2.13
        - trilead-api:1.0.13
        - variant:1.4
        - violation-comments-to-stash:1.126
        - windows-slaves:1.7
        - workflow-aggregator:2.6
        - workflow-api:2.41
        - workflow-basic-steps:2.23
        - workflow-cps:2.87
        - workflow-cps-global-lib:2.17
        - workflow-durable-task-step:2.37
        - workflow-job:2.40
        - workflow-multibranch:2.22
        - workflow-scm-step:2.11
        - workflow-step-api:2.23
        - workflow-support:3.7
        - ws-cleanup:0.38 

          [JENKINS-64727] Jenkins 2.263.3 not triggering jobs or pipelines.

          Stephane Odul added a comment -

          From the JavaMelody plugin we can see that the http requests are just stuck forever:

          Handling POST /job/chucknorris/build from 10.1.141.207 : Jetty (winstone)-690
          java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
          java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
          java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
          java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
          java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
          java.base@11.0.10/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)
          hudson.model.Queue.schedule2(Queue.java:595)
          hudson.model.Queue.schedule2(Queue.java:721)
          jenkins.model.ParameterizedJobMixIn.doBuild(ParameterizedJobMixIn.java:216)
          jenkins.model.ParameterizedJobMixIn$ParameterizedJob.doBuild(ParameterizedJobMixIn.java:407)
          java.base@11.0.10/java.lang.invoke.LambdaForm$DMH/0x0000000801622440.invokeInterface(LambdaForm$DMH)
          java.base@11.0.10/java.lang.invoke.LambdaForm$MH/0x00000008016a0040.invoke(LambdaForm$MH)
          java.base@11.0.10/java.lang.invoke.LambdaForm$MH/0x0000000801510840.invoke(LambdaForm$MH)
          java.base@11.0.10/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
          java.base@11.0.10/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
          org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
          org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
          org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
          org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
          org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536)
          org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
          org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
          org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
          org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
          org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
          org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
          org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
          org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
          org.kohsuke.stapler.Stapler.service(Stapler.java:240)
          javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
          org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1633)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
          org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
          jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
          hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:111)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
          jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
          io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
          io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
          com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
          net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
          net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
          net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
          org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
          jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
          hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
          org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
          org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
          hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153)
          org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
          org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
          hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
          org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
          org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
          org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
          org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
          org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
          hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
          org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
          org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
          org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
          org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
          jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
          org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
          org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609)
          org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561)
          org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
          org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
          org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
          org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
          org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
          org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
          org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
          org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
          org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
          org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
          org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
          org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
          org.eclipse.jetty.server.Server.handle(Server.java:516)
          org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
          org.eclipse.jetty.server.HttpChannel$$Lambda$477/0x00000008012f8440.dispatch(Unknown Source)
          org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
          org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
          org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
          org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
          org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
          org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
          org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
          org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
          org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
          org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
          org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
          org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
          org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
          java.base@11.0.10/java.lang.Thread.run(Thread.java:834) 

          Stephane Odul added a comment - From the JavaMelody plugin we can see that the http requests are just stuck forever: Handling POST /job/chucknorris/build from 10.1.141.207 : Jetty (winstone)-690 java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917) java.base@11.0.10/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240) java.base@11.0.10/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267) hudson.model.Queue.schedule2(Queue.java:595) hudson.model.Queue.schedule2(Queue.java:721) jenkins.model.ParameterizedJobMixIn.doBuild(ParameterizedJobMixIn.java:216) jenkins.model.ParameterizedJobMixIn$ParameterizedJob.doBuild(ParameterizedJobMixIn.java:407) java.base@11.0.10/java.lang.invoke.LambdaForm$DMH/0x0000000801622440.invokeInterface(LambdaForm$DMH) java.base@11.0.10/java.lang.invoke.LambdaForm$MH/0x00000008016a0040.invoke(LambdaForm$MH) java.base@11.0.10/java.lang.invoke.LambdaForm$MH/0x0000000801510840.invoke(LambdaForm$MH) java.base@11.0.10/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder) java.base@11.0.10/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710) org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766) org.kohsuke.stapler.Stapler.invoke(Stapler.java:898) org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766) org.kohsuke.stapler.Stapler.invoke(Stapler.java:898) org.kohsuke.stapler.Stapler.invoke(Stapler.java:694) org.kohsuke.stapler.Stapler.service(Stapler.java:240) javax.servlet.http.HttpServlet.service(HttpServlet.java:790) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1633) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:111) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153) org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51) org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36) org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1609) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:561) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612) org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582) org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) org.eclipse.jetty.server.Server.handle(Server.java:516) org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) org.eclipse.jetty.server.HttpChannel$$Lambda$477/0x00000008012f8440.dispatch(Unknown Source) org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773) org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905) java.base@11.0.10/java.lang.Thread.run(Thread.java:834)

          Stephane Odul added a comment - - edited

          We were able to reproduce the issue after upgrading an other instance to 2.277. Things went back to normal after downgrading to 2.249.3, plugins included.

          The plugins that worked with the old LTS are:

          - ace-editor:1.1
          - ansicolor:0.7.3
          - ant:1.11
          - antisamy-markup-formatter:2.1
          - anything-goes-formatter:1.0
          - apache-httpcomponents-client-4-api:4.5.10-2.0
          - artifact-manager-s3:1.12
          - audit-trail:3.7
          - authentication-tokens:1.4
          - aws-credentials:1.28
          - aws-global-configuration:1.6
          - aws-java-sdk:1.11.854
          - basic-branch-build-strategies:1.3.2
          - blueocean:1.24.3
          - blueocean-autofavorite:1.2.4
          - blueocean-bitbucket-pipeline:1.24.3
          - blueocean-commons:1.24.3
          - blueocean-config:1.24.3
          - blueocean-core-js:1.24.3
          - blueocean-dashboard:1.24.3
          - blueocean-display-url:2.4.0
          - blueocean-events:1.24.3
          - blueocean-executor-info:1.24.3
          - blueocean-git-pipeline:1.24.3
          - blueocean-github-pipeline:1.24.3
          - blueocean-i18n:1.24.3
          - blueocean-jira:1.24.3
          - blueocean-jwt:1.24.3
          - blueocean-personalization:1.24.3
          - blueocean-pipeline-api-impl:1.24.3
          - blueocean-pipeline-editor:1.24.3
          - blueocean-pipeline-scm-api:1.24.3
          - blueocean-rest:1.24.3
          - blueocean-rest-impl:1.24.3
          - blueocean-web:1.24.3
          - bootstrap4-api:4.5.3-1
          - bouncycastle-api:2.18
          - branch-api:2.6.1
          - build-monitor-plugin:1.12+build.201809061734
          - build-time-blame:2.0.3
          - build-timeout:1.20
          - build-timestamp:1.0.3
          - build-user-vars-plugin:1.6
          - build-with-parameters:1.5
          - cctray-xml:1.0
          - checks-api:1.1.0
          - chucknorris:1.3
          - cloudbees-bitbucket-branch-source:2.9.4
          - cloudbees-folder:6.14
          - cobertura:1.16
          - code-coverage-api:1.2.0
          - command-launcher:1.5
          - conditional-buildstep:1.3.6
          - config-file-provider:3.7.0
          - configuration-as-code:1.46
          - configurationslicing:1.51
          - copyartifact:1.45.2
          - credentials:2.3.13
          - credentials-binding:1.24
          - datadog:2.4.0
          - display-url-api:2.3.3
          - docker-commons:1.17
          - docker-java-api:3.1.5.2
          - docker-workflow:1.24
          - durable-task:1.35
          - ec2:1.54
          - ec2-fleet:2.2.0
          - echarts-api:4.9.0-2
          - email-ext:2.79
          - embeddable-build-status:2.0.3
          - envinject:2.3.0
          - envinject-api:1.7
          - environment-script:1.2.5
          - extended-choice-parameter:0.82
          - extensible-choice-parameter:1.7.0
          - external-monitor-job:1.7
          - favorite:2.3.2
          - folder-auth:1.3
          - font-awesome-api:5.15.1-1
          - ghprb:1.42.1
          - git:4.4.5
          - git-client:3.5.1
          - git-parameter:0.9.13
          - git-server:1.9
          - github:1.32.0
          - github-api:1.116
          - github-branch-source:2.9.1
          - google-oauth-plugin:1.0.2
          - gradle:1.36
          - gravatar:2.1
          - groovy:2.3
          - handlebars:1.1.1
          - handy-uri-templates-2-api:2.1.8-1.0
          - hashicorp-vault-plugin:3.6.1
          - htmlpublisher:1.23
          - ivy:2.1
          - jackson2-api:2.11.3
          - javadoc:1.6
          - jaxb:2.3.0.1
          - jdk-tool:1.4
          - jenkins-design-language:1.24.3
          - jira:3.1.3
          - job-import-plugin:3.4
          - jobConfigHistory:2.26
          - jobcacher:1.0
          - jquery:1.12.4-1
          - jquery-detached:1.2.1
          - jquery3-api:3.5.1-2
          - jsch:0.1.55.2
          - junit:1.43
          - kubernetes:1.27.5
          - kubernetes-client-api:4.11.1
          - kubernetes-credentials:0.7.0
          - kubernetes-pipeline-steps:1.6
          - ldap:1.26
          - lockable-resources:2.10
          - mailer:1.32.1
          - mapdb-api:1.0.9.0
          - mask-passwords:2.13
          - matrix-auth:2.6.4
          - matrix-project:1.18
          - maven-plugin:3.8
          - mercurial:2.12
          - momentjs:1.1.1
          - monitoring:1.86.0
          - multi-branch-project-plugin:0.7
          - multiple-scms:0.6
          - next-build-number:1.6
          - node-iterator-api:1.5.0
          - oauth-credentials:0.4
          - okhttp-api:3.14.9
          - pam-auth:1.6
          - parameterized-trigger:2.39
          - pipeline-build-step:2.13
          - pipeline-github-lib:1.0
          - pipeline-graph-analysis:1.10
          - pipeline-input-step:2.12
          - pipeline-milestone-step:1.3.1
          - pipeline-model-api:1.7.2
          - pipeline-model-declarative-agent:1.1.1
          - pipeline-model-definition:1.7.2
          - pipeline-model-extensions:1.7.2
          - pipeline-rest-api:2.18
          - pipeline-stage-step:2.5
          - pipeline-stage-tags-metadata:1.7.2
          - pipeline-stage-view:2.18
          - pipeline-utility-steps:2.6.1
          - plain-credentials:1.7
          - plugin-util-api:1.4.0
          - popper-api:1.16.0-7
          - publish-to-bitbucket:0.4
          - pubsub-light:1.13
          - rebuild:1.31
          - resource-disposer:0.14
          - run-condition:1.3
          - saferestart:0.3
          - saml:1.1.7
          - scm-api:2.6.4
          - script-security:1.75
          - simple-theme-plugin:0.6
          - skip-notifications-trait:1.0.5
          - snakeyaml-api:1.27.0
          - sse-gateway:1.24
          - ssh-credentials:1.18.1
          - ssh-slaves:1.31.2
          - startup-trigger-plugin:2.9.3
          - structs:1.20
          - subversion:2.13.2
          - test-results-analyzer:0.3.5
          - timestamper:1.11.8
          - token-macro:2.12
          - trilead-api:1.0.12
          - variant:1.3
          - violation-comments-to-stash:1.125
          - violations:0.7.11
          - windows-slaves:1.7
          - workflow-aggregator:2.6
          - workflow-api:2.40
          - workflow-basic-steps:2.22
          - workflow-cps:2.84
          - workflow-cps-global-lib:2.17
          - workflow-durable-task-step:2.36
          - workflow-job:2.40
          - workflow-multibranch:2.22
          - workflow-scm-step:2.11
          - workflow-step-api:2.23
          - workflow-support:3.6
          - ws-cleanup:0.38 

          Downgrading forced us to manually reconfigure our Kubernetes plugin and the Library (groovy library for pipelines) by hand.

          Stephane Odul added a comment - - edited We were able to reproduce the issue after upgrading an other instance to 2.277. Things went back to normal after downgrading to  2.249.3 , plugins included. The plugins that worked with the old LTS are: - ace-editor:1.1 - ansicolor:0.7.3 - ant:1.11 - antisamy-markup-formatter:2.1 - anything-goes-formatter:1.0 - apache-httpcomponents-client-4-api:4.5.10-2.0 - artifact-manager-s3:1.12 - audit-trail:3.7 - authentication-tokens:1.4 - aws-credentials:1.28 - aws-global-configuration:1.6 - aws-java-sdk:1.11.854 - basic-branch-build-strategies:1.3.2 - blueocean:1.24.3 - blueocean-autofavorite:1.2.4 - blueocean-bitbucket-pipeline:1.24.3 - blueocean-commons:1.24.3 - blueocean-config:1.24.3 - blueocean-core-js:1.24.3 - blueocean-dashboard:1.24.3 - blueocean-display-url:2.4.0 - blueocean-events:1.24.3 - blueocean-executor-info:1.24.3 - blueocean-git-pipeline:1.24.3 - blueocean-github-pipeline:1.24.3 - blueocean-i18n:1.24.3 - blueocean-jira:1.24.3 - blueocean-jwt:1.24.3 - blueocean-personalization:1.24.3 - blueocean-pipeline-api-impl:1.24.3 - blueocean-pipeline-editor:1.24.3 - blueocean-pipeline-scm-api:1.24.3 - blueocean-rest:1.24.3 - blueocean-rest-impl:1.24.3 - blueocean-web:1.24.3 - bootstrap4-api:4.5.3-1 - bouncycastle-api:2.18 - branch-api:2.6.1 - build-monitor-plugin:1.12+build.201809061734 - build-time-blame:2.0.3 - build-timeout:1.20 - build-timestamp:1.0.3 - build-user-vars-plugin:1.6 - build-with-parameters:1.5 - cctray-xml:1.0 - checks-api:1.1.0 - chucknorris:1.3 - cloudbees-bitbucket-branch-source:2.9.4 - cloudbees-folder:6.14 - cobertura:1.16 - code-coverage-api:1.2.0 - command-launcher:1.5 - conditional-buildstep:1.3.6 - config-file-provider:3.7.0 - configuration-as-code:1.46 - configurationslicing:1.51 - copyartifact:1.45.2 - credentials:2.3.13 - credentials-binding:1.24 - datadog:2.4.0 - display-url-api:2.3.3 - docker-commons:1.17 - docker-java-api:3.1.5.2 - docker-workflow:1.24 - durable-task:1.35 - ec2:1.54 - ec2-fleet:2.2.0 - echarts-api:4.9.0-2 - email-ext:2.79 - embeddable-build-status:2.0.3 - envinject:2.3.0 - envinject-api:1.7 - environment-script:1.2.5 - extended-choice-parameter:0.82 - extensible-choice-parameter:1.7.0 - external-monitor-job:1.7 - favorite:2.3.2 - folder-auth:1.3 - font-awesome-api:5.15.1-1 - ghprb:1.42.1 - git:4.4.5 - git-client:3.5.1 - git-parameter:0.9.13 - git-server:1.9 - github:1.32.0 - github-api:1.116 - github-branch-source:2.9.1 - google-oauth-plugin:1.0.2 - gradle:1.36 - gravatar:2.1 - groovy:2.3 - handlebars:1.1.1 - handy-uri-templates-2-api:2.1.8-1.0 - hashicorp-vault-plugin:3.6.1 - htmlpublisher:1.23 - ivy:2.1 - jackson2-api:2.11.3 - javadoc:1.6 - jaxb:2.3.0.1 - jdk-tool:1.4 - jenkins-design-language:1.24.3 - jira:3.1.3 - job-import-plugin:3.4 - jobConfigHistory:2.26 - jobcacher:1.0 - jquery:1.12.4-1 - jquery-detached:1.2.1 - jquery3-api:3.5.1-2 - jsch:0.1.55.2 - junit:1.43 - kubernetes:1.27.5 - kubernetes-client-api:4.11.1 - kubernetes-credentials:0.7.0 - kubernetes-pipeline-steps:1.6 - ldap:1.26 - lockable-resources:2.10 - mailer:1.32.1 - mapdb-api:1.0.9.0 - mask-passwords:2.13 - matrix-auth:2.6.4 - matrix-project:1.18 - maven-plugin:3.8 - mercurial:2.12 - momentjs:1.1.1 - monitoring:1.86.0 - multi-branch-project-plugin:0.7 - multiple-scms:0.6 - next-build-number:1.6 - node-iterator-api:1.5.0 - oauth-credentials:0.4 - okhttp-api:3.14.9 - pam-auth:1.6 - parameterized-trigger:2.39 - pipeline-build-step:2.13 - pipeline-github-lib:1.0 - pipeline-graph-analysis:1.10 - pipeline-input-step:2.12 - pipeline-milestone-step:1.3.1 - pipeline-model-api:1.7.2 - pipeline-model-declarative-agent:1.1.1 - pipeline-model-definition:1.7.2 - pipeline-model-extensions:1.7.2 - pipeline-rest-api:2.18 - pipeline-stage-step:2.5 - pipeline-stage-tags-metadata:1.7.2 - pipeline-stage-view:2.18 - pipeline-utility-steps:2.6.1 - plain-credentials:1.7 - plugin-util-api:1.4.0 - popper-api:1.16.0-7 - publish-to-bitbucket:0.4 - pubsub-light:1.13 - rebuild:1.31 - resource-disposer:0.14 - run-condition:1.3 - saferestart:0.3 - saml:1.1.7 - scm-api:2.6.4 - script-security:1.75 - simple-theme-plugin:0.6 - skip-notifications-trait:1.0.5 - snakeyaml-api:1.27.0 - sse-gateway:1.24 - ssh-credentials:1.18.1 - ssh-slaves:1.31.2 - startup-trigger-plugin:2.9.3 - structs:1.20 - subversion:2.13.2 - test-results-analyzer:0.3.5 - timestamper:1.11.8 - token-macro:2.12 - trilead-api:1.0.12 - variant:1.3 - violation-comments-to-stash:1.125 - violations:0.7.11 - windows-slaves:1.7 - workflow-aggregator:2.6 - workflow-api:2.40 - workflow-basic-steps:2.22 - workflow-cps:2.84 - workflow-cps-global-lib:2.17 - workflow-durable-task-step:2.36 - workflow-job:2.40 - workflow-multibranch:2.22 - workflow-scm-step:2.11 - workflow-step-api:2.23 - workflow-support:3.6 - ws-cleanup:0.38 Downgrading forced us to manually reconfigure our Kubernetes plugin and the Library (groovy library for pipelines) by hand.

          Stephane Odul added a comment -

          We re-upgraded a few plugins at a time and we linked the issue to upgrading from 

          kubernetes:1.27.5 to 1.27.7 and kubernetes-credentials:0.7 to 0.8.

          After upgrading them the new pipelines triggers are showing as stuck in JavaMelody, but rolling them back did not help somehow.

          So we deleted all plugins from disk, re-installed the previous known set (our helm charts take care of that). Upgraded everything exept the k8s and k8s credentials. All works fine. Install k8s credentials, works fine, install k8s plugin. works fine as well.

          We are to the point that we suspect there is some race condition in the installation of the plugins which makes the upgrade work or not under certain conditions.

          Stephane Odul added a comment - We re-upgraded a few plugins at a time and we linked the issue to upgrading from  kubernetes:1.27.5 to 1.27.7 and kubernetes-credentials:0.7 to 0.8. After upgrading them the new pipelines triggers are showing as stuck in JavaMelody, but rolling them back did not help somehow. So we deleted all plugins from disk, re-installed the previous known set (our helm charts take care of that). Upgraded everything exept the k8s and k8s credentials. All works fine. Install k8s credentials, works fine, install k8s plugin. works fine as well. We are to the point that we suspect there is some race condition in the installation of the plugins which makes the upgrade work or not under certain conditions.

          Stephane Odul added a comment -

          After upgrading to 2.263.4 and upgrading to the latest compatible plugins the issue was not reproduced. The jobs are now starting immediately.

          Stephane Odul added a comment - After upgrading to 2.263.4 and upgrading to the latest compatible plugins the issue was not reproduced. The jobs are now starting immediately.

            Unassigned Unassigned
            sodul Stephane Odul
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: