Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65581

"Configure system" UI broken, clicking "Apply" does nothing

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None
    • 2.289.2

      How to reproduce:

      1. Go to https://<jenkins_ui>/configure with any browser
      2. Make a configuration change of any sort
      3. Click on "Apply" or "Save".
      4. Using Chrome, nothing happens and the changes are not accepted. Using Firefox, a blank "Error" pop up appears.

          [JENKINS-65581] "Configure system" UI broken, clicking "Apply" does nothing

          Aviram created issue -
          Aviram made changes -
          Priority Original: Critical [ 2 ] New: Major [ 3 ]

          Mark Waite added a comment - - edited

          onallion this is likely due to the configuration form modernization that was released in Jenkins 2.277.1. Unfortunately, the information you provided is not enough for others to assist you.

          Steps you can take to help yourself and help others help you

          • Follow the instructions in the Jenkins 2.277.1 upgrade guide and the Jenkins 2.277.1 webinar (and its slides)
            • Remove unused plugins from your installation
            • Remove plugins with security issues from your installation (like the TFS plugin)
            • Remove deprecated plugins from your installation (like the DRY plugin, the checkstyle plugin, the warnings plugin, etc.)
            • Update plugins to the latest releases
          • Confirm that the issue is still visible
          • Provide the information requested in "How to report an issue", especially including the specific details :
            • Jenkins version
            • Jenkins plugin names and versions (please report with the format provided by the script)

          The format of plugin list that you provided does not match the format defined in the specific details . The format you've provided requires post-processing before it can be used to explore the issue. If you'd like help from others, please provide data in the requested format.

          Several items indicate that you have not yet followed the upgrade guide instructions:

          • Multiple SCMs plugin 0.6 is deprecated and should be removed. It is known to have issues with configuration form modernization
          • Blue Ocean 1.124.3 installed on your system is not the latest release. Install the latest release
          • Branch API plugin 2.5.8 is not the latest release and is known to have issues with configuration form modernization. Install the latest release
          • LDAP plugin 1.26 is not the latest release and is expected to have issues with the acegi to spring security transition. Install the latest release

          Mark Waite added a comment - - edited onallion this is likely due to the configuration form modernization that was released in Jenkins 2.277.1 . Unfortunately, the information you provided is not enough for others to assist you. Steps you can take to help yourself and help others help you Follow the instructions in the Jenkins 2.277.1 upgrade guide and the Jenkins 2.277.1 webinar (and its slides ) Remove unused plugins from your installation Remove plugins with security issues from your installation (like the TFS plugin) Remove deprecated plugins from your installation (like the DRY plugin, the checkstyle plugin, the warnings plugin, etc.) Update plugins to the latest releases Confirm that the issue is still visible Provide the information requested in "How to report an issue" , especially including the specific details : Jenkins version Jenkins plugin names and versions (please report with the format provided by the script ) The format of plugin list that you provided does not match the format defined in the specific details . The format you've provided requires post-processing before it can be used to explore the issue. If you'd like help from others, please provide data in the requested format. Several items indicate that you have not yet followed the upgrade guide instructions: Multiple SCMs plugin 0.6 is deprecated and should be removed. It is known to have issues with configuration form modernization Blue Ocean 1.124.3 installed on your system is not the latest release. Install the latest release Branch API plugin 2.5.8 is not the latest release and is known to have issues with configuration form modernization. Install the latest release LDAP plugin 1.26 is not the latest release and is expected to have issues with the acegi to spring security transition. Install the latest release

          R. Fitzner added a comment -

          After upgrading from Jenkins LTS vesion 2.263.4 to 2.277.3 (or 2.277.4 it doesn't matter) I have noticed that Firefox quits the session with "Secure Connection Failed" without an error code and Chrome and Edge quits with "ERR_EMPTY_RESPONSE" after pressing Save/Apply button of ANY (configure, configureSecurity, configSubmit) configuration via Jenkins GUI.

          The access.log reports a HTTP 500 error, like this:

          - - - - - - - - 1 - - [11/May/2021:16:46:09 +0200] POST /job/Master-Jobs/job/Application-Master-/configSubmit HTTP/1.1 500 2308 https://my.company.com/job/Master-Jobs/job/Application-Master-/configure Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36

          but ONLY if the time between pressing configure, do some small changes and pressing save button takes more than 10 seconds!
          If you can manage to press save in less than 10 seconds it works.

          So to see what causes the 500 error I added a custom logger and configured "org.eclipse.jetty=ALL".

          I found a lot of BUFFER_UNDERFLOW and googled for issues in winstone/jetty and found this https://github.com/eclipse/jetty.project/issues/6072

          So adding the parameter "-Djsse.SSLEngine.acceptLargeFragments=true" to Jenkins surprisingly solves the problem ONLY for IE and Firefox, but not for Chrome or Edge!

          So I continued investigating and I still found a lot of TimeoutException with 5000 ms.

          So adding the parameter "-–httpsKeepAliveTimeout=60000" (Pay attention to the character "s" in https because there is also the same parameter for non-TLS connections (httpKeepAliveTimeout)) now allows Chrome and Edge to extend the time window to 60 seconds between pressing configure, do some small changes and pressing save button.

          If I press the save button after 60 seconds, then I can still find Exceptions in the custom log, like this:

          2021-05-11 14:45:59.323+0000 [id=28] FINE o.eclipse.jetty.io.FillInterest#onFail: onFail FillInterest@e6a72a1\{SSLC.NBReadCB@4cee2a41{SslConnection@4cee2a41::SocketChannelEndPoint@63bc8525{l=/1.2.3.4:443,r=/5.6.7.8:54639,OPEN,fill=FI,flush=-,to=60001/60000}{io=1/1,kio=1,kro=1}->SslConnection@4cee2a41\{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint@4fb2b7d4\{l=/1.2.3.4:443,r=/5.6.7.8:54639,OPEN,fill=FI,flush=-,to=60002/60000}=>HttpConnection@3ba49506[p=HttpParser\{s=START,0 of -1},g=HttpGenerator@1638381f\{s=START}]=>HttpChannelOverHttp@1b341378\{s=HttpChannelState@48a09648{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=6,c=false/false,a=IDLE,uri=null,age=0}}}
java.util.concurrent.TimeoutException: Idle timeout expired: 60001/60000 ms
 at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
 at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
 
2021-05-11 14:46:06.300+0000 [id=1508] FINEST o.e.jetty.io.ManagedSelector#safeInterestOps: IGNORED EXCEPTION 
java.nio.channels.CancelledKeyException
 at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
 at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
 at org.eclipse.jetty.io.ManagedSelector.safeInterestOps(ManagedSelector.java:441)
 at org.eclipse.jetty.io.ChannelEndPoint.toEndPointString(ChannelEndPoint.java:436)
 at org.eclipse.jetty.io.AbstractEndPoint.toString(AbstractEndPoint.java:450)
 at java.lang.String.valueOf(String.java:2994)
 at java.lang.StringBuilder.append(StringBuilder.java:131)
 at org.eclipse.jetty.util.log.JavaUtilLog.format(JavaUtilLog.java:297)
 at org.eclipse.jetty.util.log.JavaUtilLog.debug(JavaUtilLog.java:238)
 at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:1101)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
 at java.lang.Thread.run(Thread.java:748)

          I also tried out other things that did not help at all, for example:

          1. Installing the plugin https://plugins.jenkins.io/strict-crumb-issuer/ and disabled "Check the session ID" and "Prevent BREACH attack". (BTW: This issue https://issues.jenkins.io/browse/JENKINS-63385 was misleading me and made it harder to find the actual root cause for the 500 errors, as mentioned before)
          2. Adding the parameter " -Dhttps.protocols=TLSv1.2"
          3. Adding the parameter "–httpKeepAliveTimeout=30000" (without "s"!)
          4. Added parameter "-Dorg.kohsuke.stapler.RequestImpl.ALLOWED_HTTP_VERBS_FOR_FORMS=GET,POST" or "-Dorg.kohsuke.stapler.RequestImpl.ALLOWED_HTTP_VERBS_FOR_FORMS=GET"
          5. Deactivated plugins like https://plugins.jenkins.io/audit-trail/ , https://plugins.jenkins.io/jobConfigHistory/ , https://plugins.jenkins.io/multiple-scms/ , https://plugins.jenkins.io/jacoco and https://plugins.jenkins.io/job-restrictions because they still generate "<table" tags and not "<div" (see https://www.jenkins.io/doc/developer/views/table-to-div-migration/ )

           

          For me this bug looks more generic concerning keeping the session alive and Firefox/IE are somehow handling it differently to Chrome/Edge.

          Do you have any idea what else I can check?

          R. Fitzner added a comment - After upgrading from Jenkins LTS vesion 2.263.4 to 2.277.3 (or 2.277.4 it doesn't matter) I have noticed that Firefox quits the session with "Secure Connection Failed" without an error code and Chrome and Edge quits with "ERR_EMPTY_RESPONSE" after pressing Save/Apply button of ANY (configure, configureSecurity, configSubmit) configuration via Jenkins GUI. The access.log reports a HTTP 500 error, like this: - - - - - - - - 1 - - [11/May/2021:16:46:09 +0200] POST /job/Master-Jobs/job/Application-Master-/configSubmit HTTP/1.1 500 2308 https: //my.company.com/job/Master-Jobs/job/Application-Master-/configure Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 but ONLY if the time between pressing configure, do some small changes and pressing save button takes more than 10 seconds! If you can manage to press save in less than 10 seconds it works. So to see what causes the 500 error I added a custom logger and configured "org.eclipse.jetty=ALL". I found a lot of BUFFER_UNDERFLOW and googled for issues in winstone/jetty and found this https://github.com/eclipse/jetty.project/issues/6072 So adding the parameter "-Djsse.SSLEngine.acceptLargeFragments=true" to Jenkins surprisingly solves the problem ONLY for IE and Firefox , but not for Chrome or Edge ! So I continued investigating and I still found a lot of TimeoutException with 5000 ms. So adding the parameter "-–httpsKeepAliveTimeout=60000" (Pay attention to the character "s" in https because there is also the same parameter for non-TLS connections (httpKeepAliveTimeout)) now allows Chrome and Edge to extend the time window to 60 seconds between pressing configure, do some small changes and pressing save button . If I press the save button after 60 seconds, then I can still find Exceptions in the custom log, like this: 2021-05-11 14:45:59.323+0000 [id=28] FINE o.eclipse.jetty.io.FillInterest#onFail: onFail FillInterest@e6a72a1\{SSLC.NBReadCB@4cee2a41{SslConnection@4cee2a41::SocketChannelEndPoint@63bc8525{l=/1.2.3.4:443,r=/5.6.7.8:54639,OPEN,fill=FI,flush=-,to=60001/60000}{io=1/1,kio=1,kro=1}->SslConnection@4cee2a41\{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=INTERESTED,flush=IDLE}~>DecryptedEndPoint@4fb2b7d4\{l=/1.2.3.4:443,r=/5.6.7.8:54639,OPEN,fill=FI,flush=-,to=60002/60000}=>HttpConnection@3ba49506[p=HttpParser\{s=START,0 of -1},g=HttpGenerator@1638381f\{s=START}]=>HttpChannelOverHttp@1b341378\{s=HttpChannelState@48a09648{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp= false se= false i= true al=0},r=6,c= false / false ,a=IDLE,uri= null ,age=0}}} java.util.concurrent.TimeoutException: Idle timeout expired: 60001/60000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171) at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) 2021-05-11 14:46:06.300+0000 [id=1508] FINEST o.e.jetty.io.ManagedSelector#safeInterestOps: IGNORED EXCEPTION java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.eclipse.jetty.io.ManagedSelector.safeInterestOps(ManagedSelector.java:441) at org.eclipse.jetty.io.ChannelEndPoint.toEndPointString(ChannelEndPoint.java:436) at org.eclipse.jetty.io.AbstractEndPoint.toString(AbstractEndPoint.java:450) at java.lang. String .valueOf( String .java:2994) at java.lang.StringBuilder.append(StringBuilder.java:131) at org.eclipse.jetty.util.log.JavaUtilLog.format(JavaUtilLog.java:297) at org.eclipse.jetty.util.log.JavaUtilLog.debug(JavaUtilLog.java:238) at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:1101) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) at java.lang. Thread .run( Thread .java:748) I also tried out other things that did not help at all, for example: Installing the plugin https://plugins.jenkins.io/strict-crumb-issuer/ and disabled "Check the session ID" and "Prevent BREACH attack". (BTW: This issue https://issues.jenkins.io/browse/JENKINS-63385 was misleading me and made it harder to find the actual root cause for the 500 errors, as mentioned before) Adding the parameter " -Dhttps.protocols=TLSv1.2" Adding the parameter "–httpKeepAliveTimeout=30000" (without "s"!) Added parameter "-Dorg.kohsuke.stapler.RequestImpl.ALLOWED_HTTP_VERBS_FOR_FORMS=GET,POST" or "-Dorg.kohsuke.stapler.RequestImpl.ALLOWED_HTTP_VERBS_FOR_FORMS=GET" Deactivated plugins like https://plugins.jenkins.io/audit-trail/ , https://plugins.jenkins.io/jobConfigHistory/ , https://plugins.jenkins.io/multiple-scms/ , https://plugins.jenkins.io/jacoco and https://plugins.jenkins.io/job-restrictions because they still generate "<table" tags and not "<div" (see https://www.jenkins.io/doc/developer/views/table-to-div-migration/ )   For me this bug looks more generic concerning keeping the session alive and Firefox/IE are somehow handling it differently to Chrome/Edge. Do you have any idea what else I can check?

          R. Fitzner added a comment -

          Oh, I forgot to post that this error also occurs:

          2021-05-11 14:46:09.829+0000 [id=1462]	FINE	o.e.j.i.s.SslConnection$DecryptedEndPoint#handleException: DecryptedEndPoint@63a46662{l=/1.2.3.4:443,r=/1.2.3.4:54649,OPEN,fill=-,flush=-,to=3434/60000} stored fill exception
javax.net.ssl.SSLHandshakeException: Encrypted buffer max length exceeded
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:735)
	at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:342)
	at org.eclipse.jetty.server.HttpConnection.fillAndParseForContent(HttpConnection.java:316)
	at org.eclipse.jetty.server.HttpInputOverHTTP.produceContent(HttpInputOverHTTP.java:33)
	at org.eclipse.jetty.server.HttpInput.nextContent(HttpInput.java:382)
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:316)
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:270)
	at org.eclipse.jetty.util.UrlEncoded.decodeUtf8To(UrlEncoded.java:485)
	at org.eclipse.jetty.util.UrlEncoded.decodeTo(UrlEncoded.java:577)
	at org.eclipse.jetty.server.Request.extractFormParameters(Request.java:582)
	at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:531)
	at org.eclipse.jetty.server.Request.getParameters(Request.java:435)
	at org.eclipse.jetty.server.Request.getParameterNames(Request.java:1093)
	at javax.servlet.ServletRequestWrapper.getParameterNames(ServletRequestWrapper.java:212)
	at hudson.security.csrf.CrumbFilter.extractCrumbFromRequest(CrumbFilter.java:166)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:135)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
	at java.lang.Thread.run(Thread.java:748)

          Maybe it is related to this issue https://github.com/eclipse/jetty.project/issues/6082
          It is part of https://github.com/jenkinsci/winstone/releases/tag/winstone-5.17 now. We should check this out soon.

          R. Fitzner added a comment - Oh, I forgot to post that this error also occurs: 2021-05-11 14:46:09.829+0000 [id=1462] FINE o.e.j.i.s.SslConnection$DecryptedEndPoint#handleException: DecryptedEndPoint@63a46662{l=/1.2.3.4:443,r=/1.2.3.4:54649,OPEN,fill=-,flush=-,to=3434/60000} stored fill exception javax.net.ssl.SSLHandshakeException: Encrypted buffer max length exceeded at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:735) at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:342) at org.eclipse.jetty.server.HttpConnection.fillAndParseForContent(HttpConnection.java:316) at org.eclipse.jetty.server.HttpInputOverHTTP.produceContent(HttpInputOverHTTP.java:33) at org.eclipse.jetty.server.HttpInput.nextContent(HttpInput.java:382) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:316) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:270) at org.eclipse.jetty.util.UrlEncoded.decodeUtf8To(UrlEncoded.java:485) at org.eclipse.jetty.util.UrlEncoded.decodeTo(UrlEncoded.java:577) at org.eclipse.jetty.server.Request.extractFormParameters(Request.java:582) at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:531) at org.eclipse.jetty.server.Request.getParameters(Request.java:435) at org.eclipse.jetty.server.Request.getParameterNames(Request.java:1093) at javax.servlet.ServletRequestWrapper.getParameterNames(ServletRequestWrapper.java:212) at hudson.security.csrf.CrumbFilter.extractCrumbFromRequest(CrumbFilter.java:166) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:135) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) at java.lang. Thread .run( Thread .java:748) Maybe it is related to this issue https://github.com/eclipse/jetty.project/issues/6082 It is part of https://github.com/jenkinsci/winstone/releases/tag/winstone-5.17 now. We should check this out soon.

          Evan Bluhm added a comment -

          rf We've been seeing the same issue with both 2.277.3 and 2.277.4, but only on Jenkins instances where we're using Jetty for SSL termination. In deployments where we run Jenkins with TLS disabled behind Nginx (like in Kubernetes), we don't see the issue at all, so it seems extremely likely that https://github.com/eclipse/jetty.project/issues/6082 is the cause.

           

          It looks like those Jetty fixes have already been pulled into the weekly Jenkins core release as of 2.290 (https://github.com/jenkinsci/jenkins/pull/5439), but unfortunately it looks like it didn't make the list of changes backported to 2.277.4 (https://github.com/jenkinsci/jenkins/pull/5441), or even 2.289.1 (https://github.com/jenkinsci/jenkins/pull/5470).  I've tested out upgrading to the latest weekly release (2.293) which does have the fix for https://github.com/eclipse/jetty.project/issues/6082 and I haven't been able to reproduce the issue there. What's the process by which changes are selected for backporting to previous LTS versions? This seems like a pretty good candidate, since the bug here https://github.com/eclipse/jetty.project/issues/6082 was introduced with the changes that were ported to 2.277.3.

          Evan Bluhm added a comment - rf We've been seeing the same issue with both 2.277.3 and 2.277.4, but only on Jenkins instances where we're using Jetty for SSL termination. In deployments where we run Jenkins with TLS disabled behind Nginx (like in Kubernetes), we don't see the issue at all, so it seems extremely likely that  https://github.com/eclipse/jetty.project/issues/6082  is the cause.   It looks like those Jetty fixes have already been pulled into the weekly Jenkins core release as of 2.290 ( https://github.com/jenkinsci/jenkins/pull/5439),  but unfortunately it looks like it didn't make the list of changes backported to 2.277.4 ( https://github.com/jenkinsci/jenkins/pull/5441 ), or even 2.289.1 ( https://github.com/jenkinsci/jenkins/pull/5470).   I've tested out upgrading to the latest weekly release (2.293) which does have the fix for  https://github.com/eclipse/jetty.project/issues/6082 and I haven't been able to reproduce the issue there. What's the process by which changes are selected for backporting to previous LTS versions? This seems like a pretty good candidate, since the bug here  https://github.com/eclipse/jetty.project/issues/6082  was introduced with the changes that were ported to 2.277.3.

          Stefan Spieker added a comment -

          I can confirm, that we have the same issue with Jenkins 2.277.4 and SSL termination with Jetty. Downgrading to 2.277.2 solves the issue.

          Stefan Spieker added a comment - I can confirm, that we have the same issue with Jenkins 2.277.4 and SSL termination with Jetty. Downgrading to 2.277.2 solves the issue.

          Aviram added a comment -

          Hi markewaite,

          We have done as you said and updated all plugins which have configuration in this screen, and removed deprecated plugins. Unfortunately the issue still persists. 

          Here is our plugin list in the format you prefer:

          Jenkins: 2.277.2
          OS: Linux - 4.14.214-118.339.amzn1.x86_64
          —
          show-build-parameters:1.0
          xcode-plugin:2.0.15
          jquery-detached:1.2.1
          gradle:1.36
          authorize-project:1.3.0
          cloudbees-folder:6.15
          persistent-parameter:1.2
          atlassian-bitbucket-server-integration:2.1.3
          ws-cleanup:0.38
          durable-task:1.34
          plain-credentials:1.7
          pipeline-stage-view:2.13
          blueocean-commons:1.24.6
          ec2-fleet:2.1.1
          rebuild:1.32
          blueocean-pipeline-api-impl:1.24.6
          extended-choice-parameter:0.82
          dtkit-api:3.0.0
          build-notifications:1.5.0
          cobertura:1.16
          nodelabelparameter:1.7.2
          htmlpublisher:1.25
          embeddable-build-status:2.0.3
          jobConfigHistory:2.27
          handy-uri-templates-2-api:2.1.8-1.0
          pipeline-utility-steps:2.6.1
          github:1.33.1
          docker-slaves:1.0.7
          sonar:2.13.1
          stashNotifier:1.20
          ant:1.11
          bouncycastle-api:2.18
          nodejs:1.3.7
          maven-plugin:3.10
          blueocean-dashboard:1.24.6
          workflow-job:2.40
          config-file-provider:3.8.0
          blueocean-i18n:1.24.6
          job-dsl:1.77
          pipeline-model-api:1.8.4
          workflow-step-api:2.23
          timestamper:1.11.5
          blueocean-bitbucket-pipeline:1.24.6
          publish-over-ssh:1.22
          mapdb-api:1.0.9.0
          code-coverage-api:1.1.6
          workflow-aggregator:2.6
          ldap:1.26
          blueocean-jira:1.24.6
          aws-bucket-credentials:1.0.0
          docker-commons:1.17
          authentication-tokens:1.4
          blueocean-github-pipeline:1.24.6
          file-operations:1.11
          configuration-as-code:1.51
          pipeline-model-extensions:1.8.4
          blueocean-display-url:2.4.1
          okhttp-api:3.14.9
          pipeline-stage-tags-metadata:1.8.4
          plugin-util-api:2.2.0
          blueocean-config:1.24.6
          build-monitor-plugin:1.12+build.201809061734
          echarts-api:5.1.0-2
          workflow-multibranch:2.22
          windows-exe-runner:1.2
          throttle-concurrents:2.2
          cloudbees-bitbucket-branch-source:2.9.8
          aws-java-sdk:1.11.821
          blueocean-rest:1.24.6
          apache-httpcomponents-client-4-api:4.5.13-1.0
          copyartifact:1.46
          ssh-credentials:1.18.1
          matrix-project:1.18
          global-slack-notifier:1.5
          ssh-slaves:1.31.5
          remote-file:1.13
          bootstrap4-api:4.6.0-3
          extensible-choice-parameter:1.7.0
          blueocean-web:1.24.6
          run-condition:1.3
          powershell:1.4
          variant:1.3
          blueocean-pipeline-editor:1.24.6
          lockable-resources:2.10
          workflow-support:3.8
          branch-api:2.6.4
          display-url-api:2.3.4
          conditional-buildstep:1.3.6
          amazon-ecr:1.6
          envinject-api:1.7
          popper-api:1.16.1-2
          pam-auth:1.6
          momentjs:1.1.1
          bitbucket:1.1.28
          blueocean-personalization:1.24.6
          ace-editor:1.1
          saml:1.1.6
          jquery:1.12.4-1
          multibranch-build-strategy-extension:1.0.10
          workflow-durable-task-step:2.35
          workflow-cps-global-lib:2.19
          publish-over:0.22
          subversion:2.14.1
          msbuild:1.29
          pipeline-stage-step:2.5
          extended-read-permission:3.2
          allure-jenkins-plugin:2.29.0
          favorite:2.3.2
          token-macro:2.15
          docker-workflow:1.26
          dashboard-view:2.16
          blueocean-jwt:1.24.6
          external-monitor-job:1.7
          git-client:3.7.1
          blueocean-events:1.24.6
          git-server:1.9
          xunit:3.0.2
          ivy:2.1
          blueocean-rest-impl:1.24.6
          jsch:0.1.55.2
          build-failure-analyzer:2.0.0
          checks-api:1.7.0
          multibranch-scan-webhook-trigger:1.0.5
          http_request:1.8.26
          blueocean-pipeline-scm-api:1.24.6
          javadoc:1.6
          structs:1.22
          artifactory:3.11.0
          caffeine-api:2.9.1-23.v51c4e2c879c8
          pipeline-rest-api:2.13
          blueocean-autofavorite:1.2.4
          git:4.7.1
          matrix-auth:2.6.7
          node-iterator-api:1.5.0
          build-timeout:1.20
          mercurial:2.15
          handlebars:1.1.1
          antisamy-markup-formatter:2.1
          convert-to-pipeline:1.0
          jdk-tool:1.4
          workflow-scm-step:2.12
          content-replace:1.5.1
          pipeline-milestone-step:1.3.1
          github-branch-source:2.10.2
          snakeyaml-api:1.27.0
          pipeline-build-step:2.13
          pipeline-input-step:2.12
          trilead-api:1.0.13
          font-awesome-api:5.15.3-2
          role-strategy:3.1.1
          git-parameter:0.9.13
          basic-branch-build-strategies:1.3.2
          workflow-api:2.42
          jackson2-api:2.12.3
          test-results-analyzer:0.3.5
          terraform:1.0.10
          emailext-template:1.2
          build-name-setter:2.1.0
          blueocean-git-pipeline:1.24.6
          build-environment:1.7
          scalable-amazon-ecs:1.0
          extra-columns:1.23
          uno-choice:2.5.6
          github-api:1.123
          ssh-agent:1.22
          sse-gateway:1.24
          ansicolor:1.0.0
          script-security:1.76
          jquery3-api:3.6.0-1
          jenkins-design-language:1.24.6
          mailer:1.32.1
          pipeline-github-lib:1.0
          ec2:1.51
          scm-api:2.6.4
          pipeline-model-definition:1.8.4
          junit:1.49
          text-finder:1.15
          email-ext:2.82
          aws-credentials:1.29
          hashicorp-vault-plugin:3.8.0
          s3:0.11.7
          monitor-pro:1.0.9
          jira:3.3
          pipeline-graph-analysis:1.10
          jjwt-api:0.11.2-9.c8b45b8bb173
          workflow-basic-steps:2.20
          envinject:2.4.0
          windows-slaves:1.6
          pubsub-light:1.13
          credentials-binding:1.24
          resource-disposer:0.14
          slack:2.48
          pipeline-npm:0.9.2
          blueocean:1.24.6
          command-launcher:1.4
          active-directory:2.24
          ssh:2.6.1
          parameterized-trigger:2.40
          credentials:2.4.1
          amazon-ecs:1.36
          workflow-cps:2.92
          progress-bar-column-plugin:1.0
          blueocean-core-js:1.24.6
          nunit:0.27
          pipeline-aws:1.42
          Result: [Plugin:show-build-parameters, Plugin:xcode-plugin, Plugin:jquery-detached, Plugin:gradle, Plugin:authorize-project, Plugin:cloudbees-folder, Plugin:persistent-parameter, Plugin:atlassian-bitbucket-server-integration, Plugin:ws-cleanup, Plugin:durable-task, Plugin:plain-credentials, Plugin:pipeline-stage-view, Plugin:blueocean-commons, Plugin:ec2-fleet, Plugin:rebuild, Plugin:blueocean-pipeline-api-impl, Plugin:extended-choice-parameter, Plugin:dtkit-api, Plugin:build-notifications, Plugin:cobertura, Plugin:nodelabelparameter, Plugin:htmlpublisher, Plugin:embeddable-build-status, Plugin:jobConfigHistory, Plugin:handy-uri-templates-2-api, Plugin:pipeline-utility-steps, Plugin:github, Plugin:docker-slaves, Plugin:sonar, Plugin:stashNotifier, Plugin:ant, Plugin:bouncycastle-api, Plugin:nodejs, Plugin:maven-plugin, Plugin:blueocean-dashboard, Plugin:workflow-job, Plugin:config-file-provider, Plugin:blueocean-i18n, Plugin:job-dsl, Plugin:pipeline-model-api, Plugin:workflow-step-api, Plugin:timestamper, Plugin:blueocean-bitbucket-pipeline, Plugin:publish-over-ssh, Plugin:mapdb-api, Plugin:code-coverage-api, Plugin:workflow-aggregator, Plugin:ldap, Plugin:blueocean-jira, Plugin:aws-bucket-credentials, Plugin:docker-commons, Plugin:authentication-tokens, Plugin:blueocean-github-pipeline, Plugin:file-operations, Plugin:configuration-as-code, Plugin:pipeline-model-extensions, Plugin:blueocean-display-url, Plugin:okhttp-api, Plugin:pipeline-stage-tags-metadata, Plugin:plugin-util-api, Plugin:blueocean-config, Plugin:build-monitor-plugin, Plugin:echarts-api, Plugin:workflow-multibranch, Plugin:windows-exe-runner, Plugin:throttle-concurrents, Plugin:cloudbees-bitbucket-branch-source, Plugin:aws-java-sdk, Plugin:blueocean-rest, Plugin:apache-httpcomponents-client-4-api, Plugin:copyartifact, Plugin:ssh-credentials, Plugin:matrix-project, Plugin:global-slack-notifier, Plugin:ssh-slaves, Plugin:remote-file, Plugin:bootstrap4-api, Plugin:extensible-choice-parameter, Plugin:blueocean-web, Plugin:run-condition, Plugin:powershell, Plugin:variant, Plugin:blueocean-pipeline-editor, Plugin:lockable-resources, Plugin:workflow-support, Plugin:branch-api, Plugin:display-url-api, Plugin:conditional-buildstep, Plugin:amazon-ecr, Plugin:envinject-api, Plugin:popper-api, Plugin:pam-auth, Plugin:momentjs, Plugin:bitbucket, Plugin:blueocean-personalization, Plugin:ace-editor, Plugin:saml, Plugin:jquery, Plugin:multibranch-build-strategy-extension, Plugin:workflow-durable-task-step, Plugin:workflow-cps-global-lib, Plugin:publish-over, Plugin:subversion, Plugin:msbuild, Plugin:pipeline-stage-step, Plugin:extended-read-permission, Plugin:allure-jenkins-plugin, Plugin:favorite, Plugin:token-macro, Plugin:docker-workflow, Plugin:dashboard-view, Plugin:blueocean-jwt, Plugin:external-monitor-job, Plugin:git-client, Plugin:blueocean-events, Plugin:git-server, Plugin:xunit, Plugin:ivy, Plugin:blueocean-rest-impl, Plugin:jsch, Plugin:build-failure-analyzer, Plugin:checks-api, Plugin:multibranch-scan-webhook-trigger, Plugin:http_request, Plugin:blueocean-pipeline-scm-api, Plugin:javadoc, Plugin:structs, Plugin:artifactory, Plugin:caffeine-api, Plugin:pipeline-rest-api, Plugin:blueocean-autofavorite, Plugin:git, Plugin:matrix-auth, Plugin:node-iterator-api, Plugin:build-timeout, Plugin:mercurial, Plugin:handlebars, Plugin:antisamy-markup-formatter, Plugin:convert-to-pipeline, Plugin:jdk-tool, Plugin:workflow-scm-step, Plugin:content-replace, Plugin:pipeline-milestone-step, Plugin:github-branch-source, Plugin:snakeyaml-api, Plugin:pipeline-build-step, Plugin:pipeline-input-step, Plugin:trilead-api, Plugin:font-awesome-api, Plugin:role-strategy, Plugin:git-parameter, Plugin:basic-branch-build-strategies, Plugin:workflow-api, Plugin:jackson2-api, Plugin:test-results-analyzer, Plugin:terraform, Plugin:emailext-template, Plugin:build-name-setter, Plugin:blueocean-git-pipeline, Plugin:build-environment, Plugin:scalable-amazon-ecs, Plugin:extra-columns, Plugin:uno-choice, Plugin:github-api, Plugin:ssh-agent, Plugin:sse-gateway, Plugin:ansicolor, Plugin:script-security, Plugin:jquery3-api, Plugin:jenkins-design-language, Plugin:mailer, Plugin:pipeline-github-lib, Plugin:ec2, Plugin:scm-api, Plugin:pipeline-model-definition, Plugin:junit, Plugin:text-finder, Plugin:email-ext, Plugin:aws-credentials, Plugin:hashicorp-vault-plugin, Plugin:s3, Plugin:monitor-pro, Plugin:jira, Plugin:pipeline-graph-analysis, Plugin:jjwt-api, Plugin:workflow-basic-steps, Plugin:envinject, Plugin:windows-slaves, Plugin:pubsub-light, Plugin:credentials-binding, Plugin:resource-disposer, Plugin:slack, Plugin:pipeline-npm, Plugin:blueocean, Plugin:command-launcher, Plugin:active-directory, Plugin:ssh, Plugin:parameterized-trigger, Plugin:credentials, Plugin:amazon-ecs, Plugin:workflow-cps, Plugin:progress-bar-column-plugin, Plugin:blueocean-core-js, Plugin:nunit, Plugin:pipeline-aws]

           

          Any help would be very appreciated

           

          Thanks

          Aviram added a comment - Hi markewaite , We have done as you said and updated all plugins which have configuration in this screen, and removed deprecated plugins. Unfortunately the issue still persists.  Here is our plugin list in the format you prefer: Jenkins: 2.277.2 OS: Linux - 4.14.214-118.339.amzn1.x86_64 — show-build-parameters:1.0 xcode-plugin:2.0.15 jquery-detached:1.2.1 gradle:1.36 authorize-project:1.3.0 cloudbees-folder:6.15 persistent-parameter:1.2 atlassian-bitbucket-server-integration:2.1.3 ws-cleanup:0.38 durable-task:1.34 plain-credentials:1.7 pipeline-stage-view:2.13 blueocean-commons:1.24.6 ec2-fleet:2.1.1 rebuild:1.32 blueocean-pipeline-api-impl:1.24.6 extended-choice-parameter:0.82 dtkit-api:3.0.0 build-notifications:1.5.0 cobertura:1.16 nodelabelparameter:1.7.2 htmlpublisher:1.25 embeddable-build-status:2.0.3 jobConfigHistory:2.27 handy-uri-templates-2-api:2.1.8-1.0 pipeline-utility-steps:2.6.1 github:1.33.1 docker-slaves:1.0.7 sonar:2.13.1 stashNotifier:1.20 ant:1.11 bouncycastle-api:2.18 nodejs:1.3.7 maven-plugin:3.10 blueocean-dashboard:1.24.6 workflow-job:2.40 config-file-provider:3.8.0 blueocean-i18n:1.24.6 job-dsl:1.77 pipeline-model-api:1.8.4 workflow-step-api:2.23 timestamper:1.11.5 blueocean-bitbucket-pipeline:1.24.6 publish-over-ssh:1.22 mapdb-api:1.0.9.0 code-coverage-api:1.1.6 workflow-aggregator:2.6 ldap:1.26 blueocean-jira:1.24.6 aws-bucket-credentials:1.0.0 docker-commons:1.17 authentication-tokens:1.4 blueocean-github-pipeline:1.24.6 file-operations:1.11 configuration-as-code:1.51 pipeline-model-extensions:1.8.4 blueocean-display-url:2.4.1 okhttp-api:3.14.9 pipeline-stage-tags-metadata:1.8.4 plugin-util-api:2.2.0 blueocean-config:1.24.6 build-monitor-plugin:1.12+build.201809061734 echarts-api:5.1.0-2 workflow-multibranch:2.22 windows-exe-runner:1.2 throttle-concurrents:2.2 cloudbees-bitbucket-branch-source:2.9.8 aws-java-sdk:1.11.821 blueocean-rest:1.24.6 apache-httpcomponents-client-4-api:4.5.13-1.0 copyartifact:1.46 ssh-credentials:1.18.1 matrix-project:1.18 global-slack-notifier:1.5 ssh-slaves:1.31.5 remote- file:1.13 bootstrap4-api:4.6.0-3 extensible-choice-parameter:1.7.0 blueocean-web:1.24.6 run-condition:1.3 powershell:1.4 variant:1.3 blueocean-pipeline-editor:1.24.6 lockable-resources:2.10 workflow-support:3.8 branch-api:2.6.4 display-url-api:2.3.4 conditional-buildstep:1.3.6 amazon-ecr:1.6 envinject-api:1.7 popper-api:1.16.1-2 pam-auth:1.6 momentjs:1.1.1 bitbucket:1.1.28 blueocean-personalization:1.24.6 ace-editor:1.1 saml:1.1.6 jquery:1.12.4-1 multibranch-build-strategy-extension:1.0.10 workflow-durable-task-step:2.35 workflow-cps-global-lib:2.19 publish-over:0.22 subversion:2.14.1 msbuild:1.29 pipeline-stage-step:2.5 extended-read-permission:3.2 allure-jenkins-plugin:2.29.0 favorite:2.3.2 token-macro:2.15 docker-workflow:1.26 dashboard-view:2.16 blueocean-jwt:1.24.6 external-monitor-job:1.7 git-client:3.7.1 blueocean-events:1.24.6 git-server:1.9 xunit:3.0.2 ivy:2.1 blueocean-rest-impl:1.24.6 jsch:0.1.55.2 build-failure-analyzer:2.0.0 checks-api:1.7.0 multibranch-scan-webhook-trigger:1.0.5 http_request:1.8.26 blueocean-pipeline-scm-api:1.24.6 javadoc:1.6 structs:1.22 artifactory:3.11.0 caffeine-api:2.9.1-23.v51c4e2c879c8 pipeline-rest-api:2.13 blueocean-autofavorite:1.2.4 git:4.7.1 matrix-auth:2.6.7 node-iterator-api:1.5.0 build-timeout:1.20 mercurial:2.15 handlebars:1.1.1 antisamy-markup-formatter:2.1 convert-to-pipeline:1.0 jdk-tool:1.4 workflow-scm-step:2.12 content-replace:1.5.1 pipeline-milestone-step:1.3.1 github-branch-source:2.10.2 snakeyaml-api:1.27.0 pipeline-build-step:2.13 pipeline-input-step:2.12 trilead-api:1.0.13 font-awesome-api:5.15.3-2 role-strategy:3.1.1 git-parameter:0.9.13 basic-branch-build-strategies:1.3.2 workflow-api:2.42 jackson2-api:2.12.3 test-results-analyzer:0.3.5 terraform:1.0.10 emailext-template:1.2 build-name-setter:2.1.0 blueocean-git-pipeline:1.24.6 build-environment:1.7 scalable-amazon-ecs:1.0 extra-columns:1.23 uno-choice:2.5.6 github-api:1.123 ssh-agent:1.22 sse-gateway:1.24 ansicolor:1.0.0 script-security:1.76 jquery3-api:3.6.0-1 jenkins-design-language:1.24.6 mailer:1.32.1 pipeline-github-lib:1.0 ec2:1.51 scm-api:2.6.4 pipeline-model-definition:1.8.4 junit:1.49 text-finder:1.15 email-ext:2.82 aws-credentials:1.29 hashicorp-vault-plugin:3.8.0 s3:0.11.7 monitor-pro:1.0.9 jira:3.3 pipeline-graph-analysis:1.10 jjwt-api:0.11.2-9.c8b45b8bb173 workflow-basic-steps:2.20 envinject:2.4.0 windows-slaves:1.6 pubsub-light:1.13 credentials-binding:1.24 resource-disposer:0.14 slack:2.48 pipeline-npm:0.9.2 blueocean:1.24.6 command-launcher:1.4 active-directory:2.24 ssh:2.6.1 parameterized-trigger:2.40 credentials:2.4.1 amazon-ecs:1.36 workflow-cps:2.92 progress-bar-column-plugin:1.0 blueocean-core-js:1.24.6 nunit:0.27 pipeline-aws:1.42 Result:  [Plugin:show-build-parameters, Plugin:xcode-plugin, Plugin:jquery-detached, Plugin:gradle, Plugin:authorize-project, Plugin:cloudbees-folder, Plugin:persistent-parameter, Plugin:atlassian-bitbucket-server-integration, Plugin:ws-cleanup, Plugin:durable-task, Plugin:plain-credentials, Plugin:pipeline-stage-view, Plugin:blueocean-commons, Plugin:ec2-fleet, Plugin:rebuild, Plugin:blueocean-pipeline-api-impl, Plugin:extended-choice-parameter, Plugin:dtkit-api, Plugin:build-notifications, Plugin:cobertura, Plugin:nodelabelparameter, Plugin:htmlpublisher, Plugin:embeddable-build-status, Plugin:jobConfigHistory, Plugin:handy-uri-templates-2-api, Plugin:pipeline-utility-steps, Plugin:github, Plugin:docker-slaves, Plugin:sonar, Plugin:stashNotifier, Plugin:ant, Plugin:bouncycastle-api, Plugin:nodejs, Plugin:maven-plugin, Plugin:blueocean-dashboard, Plugin:workflow-job, Plugin:config-file-provider, Plugin:blueocean-i18n, Plugin:job-dsl, Plugin:pipeline-model-api, Plugin:workflow-step-api, Plugin:timestamper, Plugin:blueocean-bitbucket-pipeline, Plugin:publish-over-ssh, Plugin:mapdb-api, Plugin:code-coverage-api, Plugin:workflow-aggregator, Plugin:ldap, Plugin:blueocean-jira, Plugin:aws-bucket-credentials, Plugin:docker-commons, Plugin:authentication-tokens, Plugin:blueocean-github-pipeline, Plugin:file-operations, Plugin:configuration-as-code, Plugin:pipeline-model-extensions, Plugin:blueocean-display-url, Plugin:okhttp-api, Plugin:pipeline-stage-tags-metadata, Plugin:plugin-util-api, Plugin:blueocean-config, Plugin:build-monitor-plugin, Plugin:echarts-api, Plugin:workflow-multibranch, Plugin:windows-exe-runner, Plugin:throttle-concurrents, Plugin:cloudbees-bitbucket-branch-source, Plugin:aws-java-sdk, Plugin:blueocean-rest, Plugin:apache-httpcomponents-client-4-api, Plugin:copyartifact, Plugin:ssh-credentials, Plugin:matrix-project, Plugin:global-slack-notifier, Plugin:ssh-slaves, Plugin:remote-file, Plugin:bootstrap4-api, Plugin:extensible-choice-parameter, Plugin:blueocean-web, Plugin:run-condition, Plugin:powershell, Plugin:variant, Plugin:blueocean-pipeline-editor, Plugin:lockable-resources, Plugin:workflow-support, Plugin:branch-api, Plugin:display-url-api, Plugin:conditional-buildstep, Plugin:amazon-ecr, Plugin:envinject-api, Plugin:popper-api, Plugin:pam-auth, Plugin:momentjs, Plugin:bitbucket, Plugin:blueocean-personalization, Plugin:ace-editor, Plugin:saml, Plugin:jquery, Plugin:multibranch-build-strategy-extension, Plugin:workflow-durable-task-step, Plugin:workflow-cps-global-lib, Plugin:publish-over, Plugin:subversion, Plugin:msbuild, Plugin:pipeline-stage-step, Plugin:extended-read-permission, Plugin:allure-jenkins-plugin, Plugin:favorite, Plugin:token-macro, Plugin:docker-workflow, Plugin:dashboard-view, Plugin:blueocean-jwt, Plugin:external-monitor-job, Plugin:git-client, Plugin:blueocean-events, Plugin:git-server, Plugin:xunit, Plugin:ivy, Plugin:blueocean-rest-impl, Plugin:jsch, Plugin:build-failure-analyzer, Plugin:checks-api, Plugin:multibranch-scan-webhook-trigger, Plugin:http_request, Plugin:blueocean-pipeline-scm-api, Plugin:javadoc, Plugin:structs, Plugin:artifactory, Plugin:caffeine-api, Plugin:pipeline-rest-api, Plugin:blueocean-autofavorite, Plugin:git, Plugin:matrix-auth, Plugin:node-iterator-api, Plugin:build-timeout, Plugin:mercurial, Plugin:handlebars, Plugin:antisamy-markup-formatter, Plugin:convert-to-pipeline, Plugin:jdk-tool, Plugin:workflow-scm-step, Plugin:content-replace, Plugin:pipeline-milestone-step, Plugin:github-branch-source, Plugin:snakeyaml-api, Plugin:pipeline-build-step, Plugin:pipeline-input-step, Plugin:trilead-api, Plugin:font-awesome-api, Plugin:role-strategy, Plugin:git-parameter, Plugin:basic-branch-build-strategies, Plugin:workflow-api, Plugin:jackson2-api, Plugin:test-results-analyzer, Plugin:terraform, Plugin:emailext-template, Plugin:build-name-setter, Plugin:blueocean-git-pipeline, Plugin:build-environment, Plugin:scalable-amazon-ecs, Plugin:extra-columns, Plugin:uno-choice, Plugin:github-api, Plugin:ssh-agent, Plugin:sse-gateway, Plugin:ansicolor, Plugin:script-security, Plugin:jquery3-api, Plugin:jenkins-design-language, Plugin:mailer, Plugin:pipeline-github-lib, Plugin:ec2, Plugin:scm-api, Plugin:pipeline-model-definition, Plugin:junit, Plugin:text-finder, Plugin:email-ext, Plugin:aws-credentials, Plugin:hashicorp-vault-plugin, Plugin:s3, Plugin:monitor-pro, Plugin:jira, Plugin:pipeline-graph-analysis, Plugin:jjwt-api, Plugin:workflow-basic-steps, Plugin:envinject, Plugin:windows-slaves, Plugin:pubsub-light, Plugin:credentials-binding, Plugin:resource-disposer, Plugin:slack, Plugin:pipeline-npm, Plugin:blueocean, Plugin:command-launcher, Plugin:active-directory, Plugin:ssh, Plugin:parameterized-trigger, Plugin:credentials, Plugin:amazon-ecs, Plugin:workflow-cps, Plugin:progress-bar-column-plugin, Plugin:blueocean-core-js, Plugin:nunit, Plugin:pipeline-aws]   Any help would be very appreciated   Thanks

          Basil Crow added a comment -

          onallion please follow the steps described in the Debugging section to identify the plugin that is broken.

          Basil Crow added a comment - onallion please follow the steps described in the Debugging section to identify the plugin that is broken.

          mike bayliss added a comment -

          I encountered this problem in two different ways, both at 2.277.3 and 2.277.4

          The save button doing nothing occurred only on the configure system form. Uninstalling the "Job Mail Direct" plugin solved the problem.

          I am also seeing the browser errors, but only intermittently, on a job update submission and only when accessing over my employer's VPN. I'm not seeing any errors logged (but I'm not sure how to enable the relevant logs ).

          mike bayliss added a comment - I encountered this problem in two different ways, both at 2.277.3 and 2.277.4 The save button doing nothing occurred only on the configure system form. Uninstalling the "Job Mail Direct" plugin solved the problem. I am also seeing the browser errors, but only intermittently, on a job update submission and only when accessing over my employer's VPN. I'm not seeing any errors logged (but I'm not sure how to enable the relevant logs ).

            Unassigned Unassigned
            onallion Aviram
            Votes:
            5 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: