-
Bug
-
Resolution: Unresolved
-
Major
I'm opening this as a continuation of https://issues.jenkins.io/browse/JENKINS-64341 as suggested. This is to add a few more details to Ryan's comment based on Mark's response.
We're seeing an issue with Jenkins 2.280-jdk11 where when clicking on the "Save" button in Configure System (even there are no changes made), it breaks and shows the error screen:
Error screen
Logs
2021-02-26 22:08:17.336+0000 [id=2063] WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID 45950dec-8624-44ad-8878-aa2fd8e2858f2021-02-26 22:08:17.336+0000 [id=2063] WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID 45950dec-8624-44ad-8878-aa2fd8e2858fnet.sf.json.JSONException: JSONObject["rememberPasswordEnabled"] not found. at net.sf.json.JSONObject.getString(JSONObject.java:2040) at com.sonyericsson.rebuild.RebuildDescriptor.configure(RebuildDescriptor.java:28) at jenkins.model.Jenkins.configureDescriptor(Jenkins.java:3859) at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:3823) at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) at org.kohsuke.stapler.SelectionInterceptedFunction$Adapter.invoke(SelectionInterceptedFunction.java:36) at org.kohsuke.stapler.verb.HttpVerbInterceptor.invoke(HttpVerbInterceptor.java:48) at org.kohsuke.stapler.SelectionInterceptedFunction.bindAndInvoke(SelectionInterceptedFunction.java:26) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694) at org.kohsuke.stapler.Stapler.service(Stapler.java:240) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:111) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905) at java.base/java.lang.Thread.run(Thread.java:834)
List of plugins
Jenkins: 2.280 OS: Linux - 5.4.49+ --- pipeline-input-step:2.12 multiple-scms:0.6 pipeline-stage-view:2.19 analysis-model-api:9.8.0 role-strategy:3.1 graphiteIntegrator:1.2 blueocean-i18n:1.24.4 workflow-basic-steps:2.23 external-monitor-job:1.7 pipeline-rest-api:2.19 blueocean-pipeline-api-impl:1.24.4 naginator:1.18.1 job-parameter-summary:0.5 mercurial:2.12 cucumber-reports:5.5.0 basic-branch-build-strategies:1.3.2 build-environment:1.7 pam-auth:1.6 stashNotifier:1.20 checks-api:1.5.0 configurationslicing:1.51 job-dsl:1.77 envinject:2.4.0 build-failure-analyzer:2.0.0 blueocean-git-pipeline:1.24.4 uno-choice:2.5.2 workflow-cps:2.89 build-history-metrics-plugin:1.2 octopusdeploy:3.1.2 pipeline-npm:0.9.2 kubernetes-credentials:0.8.0 jobcopy-builder:1.4.2 blueocean-rest-impl:1.24.4 dashboard-view:2.14 build-monitor-plugin:1.12+build.201809061734 badge:1.8 plugin-usage-plugin:1.2 config-file-provider:3.7.0 nodejs:1.3.11 build-user-vars-plugin:1.7 blueocean-jwt:1.24.4 cloudbees-bitbucket-branch-source:2.9.7 websocket:1.0.6 hipchat:2.2.1 favorite:2.3.3 script-security:1.76 metrics:4.0.2.7 jqs-monitoring:1.4 hudson-wsclean-plugin:1.0.8 purge-job-history:1.6 trilead-api:1.0.13 font-awesome-api:5.15.2-1 google-container-registry-auth:0.3 ssh-credentials:1.18.1 postbuildscript:2.11.0 extensible-choice-parameter:1.7.0 stash-pullrequest-builder:1.17 matrix-project:1.18 pipeline-model-definition:1.8.4 conditional-buildstep:1.4.1 github-api:1.122 dependency-check-jenkins-plugin:5.1.1 cobertura:1.16 copyartifact:1.46 bootstrap4-api:4.6.0-1 disk-usage:0.28 dependency-track:3.1.0 cloudbees-folder:6.15 simple-theme-plugin:0.6 htmlpublisher:1.25 systemloadaverage-monitor:1.2 kubernetes-client-api:4.13.2-1 purge-build-queue-plugin:1.0 workflow-step-api:2.23 delivery-pipeline-plugin:1.4.2 pipeline-build-step:2.13 groovy-postbuild:2.5 workflow-durable-task-step:2.37 sonar:2.13 pipeline-model-api:1.8.4 workflow-job:2.40 plugin-util-api:1.7.1 nexus-artifact-uploader:2.13 handlebars:1.1.1 blueocean-dashboard:1.24.4 fortify-on-demand-uploader:6.1.0 flexible-publish:0.16.1 jjwt-api:0.11.2-9.c8b45b8bb173 blueocean-autofavorite:1.2.4 kubernetes:1.24.1-efx accelerated-build-now-plugin:1.0.1 workflow-support:3.7 ssh-agent:1.21 managed-scripts:1.5.4 jquery:1.12.4-1 git-tag-message:1.7.1 remote-terminal-access:1.6 mapdb-api:1.0.9.0 credentials-binding:1.24 git:4.6.0 git-parameter:0.9.13 warnings-ng:8.9.1 jobgenerator:1.22 pipeline-stage-tags-metadata:1.8.4 pipeline-multibranch-defaults:2.1 slave-status:1.6 ssh-slaves:1.31.5 forensics-api:0.10.1 project-build-times:1.2.1 blueocean-events:1.24.4 gradle:1.36 workflow-cps-global-lib:2.18 jdk-tool:1.5 jaxb:2.3.0.1 bouncycastle-api:2.18 pipeline-utility-steps:2.6.1 blueocean-pipeline-editor:1.24.4 lockable-resources:2.10 git-server:1.9 greenballs:1.15.1 google-oauth-plugin:1.0.3 subversion:2.14.0 schedule-build:0.5.1 all-changes:1.5 ldap:2.4 nuget:1.0 parameterized-trigger:2.40 git-client:3.6.0 groovy:2.3 jacoco:3.1.1 display-url-api:2.3.4 jenkins-design-language:1.24.4 sse-gateway:1.24 blueocean-jira:1.24.4 blueocean-web:1.24.4 blueocean-bitbucket-pipeline:1.24.4 blueocean-core-js:1.24.4 workflow-multibranch:2.22 github:1.33.1 oauth-credentials:0.4 docker-build-step:2.6 bulk-builder:1.5 plain-credentials:1.7 handy-uri-templates-2-api:2.1.8-1.0 bitbucket:1.1.27 aws-java-sdk:1.11.955 docker-commons:1.17 blueocean:1.24.4 jackson2-api:2.12.1 blueocean-personalization:1.24.4 jobConfigHistory:2.26 run-condition:1.5 http_request:1.8.27 structs:1.22 cloudbees-disk-usage-simple:0.10 javadoc:1.6 credentials:2.3.15 workflow-api:2.41 variant:1.4 blueocean-rest:1.24.4 secondary-timestamper-plugin:1.1 aws-credentials:1.28 StashBranchParameter:0.3.0 durable-task:1.35 built-on-column:1.1 nodelabelparameter:1.7.2 cvs:2.17 jsch:0.1.55.2 next-build-number:1.6 timestamper:1.11.8 global-build-stats:1.5 scm-api:2.6.4 junit:1.48 pubsub-light:1.13 momentjs:1.1.1 slave-setup:1.10 windows-slaves:1.7 extended-read-permission:3.2 blueocean-github-pipeline:1.24.4 ansicolor:0.7.5 active-directory:2.23 gcr-scanner:1.0 translation:1.16 metrics-datadog:1.0 resource-disposer:0.14 rebuild:1.31 backup:1.6.1 jquery3-api:3.5.1-2 workflow-scm-step:2.12 pipeline-milestone-step:1.3.2 publish-over-ssh:1.22 mailer:1.32.1 Parameterized-Remote-Trigger:3.1.5.1 ant:1.11 jquery-detached:1.2.1 google-chat-notification:1.4 echarts-api:4.9.0-3 validating-string-parameter:2.8 envinject-api:1.7 code-coverage-api:1.3.1 git-chooser-alternative:1.1 jira:3.2 matrix-auth:2.6.5 popper-api:1.16.1-1 embeddable-build-status:2.0.3 maven-plugin:3.9 blueocean-pipeline-scm-api:1.24.4 antisamy-markup-formatter:2.1 jsoup:1.10.3 pipeline-model-extensions:1.8.4 pipeline-aggregator-view:1.11 saml:1.1.7 audit-trail:3.8 authentication-tokens:1.4 blueocean-config:1.24.4 statusmonitor:1.3 blueocean-display-url:2.4.1 build-timeout:1.20 authorize-project:1.3.0 blueocean-commons:1.24.4 branch-api:2.6.3 view-job-filters:2.3 any-buildstep:0.1 pipeline-stage-step:2.5 pipeline-graph-analysis:1.10 ace-editor:1.1 apache-httpcomponents-client-4-api:4.5.13-1.0 okhttp-api:3.14.9 thinBackup:1.10 monitoring:1.86.0 confluence-publisher:2.0.6 workflow-aggregator:2.6 powershell:1.4 job-import-plugin:3.4 extended-choice-parameter:0.82 publish-over:0.22 hashicorp-vault-plugin:3.7.0 data-tables-api:1.10.23-2 prometheus:2.0.8 rundeck:3.6.9 file-operations:1.11 jenkins-multijob-plugin:1.36 docker-workflow:1.25 nexus-jenkins-plugin:3.8.20191024-124504.15c0353 bitbucket-build-status-notifier:1.4.2 CustomHistory:1.6 ws-cleanup:0.38 command-launcher:1.5 github-branch-source:2.10.1 token-macro:2.15 email-ext:2.81 configuration-as-code:1.47 implied-labels:0.7 snakeyaml-api:1.27.0 shiningpanda:0.24 heavy-job:1.1
How to reproduce
- Click Manage Jenkins
- Click Configure System
- Click Save button (don't need to make a change)
- Error screen shows up
Research Tests Performed
- JDK Versioning: Jenkins was tested using on v2.280 using both jdk8 and jdk11. The error persisted through both versions of JDK.
- Plugins:
- The plugin list from the last working deployment was used in all Jenkins version testing. The error still appeared, regardless of the list of plugins used.
- The error in the logs was related to a plugin. The solution given by https://issues.jenkins.io/browse/JENKINS-64341 to disable the related plugin was applied. On the next attempt, the same error with a new plugin reference was given. We were not able to reach a final iteration to fully get rid of the error during the minimal testing of this component.
- Jenkins Version:
- The following Jenkins versions were tested: v2.281-jdk11, v2.280-jdk11, v2.280-jdk8, v2.277-jdk11, v2.274-jdk11, v2.267-jdk11, v2.266-jdk11, v2.264-jdk11, v2.263.4-lts-jdk11, v2.262-jdk11
- Of these versions, the only two versions that did not contain the error were v2.263.4-lts-jdk11 and v2.262-jdk11
- This led me to believe that the error originated in a code change between v2.263 and v2.264.
- Note the "Rollbacks" for v2.264 compared to v2.263 in the Jenkins Changelog statistics: https://www.jenkins.io/changelog/
Jenkins 2.264 and later include the changes described in "Major changes in weekly releases". Based on your list of plugin versions and some assumptions, the Google Container Registry Auth plugin is causing the failure when trying to save the global configuration. That plugin was last released 5 years ago. It has not been updated to support the changes that have already arrived in Jenkins weekly 2.264 and later and will be arriving in Jenkins LTS 2.277.1 on March 10, 2021.
Your choices include:
Since you're running a custom version of the Kubernetes plugin, I assume you are able to find someone to build and use plugins with custom changes for your environment.
There were other changes that I made while working with your list of plugins. I assumed that it was safe to upgrade all of them to the most recent available version for Jenkins 2.277 or later. I assumed that kubernetes plugin 1.29.1 was sufficient to replace the custom version you're running based on 1.24.1.
The Dockerfile I used to build my image for testing was:
The plugins.txt file that I used is uploaded as a separate file named 'plugins.txt'
plugins.txt
I updated to the latest plugins as of Feb 27, 2021 and that is available as plugins-latest.txt
If you decide that you'd like to submit a fix for the Google Container Registry Auth plugin, the tables to divs migration guide provides good instructions for the change.