Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60857

Wildcard certificates rejected by Winstone after Jetty update

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core, winstone-jetty
    • Jenkins 2.217 on CentOS 6.10
      Jenkins 2.204.3 LTS on Windows Server 2012 R2
      Wildcard-SSL-Certificate in Java-Keystore in PKCS12 format
      Jenkins 2.204.3 LTS on Ubuntu 18.04.4 LTS
    • 2.218 2.204.4

      With 2.217 Jenkins no longer accepts the supplied keystore which worked flawlessly with all former versions.

      It complains about "multiple certificates" even if there is only one stored in the keystore.

      Re-creating the keystore doesn't change a thing.

      Here's the log output:

      2020-01-24 09:59:56.255+0000 [id=1]     SEVERE  winstone.Logger#logInternal: Container startup failed
      java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
              at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275)
              at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256)
              at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374)
              at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
              at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
              at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
              at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
              at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92)
              at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
              at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
              at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
              at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320)
              at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
              at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
              at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
              at org.eclipse.jetty.server.Server.doStart(Server.java:385)
              at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
              at winstone.Launcher.<init>(Launcher.java:188)
      Caused: java.io.IOException: Failed to start Jetty
              at winstone.Launcher.<init>(Launcher.java:190)
              at winstone.Launcher.main(Launcher.java:359)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at Main._main(Main.java:375)
              at Main.main(Main.java:151)
      2020-01-24 09:59:56.256+0000 [id=22]    WARNING o.j.h.a.Index$2$1#fetch: Failed to load hudson.model.Queue
      java.lang.ClassNotFoundException: hudson.model.queue.QueueSorter
              at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
              at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:543)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
      Caused: java.lang.NoClassDefFoundError: hudson/model/queue/QueueSorter
              at java.lang.Class.getDeclaredMethods0(Native Method)
              at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
              at java.lang.Class.getDeclaredMethods(Class.java:1975)
              at org.jvnet.hudson.annotation_indexer.Index$2$1.fetch(Index.java:103)
              at org.jvnet.hudson.annotation_indexer.Index$2$1.hasNext(Index.java:73)
              at org.jvnet.hudson.annotation_indexer.SubtypeIterator.fetch(SubtypeIterator.java:18)
              at org.jvnet.hudson.annotation_indexer.SubtypeIterator.hasNext(SubtypeIterator.java:28)
              at hudson.init.TaskMethodFinder.discoverTasks(TaskMethodFinder.java:56)
              at hudson.init.InitializerFinder.discoverTasks(InitializerFinder.java:33)
              at hudson.init.TaskMethodFinder.discoverTasks(TaskMethodFinder.java:32)
              at org.jvnet.hudson.reactor.TaskBuilder$2.discoverTasks(TaskBuilder.java:61)
              at org.jvnet.hudson.reactor.Reactor.<init>(Reactor.java:151)
              at org.jvnet.hudson.reactor.Reactor.<init>(Reactor.java:156)
              at jenkins.model.Jenkins$5.<init>(Jenkins.java:1127)
              at jenkins.model.Jenkins.executeReactor(Jenkins.java:1127)
              at jenkins.model.Jenkins.<init>(Jenkins.java:966)
              at hudson.model.Hudson.<init>(Hudson.java:85)
              at hudson.model.Hudson.<init>(Hudson.java:81)
              at hudson.WebAppMain$3.run(WebAppMain.java:233)
      
      

          [JENKINS-60857] Wildcard certificates rejected by Winstone after Jetty update

          H. Feldker added a comment -

          Hello,

          we encountered a similar issue (same exception), but our keystore is in JKS format and we do not have a wildcard certificate.

          My guess is that this issue is connected with the jetty update 9.4.23:

          releasenotes entry: 4325 Deprecate SniX509ExtendedKeyManager constructor without SslContextFactory$Server)

           

          Best regards,

          Heiko

          H. Feldker added a comment - Hello, we encountered a similar issue (same exception), but our keystore is in JKS format and we do not have a wildcard certificate. My guess is that this issue is connected with the jetty update 9.4.23: releasenotes entry: 4325 Deprecate SniX509ExtendedKeyManager constructor without SslContextFactory$Server)   Best regards, Heiko

          Christian Keck added a comment - - edited

          I agree with Heiko that it must be related with a change in the API of Jetty or at least a change in the acceptance of missing parameters in some methods.

          However, the noted deprecation seems to lead to a different error message as found here for example: https://github.com/eclipse/jetty.project/issues/4425

          EDIT: Nevermind - the message had been updated in 4.25...

          Christian Keck added a comment - - edited I agree with Heiko that it must be related with a change in the API of Jetty or at least a change in the acceptance of missing parameters in some methods. However, the noted deprecation seems to lead to a different error message as found here for example: https://github.com/eclipse/jetty.project/issues/4425 EDIT: Nevermind - the message had been updated in 4.25...

          I got the same issue after upgrading to 2.217.

          François Isabelle added a comment - I got the same issue after upgrading to 2.217.

          Same for me

          Alexander Gängel added a comment - Same for me

          Jesse Glick added a comment -

          I will check if I can reproduce the issue and look for a fix.

          Jesse Glick added a comment - I will check if I can reproduce the issue and look for a fix.

          Shawn Bates added a comment - - edited

          Same issue for me too.  Is there a way to revert back to previous version using Linux?

          Shawn Bates added a comment - - edited Same issue for me too.  Is there a way to revert back to previous version using Linux?

          shawnbat: Yes, you can savely downgrade to a former version.

          Christian Keck added a comment - shawnbat : Yes, you can savely downgrade to a former version.

          Shawn Bates added a comment -

          Would you have steps or commands for downgrading on Linux?

          Shawn Bates added a comment - Would you have steps or commands for downgrading on Linux?

          Redhat/Centos: "yum downgrade jenkins"

          Debian/Ubuntu: "apt-get install jenkins=<version>"

          You may need to find the exakt <version> sting by invoking "apt-cache madison jenkins" before.

          HTH

          Christian Keck added a comment - Redhat/Centos: "yum downgrade jenkins" Debian/Ubuntu: "apt-get install jenkins=<version>" You may need to find the exakt <version> sting by invoking "apt-cache madison jenkins" before. HTH

          Shawn Bates added a comment -

          That worked.  Thank you Sir!

          Shawn Bates added a comment - That worked.  Thank you Sir!

          Jesse Glick added a comment -

          I am not able to reproduce this error. First of all, HttpsConnectorFactoryTest passes in Winstone sources. Second, I followed these instructions (amended only in trivial ways) and was able to run a development build of Jenkins without any issues, including using curl -ik to access the index page after it started up. Is there some crucial factor I am missing? I can offer a pull request for the deprecation warning but it is senseless to ship a purported fix without being able to verify the effect.

          Jesse Glick added a comment - I am not able to reproduce this error. First of all, HttpsConnectorFactoryTest passes in Winstone sources. Second, I followed these instructions (amended only in trivial ways) and was able to run a development build of Jenkins without any issues, including using curl -ik to access the index page after it started up. Is there some crucial factor I am missing? I can offer a pull request for the deprecation warning but it is senseless to ship a purported fix without being able to verify the effect.

          Jesse Glick added a comment -

          Seems to only affect keystores using SNI: wildcards, multiple hosts, multiple aliases. I am looking for a sample to test against.

          Jesse Glick added a comment - Seems to only affect keystores using SNI: wildcards, multiple hosts, multiple aliases. I am looking for a sample to test against.

          Jesse Glick added a comment -

          Tried to pick up a test keystore from Jetty tests, but so far I have been unable to get Winstone/Jenkins to open it. Not a topic I am at all familiar with.

          Jesse Glick added a comment - Tried to pick up a test keystore from Jetty tests , but so far I have been unable to get Winstone/Jenkins to open it. Not a topic I am at all familiar with.

          François Isabelle added a comment - - edited

          Confirmed, I am using wildcard.

          Instructions to generate the keystore (as Ansible snippets)

          With some variables defined

          jenkins_ssl_cert: "\{{ jenkins_ssl_dest }}/your_domain_example_org.crt"
          jenkins_ssl_key: "\{{ jenkins_ssl_dest }}/your_domain_example_org.key"
          jenkins_ssl_jks: "\{{ jenkins_ssl_dest }}/jenkins.jks"
          jenkins_ssl_chain: "\{{ jenkins_ssl_dest }}/DigiCertCA.crt"
          
          

          The keystores are created in 2 steps.

           - name: Create PKCS
           command: "openssl pkcs12 -inkey \{{ jenkins_ssl_key }} -in \{{ jenkins_ssl_cert }} -export -out \{{ jenkins_ssl_dest }}/keys.pkcs12 -passout pass:$SECRET”
           become: true
           become_user: jenkins
           register: create_pkcs_result
          
           - name: Create JKS
           command: " keytool -importkeystore -srckeystore \{{ jenkins_ssl_dest }}/keys.pkcs12 -srcstoretype pkcs12 -destkeystore \{{ jenkins_ssl_jks }} -srcstorepass jenkins -deststorepass $SECRET -noprompt"
           become: true
           become_user: jenkins
           register: create_jks_result
           

          Maybe that can be useful.

          François Isabelle added a comment - - edited Confirmed, I am using wildcard. Instructions to generate the keystore (as Ansible snippets) With some variables defined jenkins_ssl_cert: "\{{ jenkins_ssl_dest }}/your_domain_example_org.crt" jenkins_ssl_key: "\{{ jenkins_ssl_dest }}/your_domain_example_org.key" jenkins_ssl_jks: "\{{ jenkins_ssl_dest }}/jenkins.jks" jenkins_ssl_chain: "\{{ jenkins_ssl_dest }}/DigiCertCA.crt" The keystores are created in 2 steps. - name: Create PKCS command: "openssl pkcs12 -inkey \{{ jenkins_ssl_key }} -in \{{ jenkins_ssl_cert }} -export -out \{{ jenkins_ssl_dest }}/keys.pkcs12 -passout pass:$SECRET” become: true become_user: jenkins register: create_pkcs_result - name: Create JKS command: " keytool -importkeystore -srckeystore \{{ jenkins_ssl_dest }}/keys.pkcs12 -srcstoretype pkcs12 -destkeystore \{{ jenkins_ssl_jks }} -srcstorepass jenkins -deststorepass $SECRET -noprompt" become: true become_user: jenkins register: create_jks_result   Maybe that can be useful.

          jglick, I don't think it's related to SNI or multi-hosts only. Maybe it has something in common with the way the keystore was/is created.

          I our case we created the keystore file like described here: https://coderwall.com/p/3t4xka/import-private-key-and-certificate-into-java-keystore

          Maybe we are missing something here? AFAIK all environment-variables that Jenkins need to operate with a keystore are

          JENKINS_HTTPS_KEYSTORE and JENKINS_HTTPS_KEYSTORE_PASSWORD - is that still valid?

          Christian Keck added a comment - jglick , I don't think it's related to SNI or multi-hosts only. Maybe it has something in common with the way the keystore was/is created. I our case we created the keystore file like described here: https://coderwall.com/p/3t4xka/import-private-key-and-certificate-into-java-keystore Maybe we are missing something here? AFAIK all environment-variables that Jenkins need to operate with a keystore are JENKINS_HTTPS_KEYSTORE and JENKINS_HTTPS_KEYSTORE_PASSWORD - is that still valid?

          Jesse Glick added a comment -

          Instructions to generate the keystore (as Ansible snippets)

          Sorry, I do not do Ansible, and anyway if I understand those instructions correctly they presume that you have a certificate & key as input. For steps to reproduce, I need something I can run from scratch. A shell command to create a new keystore (self-signed is fine I suppose), then the jenkins.war arguments to run with it which worked in older releases and fails now.

          I don't think it's related to SNI or multi-hosts only.

          I am just going by what I read here. I do not claim to know exactly what I am looking at.

          all environment-variables that Jenkins need to operate with a keystore are JENKINS_HTTPS_KEYSTORE and JENKINS_HTTPS_KEYSTORE_PASSWORD - is that still valid?

          I am not aware of such environment variables (perhaps you are using some Docker image?). The Winstone arguments that I know of are named --httpsKeyStore and --httpsKeyStorePassword.

          Jesse Glick added a comment - Instructions to generate the keystore (as Ansible snippets) Sorry, I do not do Ansible, and anyway if I understand those instructions correctly they presume that you have a certificate & key as input. For steps to reproduce, I need something I can run from scratch . A shell command to create a new keystore (self-signed is fine I suppose), then the jenkins.war arguments to run with it which worked in older releases and fails now. I don't think it's related to SNI or multi-hosts only. I am just going by what I read here . I do not claim to know exactly what I am looking at. all environment-variables that Jenkins need to operate with a keystore are JENKINS_HTTPS_KEYSTORE and JENKINS_HTTPS_KEYSTORE_PASSWORD - is that still valid? I am not aware of such environment variables (perhaps you are using some Docker image?). The Winstone arguments that I know of are named --httpsKeyStore and --httpsKeyStorePassword .

          Jesse Glick added a comment -

          Ah, I think christg74 was referring to an RPM option.

          Jesse Glick added a comment - Ah, I think christg74 was referring to an RPM option .

          Jesse Glick added a comment -

          I think I finally managed to reproduce the issue and will work on codifying this into an automated test.

          Jesse Glick added a comment - I think I finally managed to reproduce the issue and will work on codifying this into an automated test.

          Oleg Nenashev added a comment -

          https://github.com/jenkinsci/jenkins/pull/4454 should integrated the fix. Hopefully we will get it released in the next weekly on Sunday

          Oleg Nenashev added a comment - https://github.com/jenkinsci/jenkins/pull/4454  should integrated the fix. Hopefully we will get it released in the next weekly on Sunday

          Oleg Nenashev added a comment -

          It was released in 2.218

          Oleg Nenashev added a comment - It was released in 2.218

          Confirming 2.218 installs/upgrades successfully on my Windows Server, while 2.217 gave the "multiple certificates" error.

          Thanks for the fix.

          Michael Litwak added a comment - Confirming 2.218 installs/upgrades successfully on my Windows Server, while 2.217 gave the "multiple certificates" error. Thanks for the fix.

          I can also confirm that the fix solves the issue.

          Thanks for the great and fast support!

          Christian Keck added a comment - I can also confirm that the fix solves the issue. Thanks for the great and fast support!

          Thanks for the release it's working again

          Alexander Gängel added a comment - Thanks for the release it's working again

          Nick Jones added a comment -

          This just bit us with 2.204.3 LTS. I don't see a corresponding bug report for this same error with the LTS release, so expect this should be reopened (and the Environment field updated to cover LTS) until it's resolved on that release line too.

          Nick Jones added a comment - This just bit us with 2.204.3 LTS. I don't see a corresponding bug report for this same error with the LTS release, so expect this should be reopened (and the Environment field updated to cover LTS) until it's resolved on that release line too.

          Daniel Beck added a comment - - edited

          Looks like JENKINS-60821 was backported into 2.204.3 without considering this regression:

          https://github.com/jenkinsci/jenkins/commit/23fce281bd4aa92791ab8e5793ea884e543d841f is the backport, and only https://github.com/jenkinsci/winstone/releases/tag/winstone-5.8 has the SSL fix.

          Daniel Beck added a comment - - edited Looks like JENKINS-60821 was backported into 2.204.3 without considering this regression: https://github.com/jenkinsci/jenkins/commit/23fce281bd4aa92791ab8e5793ea884e543d841f is the backport, and only https://github.com/jenkinsci/winstone/releases/tag/winstone-5.8 has the SSL fix.

          Daniel Beck added a comment -

          oleg_nenashev olivergondza Does this make the cut for a regression that justifies an unscheduled .4 release?

          Daniel Beck added a comment - oleg_nenashev olivergondza Does this make the cut for a regression that justifies an unscheduled .4 release?

          Oleg Nenashev added a comment -

          IMO yes

          P.S: Sorry. I was so busy with weekly and other things that I dis not even take a look at the RC

          Oleg Nenashev added a comment - IMO yes P.S: Sorry. I was so busy with weekly and other things that I dis not even take a look at the RC

          Jesse Glick added a comment -

          So IIUC it would suffice for winstone to be bumped from 5.7 to 5.8 in the stable-2.204 branch?

          Jesse Glick added a comment - So IIUC it would suffice for winstone to be bumped from 5.7 to 5.8 in the stable-2.204 branch?

          Jonathan Gray added a comment - - edited

          For what it's worth, this prevented my instance from starting and I had to rollback to the 2.204.2 docker container.

          Running from: /usr/share/jenkins/jenkins.war
          webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
          [2020-03-01 17:20:09] [INFO   ] Logging initialized @382ms to org.eclipse.jetty.util.log.JavaUtilLog
          [2020-03-01 17:20:09] [INFO   ] Beginning extraction from war file
          [2020-03-01 17:20:10] [WARNING] Empty contextPath
          [2020-03-01 17:20:10] [WARNING] Using the --httpsPrivateKey/--httpsCertificate options currently relies on unsupported APIs in the Oracle JRE.
          Please use --httpsKeyStore and related options instead.
          [2020-03-01 17:20:10] [INFO   ] Exclude Ciphers [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$]
          [2020-03-01 17:20:10] [INFO   ] jetty-9.4.25.v20191220; built: 2019-12-20T17:00:00.294Z; git: a9729c7e7f33a459d2616a8f9e9ba8a90f432e95; jvm 1.8.0_242-b08
          [2020-03-01 17:20:10] [INFO   ] NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
          [2020-03-01 17:20:10] [INFO   ] DefaultSessionIdManager workerName=node0
          [2020-03-01 17:20:10] [INFO   ] No SessionScavenger set, using defaults
          [2020-03-01 17:20:10] [INFO   ] node0 Scavenging every 660000ms
          [2020-03-01 17:20:11] [INFO   ] Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
          [2020-03-01 17:20:11] [INFO   ] Started w.@2ad48653{Jenkins v2.204.3,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
          [2020-03-01 17:20:11] [INFO   ] Started ServerConnector@77b52d12{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
          [2020-03-01 17:20:11] [INFO   ] x509=X509@17d919b6(hudson,h=[ci.teamlab.domain.invalid, team-jenkins-oshoc-01.team.domain.invalid, cinew.teamlab.domain.invalid, www.ci.teamlab.domain.invalid, www.cinew.teamlab.domain.invalid],w=[]) for SslContextFactory@53f3bdbd[provider=null,keyStore=null,trustStore=null]
          [2020-03-01 17:20:11] [INFO   ] Stopped ServerConnector@77b52d12{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
          [2020-03-01 17:20:11] [INFO   ] Stopped ServerConnector@100fc185{SSL,[ssl, http/1.1]}{0.0.0.0:8443}
          [2020-03-01 17:20:11] [INFO   ] node0 Stopped scavenging
          [2020-03-01 17:20:11] [INFO   ] Shutting down a Jenkins instance that was still starting up
          [2020-03-01 17:20:11] [INFO   ] Stopped w.@2ad48653{Jenkins v2.204.3,/,null,UNAVAILABLE}{/var/jenkins_home/war}
          [2020-03-01 17:20:11] [INFO   ] Jetty shutdown successfully
          java.io.IOException: Failed to start Jetty
          	at winstone.Launcher.<init>(Launcher.java:191)
          	at winstone.Launcher.main(Launcher.java:362)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at Main._main(Main.java:375)
          	at Main.main(Main.java:151)
          Caused by: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
          	at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275)
          	at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256)
          	at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374)
          	at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
          	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
          	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
          	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
          	at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92)
          	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
          	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
          	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
          	at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320)
          	at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
          	at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
          	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
          	at org.eclipse.jetty.server.Server.doStart(Server.java:385)
          	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
          	at winstone.Launcher.<init>(Launcher.java:189)
          	... 7 more
          Exception in thread "Jenkins initialization thread" java.lang.NoClassDefFoundError: hudson/util/HudsonFailedToLoad
          	at hudson.WebAppMain$3.run(WebAppMain.java:247)
          Caused by: java.lang.ClassNotFoundException: hudson.util.HudsonFailedToLoad
          	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
          	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:543)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
          	... 1 more
          [2020-03-01 17:20:11] [SEVERE ] Container startup failed
          

          edit: Added logs

          Jonathan Gray added a comment - - edited For what it's worth, this prevented my instance from starting and I had to rollback to the 2.204.2 docker container. Running from: /usr/share/jenkins/jenkins.war webroot: EnvVars.masterEnvVars.get("JENKINS_HOME") [2020-03-01 17:20:09] [INFO ] Logging initialized @382ms to org.eclipse.jetty.util.log.JavaUtilLog [2020-03-01 17:20:09] [INFO ] Beginning extraction from war file [2020-03-01 17:20:10] [WARNING] Empty contextPath [2020-03-01 17:20:10] [WARNING] Using the --httpsPrivateKey/--httpsCertificate options currently relies on unsupported APIs in the Oracle JRE. Please use --httpsKeyStore and related options instead. [2020-03-01 17:20:10] [INFO ] Exclude Ciphers [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$] [2020-03-01 17:20:10] [INFO ] jetty-9.4.25.v20191220; built: 2019-12-20T17:00:00.294Z; git: a9729c7e7f33a459d2616a8f9e9ba8a90f432e95; jvm 1.8.0_242-b08 [2020-03-01 17:20:10] [INFO ] NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet [2020-03-01 17:20:10] [INFO ] DefaultSessionIdManager workerName=node0 [2020-03-01 17:20:10] [INFO ] No SessionScavenger set, using defaults [2020-03-01 17:20:10] [INFO ] node0 Scavenging every 660000ms [2020-03-01 17:20:11] [INFO ] Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME") [2020-03-01 17:20:11] [INFO ] Started w.@2ad48653{Jenkins v2.204.3,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war} [2020-03-01 17:20:11] [INFO ] Started ServerConnector@77b52d12{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} [2020-03-01 17:20:11] [INFO ] x509=X509@17d919b6(hudson,h=[ci.teamlab.domain.invalid, team-jenkins-oshoc-01.team.domain.invalid, cinew.teamlab.domain.invalid, www.ci.teamlab.domain.invalid, www.cinew.teamlab.domain.invalid],w=[]) for SslContextFactory@53f3bdbd[provider=null,keyStore=null,trustStore=null] [2020-03-01 17:20:11] [INFO ] Stopped ServerConnector@77b52d12{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} [2020-03-01 17:20:11] [INFO ] Stopped ServerConnector@100fc185{SSL,[ssl, http/1.1]}{0.0.0.0:8443} [2020-03-01 17:20:11] [INFO ] node0 Stopped scavenging [2020-03-01 17:20:11] [INFO ] Shutting down a Jenkins instance that was still starting up [2020-03-01 17:20:11] [INFO ] Stopped w.@2ad48653{Jenkins v2.204.3,/,null,UNAVAILABLE}{/var/jenkins_home/war} [2020-03-01 17:20:11] [INFO ] Jetty shutdown successfully java.io.IOException: Failed to start Jetty at winstone.Launcher.<init>(Launcher.java:191) at winstone.Launcher.main(Launcher.java:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at Main._main(Main.java:375) at Main.main(Main.java:151) Caused by: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275) at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256) at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.server.Server.doStart(Server.java:385) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at winstone.Launcher.<init>(Launcher.java:189) ... 7 more Exception in thread "Jenkins initialization thread" java.lang.NoClassDefFoundError: hudson/util/HudsonFailedToLoad at hudson.WebAppMain$3.run(WebAppMain.java:247) Caused by: java.lang.ClassNotFoundException: hudson.util.HudsonFailedToLoad at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:543) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) ... 1 more [2020-03-01 17:20:11] [SEVERE ] Container startup failed edit: Added logs

          Paul Clark added a comment -

          Is this just simply updating the winstone version at https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.3/war/pom.xml#L104 to 5.8?

          Paul Clark added a comment - Is this just simply updating the winstone version at  https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.3/war/pom.xml#L104  to 5.8?

          Horst Platz added a comment -

          same for me...

          Ubuntu 18.04.4 LTS

          :~$ java -version
          openjdk version "11.0.6" 2020-01-14
          OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1)
          OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing)

          after Update from 2.204.2 to 2.204.3

          Mar 2 09:26:39 build01 hpljenkins[23864]: 2020-03-02 08:26:39.735+0000 [id=1]#011SEVERE#011winstone.Logger#logInternal: Container startup failed
          Mar 2 09:26:39 build01 hpljenkins[23864]: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
          [...]
          Mar 2 09:26:39 build01 systemd[1]: hpljenkins.service: Main process exited, code=exited, status=1/FAILURE
          Mar 2 09:26:39 build01 systemd[1]: hpljenkins.service: Failed with result 'exit-code'.

           

          Rollback to 2.204.2

          Horst Platz added a comment - same for me... Ubuntu 18.04.4 LTS :~$ java -version openjdk version "11.0.6" 2020-01-14 OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1) OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing) after Update from 2.204.2 to 2.204.3 Mar 2 09:26:39 build01 hpljenkins [23864] : 2020-03-02 08:26:39.735+0000 [id=1] #011SEVERE#011winstone.Logger#logInternal: Container startup failed Mar 2 09:26:39 build01 hpljenkins [23864] : java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) [...] Mar 2 09:26:39 build01 systemd [1] : hpljenkins.service: Main process exited, code=exited, status=1/FAILURE Mar 2 09:26:39 build01 systemd [1] : hpljenkins.service: Failed with result 'exit-code'.   Rollback to 2.204.2

          Oleg Nenashev added a comment -

          I will submit a patch and coordinate .4 with other stakeholders

          Oleg Nenashev added a comment - I will submit a patch and coordinate .4 with other stakeholders

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - https://github.com/jenkinsci/jenkins/pull/4539

          Jesse Glick added a comment -

          LTS patch merged.

          Jesse Glick added a comment - LTS patch merged.

          I update my jenkins to LTS 2.204.3 and it would not even launch 

           

          $ cat failed-boot-attempts.txt Mon Mar 02 08:01:43 PST 2020 Mon Mar 02 08:08:07 PST 2020 Mon Mar 02 08:08:54 PST 2020 Mon Mar 02 08:35:52 PST 2020 Mon Mar 02 08:37:10 PST 2020 Mon Mar 02 08:40:31 PST 2020 Mon Mar 02 08:42:34 PST 2020 Mon Mar 02 08:44:01 PST 2020 Mon Mar 02 08:46:54 PST 2020 Mon Mar 02 08:47:53 PST 2020 Mon Mar 02 08:56:15 PST 2020 [08:58:31] root@mtjenkins01/var/lib/jenkins $ cat /var/log/jenkins/jenkins.log Running from: /usr/lib/jenkins/jenkins.war 2020-03-02 16:56:15.111+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMax is now deprecated 2020-03-02 16:56:15.127+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMaxIdle is now deprecated 2020-03-02 16:56:15.132+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @498ms to org.eclipse.jetty.util.log.JavaUtilLog 2020-03-02 16:56:15.179+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file 2020-03-02 16:56:15.207+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath 2020-03-02 16:56:15.412+0000 [id=1] INFO winstone.Logger#logInternal: Exclude Ciphers [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$] 2020-03-02 16:56:15.438+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.25.v20191220; built: 2019-12-20T17:00:00.294Z; git: a9729c7e7f33a459d2616a8f9e9ba8a90f432e95; jvm 1.8.0_222-b10 2020-03-02 16:56:15.645+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet 2020-03-02 16:56:15.679+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0 2020-03-02 16:56:15.679+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults 2020-03-02 16:56:15.681+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 660000ms 2020-03-02 16:56:15.967+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: SystemProperties.getProperty("JENKINS_HOME") 2020-03-02 16:56:16.038+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@2a7ed1f{Jenkins v2.204.3,/,file:///var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/war} 2020-03-02 16:56:16.084+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@463fd068(mtcoveritydb01,h=[],w=[]) for SslContextFactory@895e367[provider=null,keyStore=null,trustStore=null] 2020-03-02 16:56:16.084+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@1b266842(starfieldclass2ca,h=[],w=[]) for SslContextFactory@895e367[provider=null,keyStore=null,trustStore=null] 2020-03-02 16:56:16.085+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@7a3793c7(taiwangrca,h=[],w=[]) for SslContextFactory@895e367[provider=null,keyStore=null,trustStore=null] 2020-03-02 16:56:16.085+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@42b3b079(itservices.def.com,h=[gaalpa2adssrv53.itservices.def.com, itservices.def.com, gaalpa2adssrv53, itservices, its-ad-ldap.it.xxx.com, its-ad-ldap.lrns.def.com, its-ad-ldap.cci.xxx.com],w=[]) for SslContextFactory@895e367[provider=null,keyStore=null,trustStore=null] 2020-03-02 16:56:16.086+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@651aed93(geotrustglobalca,h=[],w=[]) for SslContextFactory@895e367[provider=null,keyStore=null,trustStore=null] 2020-03-02 16:56:16.086+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@4dd6fd0a(1,h=[mtjenkins01.quantum.xxx.com],w=[]) for SslContextFactory@895e367[provider=null,keyStore=null,trustStore=null] 2020-03-02 16:56:16.087+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@bb9e6dc(verisignclass3publicprimarycertificationauthority-g3,h=[],w=[]) for SslContextFactory@895e367[provider=null,keyStore=null,trustStore=null] 2020-03-02 16:56:16.087+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@5456afaa(godaddyclass2ca,h=[],w=[]) for SslContextFactory@895e367[provider=null,keyStore=null,trustStore=null] 2020-03-02 16:56:16.088+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@6692b6c6(trustisfpsrootca,h=[],w=[]) for SslContextFactory@895e367[provider=null,keyStore=null,trustStore=null] 2020-03-02 16:56:16.088+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@1cd629b3(epkirootcertificationauthority,h=[],w=[]) for SslContextFactory@895e367[provider=null,keyStore=null,trustStore=null] 2020-03-02 16:56:16.092+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStop: Stopped ServerConnector@5702b3b1{SSL,[ssl, http/1.1]}{0.0.0.0:8443} 2020-03-02 16:56:16.092+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#stopScavenging: node0 Stopped scavenging 2020-03-02 16:56:16.094+0000 [id=1] INFO hudson.WebAppMain#contextDestroyed: Shutting down a Jenkins instance that was still starting up java.lang.Throwable: reason at hudson.WebAppMain.contextDestroyed(WebAppMain.java:388) at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:937) at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:565) at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:905) at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:367) at org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1450) at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1415) at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:980) at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:284) at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:547) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:93) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201) at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:111) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:93) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201) at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:111) at org.eclipse.jetty.server.Server.doStop(Server.java:454) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:93) at winstone.Launcher.shutdown(Launcher.java:311) at winstone.Launcher.<init>(Launcher.java:202) at winstone.Launcher.main(Launcher.java:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at Main._main(Main.java:375) at Main.main(Main.java:151) 2020-03-02 16:56:16.097+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStop: Stopped w.@2a7ed1f{Jenkins v2.204.3,/,null,UNAVAILABLE}{/var/cache/jenkins/war} Exception in thread "Jenkins initialization thread" java.lang.NoClassDefFoundError: hudson/util/HudsonFailedToLoad at hudson.WebAppMain$3.run(WebAppMain.java:247) Caused by: java.lang.ClassNotFoundException: hudson.util.HudsonFailedToLoad at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:543) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more 2020-03-02 16:56:16.097+0000 [id=1] INFO winstone.Logger#logInternal: Jetty shutdown successfully java.io.IOException: Failed to start Jetty at winstone.Launcher.<init>(Launcher.java:191) at winstone.Launcher.main(Launcher.java:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at Main._main(Main.java:375) at Main.main(Main.java:151) Caused by: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275) at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256) at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.server.Server.doStart(Server.java:385) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at winstone.Launcher.<init>(Launcher.java:189) ... 7 more 2020-03-02 16:56:16.098+0000 [id=1] SEVERE winstone.Logger#logInternal: Container startup failed java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275) at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256) at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.server.Server.doStart(Server.java:385) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at winstone.Launcher.<init>(Launcher.java:189) Caused: java.io.IOException: Failed to start Jetty at winstone.Launcher.<init>(Launcher.java:191) at winstone.Launcher.main(Launcher.java:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at Main._main(Main.java:375) at Main.main(Main.java:151) [08:58:55] root@mtjenkins01/var/lib/jenkins $
          
           

           I had to revert back to 2.190.1 

          and it messed up my jenkins

          Please fix

           

          Steven Fransen added a comment - I update my jenkins to LTS 2.204.3 and it would not even launch    $ cat failed-boot-attempts.txt Mon Mar 02 08:01:43 PST 2020 Mon Mar 02 08:08:07 PST 2020 Mon Mar 02 08:08:54 PST 2020 Mon Mar 02 08:35:52 PST 2020 Mon Mar 02 08:37:10 PST 2020 Mon Mar 02 08:40:31 PST 2020 Mon Mar 02 08:42:34 PST 2020 Mon Mar 02 08:44:01 PST 2020 Mon Mar 02 08:46:54 PST 2020 Mon Mar 02 08:47:53 PST 2020 Mon Mar 02 08:56:15 PST 2020 [08:58:31] root@mtjenkins01/ var /lib/jenkins $ cat / var /log/jenkins/jenkins.log Running from: /usr/lib/jenkins/jenkins.war 2020-03-02 16:56:15.111+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMax is now deprecated 2020-03-02 16:56:15.127+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMaxIdle is now deprecated 2020-03-02 16:56:15.132+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @498ms to org.eclipse.jetty.util.log.JavaUtilLog 2020-03-02 16:56:15.179+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file 2020-03-02 16:56:15.207+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath 2020-03-02 16:56:15.412+0000 [id=1] INFO winstone.Logger#logInternal: Exclude Ciphers [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$] 2020-03-02 16:56:15.438+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.25.v20191220; built: 2019-12-20T17:00:00.294Z; git: a9729c7e7f33a459d2616a8f9e9ba8a90f432e95; jvm 1.8.0_222-b10 2020-03-02 16:56:15.645+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet 2020-03-02 16:56:15.679+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0 2020-03-02 16:56:15.679+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults 2020-03-02 16:56:15.681+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 660000ms 2020-03-02 16:56:15.967+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: / var /lib/jenkins found at: SystemProperties.getProperty( "JENKINS_HOME" ) 2020-03-02 16:56:16.038+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@2a7ed1f{Jenkins v2.204.3,/,file: /// var /cache/jenkins/war/,AVAILABLE}{/ var /cache/jenkins/war} 2020-03-02 16:56:16.084+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@463fd068(mtcoveritydb01,h=[],w=[]) for SslContextFactory@895e367[provider= null ,keyStore= null ,trustStore= null ] 2020-03-02 16:56:16.084+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@1b266842(starfieldclass2ca,h=[],w=[]) for SslContextFactory@895e367[provider= null ,keyStore= null ,trustStore= null ] 2020-03-02 16:56:16.085+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@7a3793c7(taiwangrca,h=[],w=[]) for SslContextFactory@895e367[provider= null ,keyStore= null ,trustStore= null ] 2020-03-02 16:56:16.085+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@42b3b079(itservices.def.com,h=[gaalpa2adssrv53.itservices.def.com, itservices.def.com, gaalpa2adssrv53, itservices, its-ad-ldap.it.xxx.com, its-ad-ldap.lrns.def.com, its-ad-ldap.cci.xxx.com],w=[]) for SslContextFactory@895e367[provider= null ,keyStore= null ,trustStore= null ] 2020-03-02 16:56:16.086+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@651aed93(geotrustglobalca,h=[],w=[]) for SslContextFactory@895e367[provider= null ,keyStore= null ,trustStore= null ] 2020-03-02 16:56:16.086+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@4dd6fd0a(1,h=[mtjenkins01.quantum.xxx.com],w=[]) for SslContextFactory@895e367[provider= null ,keyStore= null ,trustStore= null ] 2020-03-02 16:56:16.087+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@bb9e6dc(verisignclass3publicprimarycertificationauthority-g3,h=[],w=[]) for SslContextFactory@895e367[provider= null ,keyStore= null ,trustStore= null ] 2020-03-02 16:56:16.087+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@5456afaa(godaddyclass2ca,h=[],w=[]) for SslContextFactory@895e367[provider= null ,keyStore= null ,trustStore= null ] 2020-03-02 16:56:16.088+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@6692b6c6(trustisfpsrootca,h=[],w=[]) for SslContextFactory@895e367[provider= null ,keyStore= null ,trustStore= null ] 2020-03-02 16:56:16.088+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@1cd629b3(epkirootcertificationauthority,h=[],w=[]) for SslContextFactory@895e367[provider= null ,keyStore= null ,trustStore= null ] 2020-03-02 16:56:16.092+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStop: Stopped ServerConnector@5702b3b1{SSL,[ssl, http/1.1]}{0.0.0.0:8443} 2020-03-02 16:56:16.092+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#stopScavenging: node0 Stopped scavenging 2020-03-02 16:56:16.094+0000 [id=1] INFO hudson.WebAppMain#contextDestroyed: Shutting down a Jenkins instance that was still starting up java.lang.Throwable: reason at hudson.WebAppMain.contextDestroyed(WebAppMain.java:388) at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:937) at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:565) at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:905) at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:367) at org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1450) at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1415) at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:980) at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:284) at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:547) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:93) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201) at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:111) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:93) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:180) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:201) at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:111) at org.eclipse.jetty.server.Server.doStop(Server.java:454) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:93) at winstone.Launcher.shutdown(Launcher.java:311) at winstone.Launcher.<init>(Launcher.java:202) at winstone.Launcher.main(Launcher.java:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at Main._main(Main.java:375) at Main.main(Main.java:151) 2020-03-02 16:56:16.097+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStop: Stopped w.@2a7ed1f{Jenkins v2.204.3,/, null ,UNAVAILABLE}{/ var /cache/jenkins/war} Exception in thread "Jenkins initialization thread" java.lang.NoClassDefFoundError: hudson/util/HudsonFailedToLoad at hudson.WebAppMain$3.run(WebAppMain.java:247) Caused by: java.lang.ClassNotFoundException: hudson.util.HudsonFailedToLoad at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang. ClassLoader .loadClass( ClassLoader .java:424) at java.lang. ClassLoader .loadClass( ClassLoader .java:357) at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:543) at java.lang. ClassLoader .loadClass( ClassLoader .java:357) ... 1 more 2020-03-02 16:56:16.097+0000 [id=1] INFO winstone.Logger#logInternal: Jetty shutdown successfully java.io.IOException: Failed to start Jetty at winstone.Launcher.<init>(Launcher.java:191) at winstone.Launcher.main(Launcher.java:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at Main._main(Main.java:375) at Main.main(Main.java:151) Caused by: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275) at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256) at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.server.Server.doStart(Server.java:385) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at winstone.Launcher.<init>(Launcher.java:189) ... 7 more 2020-03-02 16:56:16.098+0000 [id=1] SEVERE winstone.Logger#logInternal: Container startup failed java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275) at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256) at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.server.Server.doStart(Server.java:385) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at winstone.Launcher.<init>(Launcher.java:189) Caused: java.io.IOException: Failed to start Jetty at winstone.Launcher.<init>(Launcher.java:191) at winstone.Launcher.main(Launcher.java:362) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at Main._main(Main.java:375) at Main.main(Main.java:151) [08:58:55] root@mtjenkins01/ var /lib/jenkins $    I had to revert back to 2.190.1  and it messed up my jenkins Please fix  

          Jesse Glick added a comment -

          This is known to be broken in 2.204.3. Either go back to 2.204.2, or wait for 2.204.4. Please do not comment further in JIRA or the pull request.

          Jesse Glick added a comment - This is known to be broken in 2.204.3. Either go back to 2.204.2, or wait for 2.204.4. Please do not comment further in JIRA or the pull request.

          Winstone component will be reverted in 2.204.5 to a version prior this regression ware introduced.

          Oliver Gondža added a comment - Winstone component will be reverted in 2.204.5 to a version prior this regression ware introduced.

            oleg_nenashev Oleg Nenashev
            christg74 Christian Keck
            Votes:
            12 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:
              Resolved: