Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58094

mTLS client certificates broken in 2.176.1+ by changes in jetty

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Client certificate support is broken in 2.176.1+ by changes in Jetty.

      When running jenkins with winstone/jetty configured to require client cert auth most (all?) client certs will fail validation with an error such as:

      java.security.cert.CertificateException: No subject alternative names matching IP address x.x.x.x.x found

      This error message is only visible if debug logging is enabled: JAVA_OPTS= -Djavax.net.debug=ssl:handshake

      We have tracked this down to the Winstone 5.2 upgrade in 2.176.1 (https://github.com/jenkins-infra/jenkins.io/blob/master/content/_data/changelogs/lts.yml#L2518) which in turn was caused by a change in jetty 9.4.15+

      The summary of the issue is that Jetty changed the default behavior of their SslContextFactory to improve cert verification for their HttpClient implementation. Since both the Http server and client functions in jetty use the same SslContextFactory this broke the https server use case.

      Jetty decided to address this bug by creating SslContextFactory.Server and .Client subclasses in v9.4.18+. But the broken behavior remains in the parent SslContextFactory which is what winstone is using: https://github.com/eclipse/jetty.project/pull/3480

      There are several bug/issue reports against the Jetty project for this:

      It appears that there are two possible fixes in winstone:

      1. Switch from using SslContextFactory -> SslContextFactory.Server
      2. or, set endpointIdentificationAlgorithm to "null" in the existing use of SslContextFactory

      Here is an example from the apache/felix project of how they fixed this bug by switching to the new Server subclass introduced by jetty #3480 (v9.4.18+): https://github.com/apache/felix/pull/199/files

      I have been unable to find a workaround for this such as setting the "endpointIdentificationAlgorithm" property to 'null' using JAVA_OPTS. The only fix we've found is to pin our use of jenkins to v2.164.3

       

       

        Attachments

          Activity

          Hide
          oleg_nenashev Oleg Nenashev added a comment -
          Show
          oleg_nenashev Oleg Nenashev added a comment - CC Olivier Lamy
          Hide
          mtdeguzis Michael DeGuzis added a comment -

          I wonder if this is also affecting our failure to bind to active directory using the JDK Trustore option of Jenkins > Configure Global Security > Security Ream > Active Directory > TLS Configuration.

          simple bind failed: LDAP_HOST:636
          sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
          	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
          	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
          	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
          Caused: sun.security.validator.ValidatorException: PKIX path building failed
          	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
          	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
          	at sun.security.validator.Validator.validate(Validator.java:262)
          	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
          	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
          	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
          	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
          Caused: javax.net.ssl.SSLHandshakeException
          	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
          	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
          	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
          	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
          	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
          	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
          	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
          	at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
          	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
          	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
          	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
          	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
          	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
          	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
          	at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:443)
          	at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:416)
          	at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:359)
          	at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:214
          

          As soon as I downgrade to 2.164.3, things work fine again. I setup a QA/Test master with only the Active Directory plugin.

          Thoughts?

          Show
          mtdeguzis Michael DeGuzis added a comment - I wonder if this is also affecting our failure to bind to active directory using the JDK Trustore option of Jenkins > Configure Global Security > Security Ream > Active Directory > TLS Configuration. simple bind failed: LDAP_HOST:636 sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) Caused: sun.security.validator.ValidatorException: PKIX path building failed at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) at sun.security.validator.Validator.validate(Validator.java:262) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) Caused: javax.net.ssl.SSLHandshakeException at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750) at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:443) at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:416) at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:359) at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:214 As soon as I downgrade to 2.164.3, things work fine again. I setup a QA/Test master with only the Active Directory plugin. Thoughts?
          Hide
          mtdeguzis Michael DeGuzis added a comment - - edited

          I found an answer for my issue, and I'm still not sure if it's related. For whatever reason, beyond Jenkins 2.164.3, the JVM acts as if it no longer is looking at the default trust store at /etc/ssl/certs/java/cacerts. There was no need to import the public cert from our AD LDAP server into the default truststore.

          export JAVA_OPTS="-Dhudson.plugins.active_directory.ActiveDirectorySecurityRealm.forceLdaps=true \
              -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York \
              -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \
              -Djavax.net.ssl.trustStorePassword=changeit"
          

          Full docker run:

          + sudo docker run -d --name jenkins-server-test --network=jenkins-network --volume jenkins-test-data:/var/jenkins_home --publish 8555:8443 --env 'JAVA_OPTS=-Dhudson.plugins.active_directory.ActiveDirectorySecurityRealm.forceLdaps=true     -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York     -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts     -Djavax.net.ssl.trustStorePassword=changeit' geisinger-jenkins-test:2.176.2 --sessionTimeout=15 --httpPort=-15 --httpsPort=8443 --httpsKeyStore=/var/jenkins_home/.keystore/host.domain.com.all.jks --httpsKeyStorePassword=*****************
          
          Show
          mtdeguzis Michael DeGuzis added a comment - - edited I found an answer for my issue, and I'm still not sure if it's related. For whatever reason, beyond Jenkins 2.164.3, the JVM acts as if it no longer is looking at the default trust store at /etc/ssl/certs/java/cacerts. There was no need to import the public cert from our AD LDAP server into the default truststore. export JAVA_OPTS="-Dhudson.plugins.active_directory.ActiveDirectorySecurityRealm.forceLdaps= true \ -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York \ -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \ -Djavax.net.ssl.trustStorePassword=changeit" Full docker run: + sudo docker run -d --name jenkins-server-test --network=jenkins-network --volume jenkins-test-data:/ var /jenkins_home --publish 8555:8443 --env 'JAVA_OPTS=-Dhudson.plugins.active_directory.ActiveDirectorySecurityRealm.forceLdaps= true -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York -Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts -Djavax.net.ssl.trustStorePassword=changeit' geisinger-jenkins-test:2.176.2 --sessionTimeout=15 --httpPort=-15 --httpsPort=8443 --httpsKeyStore=/ var /jenkins_home/.keystore/host.domain.com.all.jks --httpsKeyStorePassword=*****************

            People

            Assignee:
            olamy Olivier Lamy
            Reporter:
            jmiller joe miller
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated: