Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66788

There were errors checking the update sites: SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

      Hello Team,

       

      We are currently facing issue while upgrading the plugins. We are getting below SSL handshake error.

       

      There were errors checking the update sites: SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

       

      Current version of jenkins is __ 

      <version>2.289.2</version>

       

      Current version of java on system is

      openjdk version "11.0.12" 2021-07-20 LTS
      OpenJDK Runtime Environment 18.9 (build 11.0.12+7-LTS)
      OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7-LTS, mixed mode, sharing)

       

      Below is JDK configuration from jenkins config.xml

       

      <jdks>
      <jdk>
      <name>java 8</name>
      <home>/usr/lib/jvm/java-1.8.0-openjdk/</home>
      <properties/>
      </jdk>
      </jdks>

       

      Can someone help me to understand why this error is popping? We are using quite latest version jenkins and java but still it is failing to validate. 

       

      Seeking help to resolve this issue.

       

      Thanks and Regards,

      Sagar Nachare.

       

          [JENKINS-66788] There were errors checking the update sites: SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

          Similar versions: Jenkins 2.289.3 and Java 11.0.13, same error.

          I already imported https://get.jenkins.iohttps://updates.jenkins.io, and https://ftp-nyc.osuosl.org/ ssl certs into my java truststore, but still not working

          Jeffrey L. Wong added a comment - Similar versions: Jenkins 2.289.3 and Java 11.0.13, same error. I already imported  https://get.jenkins.io ,  https://updates.jenkins.io , and  https://ftp-nyc.osuosl.org/  ssl certs into my java truststore, but still not working

          I have encountered the same problem with openjdk-11, but it is intermittent.
          It appears that the certificate returned from get.jenkins.io sometimes is a self-signed Kubernetes certificate.

          Steps

          Running Jenkins with SSL logging enabled:

          -Djavax.net.debug=ssl:handshake:verbose:keymanager:trustmanager
          

          Execute following from Jenkins script console:

          try {
            def url = 'https://get.jenkins.io/'
            def connection = new URL(url).openConnection() as HttpURLConnection
            connection.with {
              println "ResponseCode: ${responseCode}"
              println "Headers: ${headerFields}"
              println "Body: ${inputStream.text}"
            }
          } catch (e) {
            println "$e"
          }
          

          Successful response

          javax.net.ssl|DEBUG|0B|Handling POST /jenkins/script from ***** : Jetty (winstone)-11|2022-03-09 10:19:48.822 GMT|X509TrustManagerImpl.java:79|adding as trusted certificates (
            "certificate" : {
              "version"            : "v3",
              "serial number"      : "00 A6 8B 79 29 00 00 00 00 50 D0 91 F9",
              "signature algorithm": "SHA384withECDSA",
              "issuer"             : "CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US",
              "not before"         : "2012-12-18 15:25:36.000 GMT",
              "not  after"         : "2037-12-18 15:55:36.000 GMT",
              "subject"            : "CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US",
              "subject public key" : "EC",
              "extensions"         : [
                {
                  ObjectId: 2.5.29.19 Criticality=true
                  BasicConstraints:[
                    CA:true
                    PathLen:2147483647
                  ]
                },
                {
                  ObjectId: 2.5.29.15 Criticality=true
                  KeyUsage [
                    Key_CertSign
                    Crl_Sign
                  ]
                },
                {
                  ObjectId: 2.5.29.14 Criticality=false
                  SubjectKeyIdentifier [
                  KeyIdentifier [
                  0000: B7 63 E7 1A DD 8D E9 08   A6 55 83 A4 E0 6A 50 41  .c.......U...jPA
                  0010: 65 11 42 49                                        e.BI
                  ]
                  ]
                }
              ]},
            "certificate" : {
              "version"            : "v3",
              "serial number"      : "0C F0 8E 5C 08 16 A5 AD 42 7F F0 EB 27 18 59 D0",
              "signature algorithm": "SHA1withRSA",
              "issuer"             : "CN=SecureTrust CA, O=SecureTrust Corporation, C=US",
              "not before"         : "2006-11-07 19:31:18.000 GMT",
              "not  after"         : "2029-12-31 19:40:55.000 GMT",
              "subject"            : "CN=SecureTrust CA, O=SecureTrust Corporation, C=US",
              "subject public key" : "RSA",
          
          

          After a few attempts the following error is encountered with a generated certificate:

          PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

          Generated certificate

          javax.net.ssl|DEBUG|0B|Handling POST /jenkins/script from ***** : Jetty (winstone)-11|2022-03-09 10:23:02.359 GMT|CertificateMessage.java:1148|Consuming server Certificate handshake message (
          "Certificate": {
            "certificate_request_context": "",
            "certificate_list": [
            {
              "certificate" : {
                "version"            : "v3",
                "serial number"      : "18 8E 0D 54 D7 BE 37 77 8B 9D 45 45 D3 DB 2E 55",
                "signature algorithm": "SHA256withRSA",
                "issuer"             : "CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co",
                "not before"         : "2022-03-08 10:49:43.000 GMT",
                "not  after"         : "2023-03-08 10:49:43.000 GMT",
                "subject"            : "CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co",
                "subject public key" : "RSA",
                "extensions"         : [
                  {
                    ObjectId: 2.5.29.19 Criticality=true
                    BasicConstraints:[
                      CA:false
                      PathLen: undefined
                    ]
                  },
                  {
                    ObjectId: 2.5.29.37 Criticality=false
                    ExtendedKeyUsages [
                      serverAuth
                    ]
                  },
                  {
                    ObjectId: 2.5.29.15 Criticality=true
                    KeyUsage [
                      DigitalSignature
                      Key_Encipherment
                    ]
                  },
                  {
                    ObjectId: 2.5.29.17 Criticality=false
                    SubjectAlternativeName [
                      DNSName: ingress.local
                    ]
                  }
                ]}
              "extensions": {
                <no extension>
              }
            },
          ]
          }
          )
          javax.net.ssl|DEBUG|0B|Handling POST /jenkins/script from ***** : Jetty (winstone)-11|2022-03-09 10:23:02.359 GMT|SSLExtensions.java:148|Ignore unavailable extension: status_request
          javax.net.ssl|ERROR|0B|Handling POST /jenkins/script from ***** : Jetty (winstone)-11|2022-03-09 10:23:02.380 GMT|TransportContext.java:313|Fatal (CERTIFICATE_UNKNOWN): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (
          "throwable" : {
            sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          

          Marcus Collins added a comment - I have encountered the same problem with openjdk-11, but it is intermittent. It appears that the certificate returned from get.jenkins.io sometimes is a self-signed Kubernetes certificate. Steps Running Jenkins with SSL logging enabled: -Djavax.net.debug=ssl:handshake:verbose:keymanager:trustmanager Execute following from Jenkins script console: try { def url = 'https: //get.jenkins.io/' def connection = new URL(url).openConnection() as HttpURLConnection connection.with { println "ResponseCode: ${responseCode}" println "Headers: ${headerFields}" println "Body: ${inputStream.text}" } } catch (e) { println "$e" } Successful response javax.net.ssl|DEBUG|0B|Handling POST /jenkins/script from ***** : Jetty (winstone)-11|2022-03-09 10:19:48.822 GMT|X509TrustManagerImpl.java:79|adding as trusted certificates ( "certificate" : { "version" : "v3", "serial number" : "00 A6 8B 79 29 00 00 00 00 50 D0 91 F9", "signature algorithm": "SHA384withECDSA", "issuer" : "CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US", "not before" : "2012-12-18 15:25:36.000 GMT", "not after" : "2037-12-18 15:55:36.000 GMT", "subject" : "CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US", "subject public key" : "EC", "extensions" : [ { ObjectId: 2.5.29.19 Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] }, { ObjectId: 2.5.29.15 Criticality=true KeyUsage [ Key_CertSign Crl_Sign ] }, { ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: B7 63 E7 1A DD 8D E9 08 A6 55 83 A4 E0 6A 50 41 .c.......U...jPA 0010: 65 11 42 49 e.BI ] ] } ]}, "certificate" : { "version" : "v3", "serial number" : "0C F0 8E 5C 08 16 A5 AD 42 7F F0 EB 27 18 59 D0", "signature algorithm": "SHA1withRSA", "issuer" : "CN=SecureTrust CA, O=SecureTrust Corporation, C=US", "not before" : "2006-11-07 19:31:18.000 GMT", "not after" : "2029-12-31 19:40:55.000 GMT", "subject" : "CN=SecureTrust CA, O=SecureTrust Corporation, C=US", "subject public key" : "RSA", After a few attempts the following error is encountered with a generated certificate: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Generated certificate javax.net.ssl|DEBUG|0B|Handling POST /jenkins/script from ***** : Jetty (winstone)-11|2022-03-09 10:23:02.359 GMT|CertificateMessage.java:1148|Consuming server Certificate handshake message ( "Certificate": { "certificate_request_context": "", "certificate_list": [ { "certificate" : { "version" : "v3", "serial number" : "18 8E 0D 54 D7 BE 37 77 8B 9D 45 45 D3 DB 2E 55", "signature algorithm": "SHA256withRSA", "issuer" : "CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co", "not before" : "2022-03-08 10:49:43.000 GMT", "not after" : "2023-03-08 10:49:43.000 GMT", "subject" : "CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co", "subject public key" : "RSA", "extensions" : [ { ObjectId: 2.5.29.19 Criticality=true BasicConstraints:[ CA:false PathLen: undefined ] }, { ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth ] }, { ObjectId: 2.5.29.15 Criticality=true KeyUsage [ DigitalSignature Key_Encipherment ] }, { ObjectId: 2.5.29.17 Criticality=false SubjectAlternativeName [ DNSName: ingress.local ] } ]} "extensions": { <no extension> } }, ] } ) javax.net.ssl|DEBUG|0B|Handling POST /jenkins/script from ***** : Jetty (winstone)-11|2022-03-09 10:23:02.359 GMT|SSLExtensions.java:148|Ignore unavailable extension: status_request javax.net.ssl|ERROR|0B|Handling POST /jenkins/script from ***** : Jetty (winstone)-11|2022-03-09 10:23:02.380 GMT|TransportContext.java:313|Fatal (CERTIFICATE_UNKNOWN): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target ( "throwable" : { sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

          Hello, I have performed this months update and found symptoms of these issue again with different causes.

          They are both intermittent, which suggests it is a misconfigured server in a load balanced set that is serving up invalid SSL certificates.

          It affects both the following URLs, the first often redirects to the other:

           

          In the certificate chain the errors happen when an invalid R3 certificate is served.

          This shows a comparison of the invalid and the expected certificates:

           

          This is the invalid certificate's signer:

           

           

          Marcus Collins added a comment - Hello, I have performed this months update and found symptoms of these issue again with different causes. They are both intermittent, which suggests it is a misconfigured server in a load balanced set that is serving up invalid SSL certificates. It affects both the following URLs, the first often redirects to the other: https://updates.jenkins.io/ https://get.jenkins.io/   In the certificate chain the errors happen when an invalid R3 certificate is served. This shows a comparison of the invalid and the expected certificates:   This is the invalid certificate's signer:    

          Mark Waite added a comment -

          mbtc the first image that you include:

          contains a surprising message that seems inconsistent with the other content in the image.  It reports that the certificate is valid from 20/01/2021 to 30/.09/2024 and yet says that the certificate has expired or is not yet valid. The reported date range should make the certificate valid, yet your system reports that the certificate is not valid.

          Are you certain that you are using the most recent CA certificates, including the CA certificate update that was delivered in September 2021?

          Mark Waite added a comment - mbtc the first image that you include: contains a surprising message that seems inconsistent with the other content in the image.  It reports that the certificate is valid from 20/01/2021 to 30/.09/2024 and yet says that the certificate has expired or is not yet valid. The reported date range should make the certificate valid, yet your system reports that the certificate is not valid. Are you certain that you are using the most recent CA certificates, including the CA certificate update that was delivered in September 2021?

          markewaite Hello, to clarify the screen-shots from the first image:

          • The certificate on the left (signed by DST Root CA X3) is a certificate that has been received in the chain from a call to 'https://updates.jenkins.io/'.
          • The certificate on the right (signed by ISRG Root X1) is one from the CA bundle in the JDK-11 trust store, and is what we expect.

          The second screen-shot shows the chain of the error certificate and the invalid signer: 'DST Root CA X3' which is expired.

          • I think this invalid chain certificate is causing the error 'certificate has expired or is not yet valid'.

           


          In either case the invalid certificates are being captured from the remote update sites and do not match the valid certificates in the JDK cacerts store.

          For info these are the commands I used to get the remote certificates:

          openssl s_client -showcerts -connect updates.jenkins.io:443 -servername updates.jenkins.io
          openssl s_client -showcerts -connect get.jenkins.io:443 -servername get.jenkins.io

          Note, without the servername you always get the 'Kubernetes_Ingress_Controller_Fake_Certificate'.

          Marcus Collins added a comment - markewaite Hello, to clarify the screen-shots from the first image: The certificate on the left (signed by DST Root CA X3) is a certificate that has been received in the chain from a call to ' https://updates.jenkins.io/'. The certificate on the right (signed by ISRG Root X1) is one from the CA bundle in the JDK-11 trust store, and is what we expect. The second screen-shot shows the chain of the error certificate and the invalid signer: 'DST Root CA X3' which is expired. I think this invalid chain certificate is causing the error 'certificate has expired or is not yet valid'.   In either case the invalid certificates are being captured from the remote update sites and do not match the valid certificates in the JDK cacerts store. For info these are the commands I used to get the remote certificates: openssl s_client -showcerts -connect updates.jenkins.io:443 -servername updates.jenkins.io openssl s_client -showcerts -connect get.jenkins.io:443 -servername get.jenkins.io Note, without the servername you always get the 'Kubernetes_Ingress_Controller_Fake_Certificate'.

          Mark Waite added a comment -

          Moved to https://github.com/jenkins-infra/helpdesk/issues/3017 so that the infra team can investigate it with their knowledge of SSL and the configuration of https://updates.jenkins.io.

          Mark Waite added a comment - Moved to https://github.com/jenkins-infra/helpdesk/issues/3017 so that the infra team can investigate it with their knowledge of SSL and the configuration of https://updates.jenkins.io .

          Hello mbtc could you confirm if you still have the issue (ideally in https://github.com/jenkins-infra/helpdesk/issues/3017 as the infra team does not monitor JIRA issues)?

          We recently changed some Apache configurations elements in https://github.com/jenkins-infra/helpdesk/issues/3020 and it helped us to understand the issue you were facing:

          Revese proxies such as Apache or Nginx mainly use the SNI (the TCP hostname used during the TLS handshake) field of your request to pick a TLS configuration in their colelction of available virtualhosts.

          When the SNI field value differs from the HTTP host value, there are 2 cases:

          • Apache (case of https://updates.jenkins.io) stop the connection and prints an error in the server errror logs
          • Nginx prints a warning and ignore the Host header

          In your case, it seems that your SNI sometimes differs from HTTP and/or is emptied leading to the reverse-proxies serving you the certificates of the fallback.

          We did some configuration fixes in the case of Apache to improve these elements, but it indicates that there is an HTTP proxy (or a likewise appliance) between your HTTP client and our server, that tampers with the HTTP/2 frames.

          Damien Duportal added a comment - Hello mbtc could you confirm if you still have the issue (ideally in https://github.com/jenkins-infra/helpdesk/issues/3017 as the infra team does not monitor JIRA issues)? We recently changed some Apache configurations elements in https://github.com/jenkins-infra/helpdesk/issues/3020 and it helped us to understand the issue you were facing: Revese proxies such as Apache or Nginx mainly use the SNI (the TCP hostname used during the TLS handshake) field of your request to pick a TLS configuration in their colelction of available virtualhosts. When the SNI field value differs from the HTTP host value, there are 2 cases: Apache (case of https://updates.jenkins.io ) stop the connection and prints an error in the server errror logs Nginx prints a warning and ignore the Host header In your case, it seems that your SNI sometimes differs from HTTP and/or is emptied leading to the reverse-proxies serving you the certificates of the fallback. We did some configuration fixes in the case of Apache to improve these elements, but it indicates that there is an HTTP proxy (or a likewise appliance) between your HTTP client and our server, that tampers with the HTTP/2 frames.

          I have attempted to upgrade Jenkins again this month and was unable to do so before now.

          For info I still encountered the SSL errors from the update site, but I have switched to running Jenkins in a later JDK which has resolved the issue.

          I think perhaps that OpenJDK-11 may have been loosing header information on cached connections, as calls would succeed for some time then fail consistently. A restart of Jenkins would restore the connection and allow the completion of updates.

          Marcus Collins added a comment - I have attempted to upgrade Jenkins again this month and was unable to do so before now. For info I still encountered the SSL errors from the update site, but I have switched to running Jenkins in a later JDK which has resolved the issue. I think perhaps that OpenJDK-11 may have been loosing header information on cached connections, as calls would succeed for some time then fail consistently. A restart of Jenkins would restore the connection and allow the completion of updates.

          I know this is closed. Adding what problem I had.

          Same SSL errors. 

          As everyone's case is different. I had to import ZScalar (Proxy ) Root certs our company uses. Java was not trusting this. Now its working. Moral This is a good link everyone knows already stackoverflow.com/questions/24563694/… Also enable SSL login with -Djavax.net.debug=all in jenkins.xml for further debugging. This will clearly show what is the cert it is not trusting just before error message on jenkins error log.

          Naveen chandra sekhara added a comment - I know this is closed. Adding what problem I had. Same SSL errors.  As everyone's case is different. I had to import ZScalar (Proxy ) Root certs our company uses. Java was not trusting this. Now its working. Moral This is a good link everyone knows already stackoverflow.com/questions/24563694/… Also enable SSL login with -Djavax.net.debug=all in jenkins.xml for further debugging. This will clearly show what is the cert it is not trusting just before error message on jenkins error log.

            Unassigned Unassigned
            sagar_nachare Sagar
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: