Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64381

api.appcenter.ms SSL certificate validation issue when uploading files

      It looks like last week (~Dec 2nd 2020), App Center has introduced some changes on their https://api.appcenter.ms endpoint configuration, which result in SSL server certificate validation issue when uploading files:

      ERROR: Build step failed with exception
      sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
      	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
      	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
      	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
      Caused: sun.security.validator.ValidatorException: PKIX path building failed
      	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
      	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
      	at sun.security.validator.Validator.validate(Validator.java:260)
      	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
      	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
      	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
      	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1479)
      Caused: javax.net.ssl.SSLHandshakeException
      

      After doing some debugging, it seems to be related to SNI and the fact that when "servername" is not specified explicitly while requesting the api, api.appcenter.ms presents a wrong (self signed Kubernetes Ingress) certificate.

      The following steps using openssl could be used to confirm that theory:

      • request without specifying the "servername" flag explicitly:
        openssl s_client -showcerts -connect api.appcenter.ms:443

        in the output ("fake" Kubernetes Ingress certificate):

        Server certificate
        subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
        issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
        ---
        No client certificate CA names sent
        ---
        SSL handshake has read 1391 bytes and written 421 bytes
        ---
        New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
        Server public key is 2048 bit
        Secure Renegotiation IS supported
        Compression: NONE
        Expansion: NONE
        SSL-Session:
            Protocol  : TLSv1.2
            Cipher    : ECDHE-RSA-AES128-GCM-SHA256
            Session-ID: 2D139F6A03B9FAF75A61FD854B03CB24CC646C25DBEC66D3808B2BE8E9512FCC
            Session-ID-ctx:
            Master-Key: A4765DD8043CDE8F44B3633E0DB2B78C32073059E232B7F6116622A9F0A0455E71F0AA4A2AC8507F9A346778F275A2AA
            Key-Arg   : None
            PSK identity: None
            PSK identity hint: None
            SRP username: None
            Start Time: 1607349341
            Timeout   : 300 (sec)
            Verify return code: 21 (unable to verify the first certificate)
        
      • request with the "servername" flag explicitly set:
        openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443

        result (correct certificate):

        Server certificate
        subject=/C=US/ST=WA/L=Redmond/O=Microsoft Corporation/CN=*.appcenter.ms
        issuer=/C=US/O=Microsoft Corporation/CN=Microsoft Azure TLS Issuing CA 06
        ---
        No client certificate CA names sent
        ---
        SSL handshake has read 4945 bytes and written 446 bytes
        ---
        New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
        Server public key is 2048 bit
        Secure Renegotiation IS supported
        Compression: NONE
        Expansion: NONE
        SSL-Session:
            Protocol  : TLSv1.2
            Cipher    : ECDHE-RSA-AES128-GCM-SHA256
            Session-ID: 91F7E6F2F13056FF416FB09864FB3331CCF63D43CC07E68B2C5682C897398E5F
            Session-ID-ctx:
            Master-Key: 03487026B3BAF22CA42FD4F037ACD7CA4C132A78767097633E7D0D2642D32EEB0C3D7470369AD569D361674A9893BA4F
            Key-Arg   : None
            PSK identity: None
            PSK identity hint: None
            SRP username: None
            Start Time: 1607349413
            Timeout   : 300 (sec)
            Verify return code: 19 (self signed certificate in certificate chain)
        

      Unfortunately there is no easy way to workaround it for now (I tried with adding the "fake" certificates to Jenkins cert store, but it looks like every time different "fake" certificate gets returned, so that doesn't work).

      I realize that it's not really App Center plugin bug, but more of a configuration oversight from App Center side, so I issued a support ticket for them as well. 

          [JENKINS-64381] api.appcenter.ms SSL certificate validation issue when uploading files

          Em Jot created issue -
          Em Jot made changes -
          Description Original: It looks like last week (~2nd Dec 2020), App Center has introduced some changes on their https://api.appcenter.ms endpoint configuration, which result in SSL server certificate validation issue when uploading files:


          {code:java}
          ERROR: Build step failed with exception
          sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
          at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
          at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
          Caused: sun.security.validator.ValidatorException: PKIX path building failed
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
          at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
          at sun.security.validator.Validator.validate(Validator.java:260)
          at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
          at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
          at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
          at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1479)
          Caused: javax.net.ssl.SSLHandshakeException
          {code}

          After doing some debugging, it seems to be related to SNI and the fact that when "servername" is not specified explicitly when requesting the api, api.appcenter.ms presents a wrong (self signed Kubernetes Ingress) certificate.

          The following steps using openssl could be used to confirm that theory:

          - request *without* specifying the "servername" flag explicitly:
          {noformat}
          openssl s_client -showcerts -connect api.appcenter.ms:443{noformat}
          in the output ("fake" Kubernetes Ingress certificate):
          {code:java}
          Server certificate
          subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 1391 bytes and written 421 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 2D139F6A03B9FAF75A61FD854B03CB24CC646C25DBEC66D3808B2BE8E9512FCC
              Session-ID-ctx:
              Master-Key: A4765DD8043CDE8F44B3633E0DB2B78C32073059E232B7F6116622A9F0A0455E71F0AA4A2AC8507F9A346778F275A2AA
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349341
              Timeout : 300 (sec)
              Verify return code: 21 (unable to verify the first certificate)
          {code}
          - request *with* the "servername" flag explicitly set:
          {code:java}
          openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443{code}
          result (correct certificate):
          {code:java}
          Server certificate
          subject=/C=US/ST=WA/L=Redmond/O=Microsoft Corporation/CN=*.appcenter.ms
          issuer=/C=US/O=Microsoft Corporation/CN=Microsoft Azure TLS Issuing CA 06
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 4945 bytes and written 446 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 91F7E6F2F13056FF416FB09864FB3331CCF63D43CC07E68B2C5682C897398E5F
              Session-ID-ctx:
              Master-Key: 03487026B3BAF22CA42FD4F037ACD7CA4C132A78767097633E7D0D2642D32EEB0C3D7470369AD569D361674A9893BA4F
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349413
              Timeout : 300 (sec)
              Verify return code: 19 (self signed certificate in certificate chain)
          {code}

          Unfortunately there is no easy way to workaround it for now (I tried with adding the "fake" certificates to Jenkins cert store, but it looks like every time different "fake" certificate gets returned, so that doesn't work).

          I realize that it's not really App Center plugin bug, but more of a configuration oversight from App Center side, so I issued a support ticket for them as well). 
          New: It looks like last week (~Dec 2nd 2020), App Center has introduced some changes on their [https://api.appcenter.ms|https://api.appcenter.ms/] endpoint configuration, which result in SSL server certificate validation issue when uploading files:
          {code:java}
          ERROR: Build step failed with exception
          sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
          at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
          at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
          Caused: sun.security.validator.ValidatorException: PKIX path building failed
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
          at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
          at sun.security.validator.Validator.validate(Validator.java:260)
          at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
          at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
          at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
          at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1479)
          Caused: javax.net.ssl.SSLHandshakeException
          {code}
          After doing some debugging, it seems to be related to SNI and the fact that when "servername" is not specified explicitly when requesting the api, api.appcenter.ms presents a wrong (self signed Kubernetes Ingress) certificate.

          The following steps using openssl could be used to confirm that theory:
           - request *without* specifying the "servername" flag explicitly:
          {noformat}
          openssl s_client -showcerts -connect api.appcenter.ms:443{noformat}
          in the output ("fake" Kubernetes Ingress certificate):
          {code:java}
          Server certificate
          subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 1391 bytes and written 421 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 2D139F6A03B9FAF75A61FD854B03CB24CC646C25DBEC66D3808B2BE8E9512FCC
              Session-ID-ctx:
              Master-Key: A4765DD8043CDE8F44B3633E0DB2B78C32073059E232B7F6116622A9F0A0455E71F0AA4A2AC8507F9A346778F275A2AA
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349341
              Timeout : 300 (sec)
              Verify return code: 21 (unable to verify the first certificate)
          {code}

           - request *with* the "servername" flag explicitly set:
          {code:java}
          openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443{code}
          result (correct certificate):
          {code:java}
          Server certificate
          subject=/C=US/ST=WA/L=Redmond/O=Microsoft Corporation/CN=*.appcenter.ms
          issuer=/C=US/O=Microsoft Corporation/CN=Microsoft Azure TLS Issuing CA 06
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 4945 bytes and written 446 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 91F7E6F2F13056FF416FB09864FB3331CCF63D43CC07E68B2C5682C897398E5F
              Session-ID-ctx:
              Master-Key: 03487026B3BAF22CA42FD4F037ACD7CA4C132A78767097633E7D0D2642D32EEB0C3D7470369AD569D361674A9893BA4F
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349413
              Timeout : 300 (sec)
              Verify return code: 19 (self signed certificate in certificate chain)
          {code}

          Unfortunately there is no easy way to workaround it for now (I tried with adding the "fake" certificates to Jenkins cert store, but it looks like every time different "fake" certificate gets returned, so that doesn't work).

          I realize that it's not really App Center plugin bug, but more of a configuration oversight from App Center side, so I issued a support ticket for them as well). 
          Em Jot made changes -
          Description Original: It looks like last week (~Dec 2nd 2020), App Center has introduced some changes on their [https://api.appcenter.ms|https://api.appcenter.ms/] endpoint configuration, which result in SSL server certificate validation issue when uploading files:
          {code:java}
          ERROR: Build step failed with exception
          sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
          at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
          at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
          Caused: sun.security.validator.ValidatorException: PKIX path building failed
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
          at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
          at sun.security.validator.Validator.validate(Validator.java:260)
          at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
          at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
          at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
          at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1479)
          Caused: javax.net.ssl.SSLHandshakeException
          {code}
          After doing some debugging, it seems to be related to SNI and the fact that when "servername" is not specified explicitly when requesting the api, api.appcenter.ms presents a wrong (self signed Kubernetes Ingress) certificate.

          The following steps using openssl could be used to confirm that theory:
           - request *without* specifying the "servername" flag explicitly:
          {noformat}
          openssl s_client -showcerts -connect api.appcenter.ms:443{noformat}
          in the output ("fake" Kubernetes Ingress certificate):
          {code:java}
          Server certificate
          subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 1391 bytes and written 421 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 2D139F6A03B9FAF75A61FD854B03CB24CC646C25DBEC66D3808B2BE8E9512FCC
              Session-ID-ctx:
              Master-Key: A4765DD8043CDE8F44B3633E0DB2B78C32073059E232B7F6116622A9F0A0455E71F0AA4A2AC8507F9A346778F275A2AA
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349341
              Timeout : 300 (sec)
              Verify return code: 21 (unable to verify the first certificate)
          {code}

           - request *with* the "servername" flag explicitly set:
          {code:java}
          openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443{code}
          result (correct certificate):
          {code:java}
          Server certificate
          subject=/C=US/ST=WA/L=Redmond/O=Microsoft Corporation/CN=*.appcenter.ms
          issuer=/C=US/O=Microsoft Corporation/CN=Microsoft Azure TLS Issuing CA 06
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 4945 bytes and written 446 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 91F7E6F2F13056FF416FB09864FB3331CCF63D43CC07E68B2C5682C897398E5F
              Session-ID-ctx:
              Master-Key: 03487026B3BAF22CA42FD4F037ACD7CA4C132A78767097633E7D0D2642D32EEB0C3D7470369AD569D361674A9893BA4F
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349413
              Timeout : 300 (sec)
              Verify return code: 19 (self signed certificate in certificate chain)
          {code}

          Unfortunately there is no easy way to workaround it for now (I tried with adding the "fake" certificates to Jenkins cert store, but it looks like every time different "fake" certificate gets returned, so that doesn't work).

          I realize that it's not really App Center plugin bug, but more of a configuration oversight from App Center side, so I issued a support ticket for them as well). 
          New: It looks like last week (~Dec 2nd 2020), App Center has introduced some changes on their [https://api.appcenter.ms|https://api.appcenter.ms/] endpoint configuration, which result in SSL server certificate validation issue when uploading files:
          {code:java}
          ERROR: Build step failed with exception
          sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
          at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
          at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
          Caused: sun.security.validator.ValidatorException: PKIX path building failed
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
          at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
          at sun.security.validator.Validator.validate(Validator.java:260)
          at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
          at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
          at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
          at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1479)
          Caused: javax.net.ssl.SSLHandshakeException
          {code}
          After doing some debugging, it seems to be related to SNI and the fact that when "servername" is not specified explicitly while requesting the api, api.appcenter.ms presents a wrong (self signed Kubernetes Ingress) certificate.

          The following steps using openssl could be used to confirm that theory:
           - request *without* specifying the "servername" flag explicitly:
          {noformat}
          openssl s_client -showcerts -connect api.appcenter.ms:443{noformat}
          in the output ("fake" Kubernetes Ingress certificate):
          {code:java}
          Server certificate
          subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 1391 bytes and written 421 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 2D139F6A03B9FAF75A61FD854B03CB24CC646C25DBEC66D3808B2BE8E9512FCC
              Session-ID-ctx:
              Master-Key: A4765DD8043CDE8F44B3633E0DB2B78C32073059E232B7F6116622A9F0A0455E71F0AA4A2AC8507F9A346778F275A2AA
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349341
              Timeout : 300 (sec)
              Verify return code: 21 (unable to verify the first certificate)
          {code}

           - request *with* the "servername" flag explicitly set:
          {code:java}
          openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443{code}
          result (correct certificate):
          {code:java}
          Server certificate
          subject=/C=US/ST=WA/L=Redmond/O=Microsoft Corporation/CN=*.appcenter.ms
          issuer=/C=US/O=Microsoft Corporation/CN=Microsoft Azure TLS Issuing CA 06
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 4945 bytes and written 446 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 91F7E6F2F13056FF416FB09864FB3331CCF63D43CC07E68B2C5682C897398E5F
              Session-ID-ctx:
              Master-Key: 03487026B3BAF22CA42FD4F037ACD7CA4C132A78767097633E7D0D2642D32EEB0C3D7470369AD569D361674A9893BA4F
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349413
              Timeout : 300 (sec)
              Verify return code: 19 (self signed certificate in certificate chain)
          {code}

          Unfortunately there is no easy way to workaround it for now (I tried with adding the "fake" certificates to Jenkins cert store, but it looks like every time different "fake" certificate gets returned, so that doesn't work).

          I realize that it's not really App Center plugin bug, but more of a configuration oversight from App Center side, so I issued a support ticket for them as well). 
          Em Jot made changes -
          Description Original: It looks like last week (~Dec 2nd 2020), App Center has introduced some changes on their [https://api.appcenter.ms|https://api.appcenter.ms/] endpoint configuration, which result in SSL server certificate validation issue when uploading files:
          {code:java}
          ERROR: Build step failed with exception
          sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
          at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
          at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
          Caused: sun.security.validator.ValidatorException: PKIX path building failed
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
          at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
          at sun.security.validator.Validator.validate(Validator.java:260)
          at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
          at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
          at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
          at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1479)
          Caused: javax.net.ssl.SSLHandshakeException
          {code}
          After doing some debugging, it seems to be related to SNI and the fact that when "servername" is not specified explicitly while requesting the api, api.appcenter.ms presents a wrong (self signed Kubernetes Ingress) certificate.

          The following steps using openssl could be used to confirm that theory:
           - request *without* specifying the "servername" flag explicitly:
          {noformat}
          openssl s_client -showcerts -connect api.appcenter.ms:443{noformat}
          in the output ("fake" Kubernetes Ingress certificate):
          {code:java}
          Server certificate
          subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 1391 bytes and written 421 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 2D139F6A03B9FAF75A61FD854B03CB24CC646C25DBEC66D3808B2BE8E9512FCC
              Session-ID-ctx:
              Master-Key: A4765DD8043CDE8F44B3633E0DB2B78C32073059E232B7F6116622A9F0A0455E71F0AA4A2AC8507F9A346778F275A2AA
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349341
              Timeout : 300 (sec)
              Verify return code: 21 (unable to verify the first certificate)
          {code}

           - request *with* the "servername" flag explicitly set:
          {code:java}
          openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443{code}
          result (correct certificate):
          {code:java}
          Server certificate
          subject=/C=US/ST=WA/L=Redmond/O=Microsoft Corporation/CN=*.appcenter.ms
          issuer=/C=US/O=Microsoft Corporation/CN=Microsoft Azure TLS Issuing CA 06
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 4945 bytes and written 446 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 91F7E6F2F13056FF416FB09864FB3331CCF63D43CC07E68B2C5682C897398E5F
              Session-ID-ctx:
              Master-Key: 03487026B3BAF22CA42FD4F037ACD7CA4C132A78767097633E7D0D2642D32EEB0C3D7470369AD569D361674A9893BA4F
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349413
              Timeout : 300 (sec)
              Verify return code: 19 (self signed certificate in certificate chain)
          {code}

          Unfortunately there is no easy way to workaround it for now (I tried with adding the "fake" certificates to Jenkins cert store, but it looks like every time different "fake" certificate gets returned, so that doesn't work).

          I realize that it's not really App Center plugin bug, but more of a configuration oversight from App Center side, so I issued a support ticket for them as well). 
          New: It looks like last week (~Dec 2nd 2020), App Center has introduced some changes on their [https://api.appcenter.ms|https://api.appcenter.ms/] endpoint configuration, which result in SSL server certificate validation issue when uploading files:
          {code:java}
          ERROR: Build step failed with exception
          sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
          at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
          at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
          at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
          Caused: sun.security.validator.ValidatorException: PKIX path building failed
          at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
          at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
          at sun.security.validator.Validator.validate(Validator.java:260)
          at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
          at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
          at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
          at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1479)
          Caused: javax.net.ssl.SSLHandshakeException
          {code}
          After doing some debugging, it seems to be related to SNI and the fact that when "servername" is not specified explicitly while requesting the api, api.appcenter.ms presents a wrong (self signed Kubernetes Ingress) certificate.

          The following steps using openssl could be used to confirm that theory:
           - request *without* specifying the "servername" flag explicitly:
          {noformat}
          openssl s_client -showcerts -connect api.appcenter.ms:443{noformat}
          in the output ("fake" Kubernetes Ingress certificate):
          {code:java}
          Server certificate
          subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 1391 bytes and written 421 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 2D139F6A03B9FAF75A61FD854B03CB24CC646C25DBEC66D3808B2BE8E9512FCC
              Session-ID-ctx:
              Master-Key: A4765DD8043CDE8F44B3633E0DB2B78C32073059E232B7F6116622A9F0A0455E71F0AA4A2AC8507F9A346778F275A2AA
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349341
              Timeout : 300 (sec)
              Verify return code: 21 (unable to verify the first certificate)
          {code}

           - request *with* the "servername" flag explicitly set:
          {code:java}
          openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443{code}
          result (correct certificate):
          {code:java}
          Server certificate
          subject=/C=US/ST=WA/L=Redmond/O=Microsoft Corporation/CN=*.appcenter.ms
          issuer=/C=US/O=Microsoft Corporation/CN=Microsoft Azure TLS Issuing CA 06
          ---
          No client certificate CA names sent
          ---
          SSL handshake has read 4945 bytes and written 446 bytes
          ---
          New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
          Server public key is 2048 bit
          Secure Renegotiation IS supported
          Compression: NONE
          Expansion: NONE
          SSL-Session:
              Protocol : TLSv1.2
              Cipher : ECDHE-RSA-AES128-GCM-SHA256
              Session-ID: 91F7E6F2F13056FF416FB09864FB3331CCF63D43CC07E68B2C5682C897398E5F
              Session-ID-ctx:
              Master-Key: 03487026B3BAF22CA42FD4F037ACD7CA4C132A78767097633E7D0D2642D32EEB0C3D7470369AD569D361674A9893BA4F
              Key-Arg : None
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              Start Time: 1607349413
              Timeout : 300 (sec)
              Verify return code: 19 (self signed certificate in certificate chain)
          {code}

          Unfortunately there is no easy way to workaround it for now (I tried with adding the "fake" certificates to Jenkins cert store, but it looks like every time different "fake" certificate gets returned, so that doesn't work).

          I realize that it's not really App Center plugin bug, but more of a configuration oversight from App Center side, so I issued a support ticket for them as well. 

          Em Jot added a comment -

          Linking GitHub Issue created for App Center:

          https://github.com/microsoft/appcenter/issues/2070

          Em Jot added a comment - Linking GitHub Issue created for App Center: https://github.com/microsoft/appcenter/issues/2070

          Em Jot added a comment -

          Any updates/plans on that one mezpahlan?

          Em Jot added a comment - Any updates/plans on that one mezpahlan ?
          Em Jot made changes -
          Issue Type Original: Improvement [ 4 ] New: Bug [ 1 ]

          Mez Pahlan added a comment -

          Hi michalcrk thanks for chasing up on this. 

          As you've mentioned on the AppCenter GitHub ticket there wasn't any changes to this Jenkins plugin to do with how we work with SSL. In fact for most (if not all) cases we don't explicitly do anything to either support or not support it. What I can say is that it works on my home laptop, the installation at my place of work, and there haven't been any other similar reports of SSL issues. So at the moment I am inclined to believe this is something unique to your setup.

           

          I appreciate that isn't very reassuring from your point of view but at the moment I don't even know where to start . I can't reproduce this in any environment I have access to and I'm not aware of any changes around this area to double check them. I truly apologise for that I would love to get to the bottom of this if for now other reason but to improve my working knowledge of this stuff.

          What might help me is if you could let me know are you connected directly to the internet from the machine that you have Jenkins installed on or are you connecting via a company proxy / firewall that may have its own self signed certificate? 

           

          Mez Pahlan added a comment - Hi michalcrk  thanks for chasing up on this.  As you've mentioned on the AppCenter GitHub ticket there wasn't any changes to this Jenkins plugin to do with how we work with SSL. In fact for most (if not all) cases we don't explicitly do anything to either support or not support it. What I can say is that it works on my home laptop, the installation at my place of work, and there haven't been any other similar reports of SSL issues. So at the moment I am inclined to believe this is something unique to your setup.   I appreciate that isn't very reassuring from your point of view but at the moment I don't even know  where to start . I can't reproduce this in any environment I have access to and I'm not aware of any changes around this area to double check them. I truly apologise for that I would love to get to the bottom of this if for now other reason but to improve my working knowledge of this stuff. What might help me is if you could let me know are you connected directly to the internet from the machine that you have Jenkins installed on or are you connecting via a company proxy / firewall that may have its own self signed certificate?   

          Em Jot added a comment -

          Thank you for your reply mezpahlan.

          What might help me is if you could let me know are you connected directly to the internet from the machine that you have Jenkins installed on or are you connecting via a company proxy / firewall that may have its own self signed certificate?

          I'm connecting directly from Jenkins master to AppCenter, there is no proxy/firewall on the way. 

          What I can say is that it works on my home laptop, the installation at my place of work

          Have you tried with the openssl command step that I described in the original message (

          openssl s_client -showcerts -connect api.appcenter.ms:443

          ) maybe? 

          Now I'm thinking that it's maybe related to OS libraries? Does the plugin code use its own SSL dependencies or the OS ones?

          and there haven't been any other similar reports of SSL issues

          There is one more reported on the GH thread actually. Maybe I could ask about that person's setup to compare it to mine.

          Em Jot added a comment - Thank you for your reply mezpahlan . What might help me is if you could let me know are you connected directly to the internet from the machine that you have Jenkins installed on or are you connecting via a company proxy / firewall that may have its own self signed certificate? I'm connecting directly from Jenkins master to AppCenter, there is no proxy/firewall on the way.  What I can say is that it works on my home laptop, the installation at my place of work Have you tried with the openssl command step that I described in the original message ( openssl s_client -showcerts -connect api.appcenter.ms:443 ) maybe?  Now I'm thinking that it's maybe related to OS libraries? Does the plugin code use its own SSL dependencies or the OS ones? and there haven't been any other similar reports of SSL issues There is one more reported on the GH thread actually. Maybe I could ask about that person's setup to compare it to mine.

          Mez Pahlan added a comment -

          >Does the plugin code use its own SSL dependencies or the OS ones?

          You can check the dependencies of the plugin via the pom.xml. OkHttp is the HTTP client that the library uses and you can see how (very little) we customise it in  AppCenterServiceFactory.java.  The plugin is (eventually) using the OS SSL dependencies as far as I am aware.

          >Have you tried with the openssl command step that I described in the original message 

          $ openssl s_client -showcerts -connect api.appcenter.ms:443
          
          Server certificate
          subject=C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = *.appcenter.msissuer=C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 06---
          No client certificate CA names sent
          Peer signing digest: SHA256
          Peer signature type: RSA-PSS
          Server Temp Key: ECDH, P-256, 256 bits
          ---
          SSL handshake has read 5124 bytes and written 729 bytes
          Verification: OK
          ---
          New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
          Server public key is 2048 bit
          Secure Renegotiation IS NOT supported
          Compression: NONE
          Expansion: NONE
          No ALPN negotiated
          Early data was not sent
          Verify return code: 0 (ok)
          ---
          ---
          Post-Handshake New Session Ticket arrived:
          SSL-Session:
              Protocol  : TLSv1.3
              Cipher    : TLS_AES_256_GCM_SHA384
              Session-ID: 8A89D7F99D378A44276A91B30EA76C0B99F45AA07F59D990456121744E5DA9FA
              Session-ID-ctx: 
              Resumption PSK: E64250231E317CE2BBE0B8CC9361C25F6E6123CF031922D956B4F25A03A24041FDA3B8EA5744E6540651110A6A55DAFE
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              TLS session ticket lifetime hint: 600 (seconds)
              TLS session ticket:
              0000 - 68 28 65 08 b2 97 f3 1a-27 2a c4 21 85 95 ef a8   h(e.....'*.!....
              0010 - 0e 05 31 ed ec 71 54 a5-ed 94 96 06 21 9d 57 73   ..1..qT.....!.Ws    Start Time: 1610571904
              Timeout   : 7200 (sec)
              Verify return code: 0 (ok)
              Extended master secret: no
              Max Early Data: 0
          $ openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443
          
          Server certificate
          subject=C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = *.appcenter.msissuer=C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 06---
          No client certificate CA names sent
          Peer signing digest: SHA256
          Peer signature type: RSA-PSS
          Server Temp Key: ECDH, P-256, 256 bits
          ---
          SSL handshake has read 5124 bytes and written 729 bytes
          Verification: OK
          ---
          New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
          Server public key is 2048 bit
          Secure Renegotiation IS NOT supported
          Compression: NONE
          Expansion: NONE
          No ALPN negotiated
          Early data was not sent
          Verify return code: 0 (ok)
          ---
          ---
          Post-Handshake New Session Ticket arrived:
          SSL-Session:
              Protocol  : TLSv1.3
              Cipher    : TLS_AES_256_GCM_SHA384
              Session-ID: 7DC6211B0D3B5EAA47A44FAE397E041990C475605CC2794E48B56C643C9056F6
              Session-ID-ctx: 
              Resumption PSK: BE3A17FB31E78DFD2292A5BE954E92ED3A1699FD72F6C0F68A80B6EFC48E38309024E643AA52178AD081ACC16D36640F
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              TLS session ticket lifetime hint: 600 (seconds)
              TLS session ticket:
              0000 - 08 21 84 08 67 f2 77 b7-f3 69 12 8a 60 7a 2b bb   .!..g.w..i..`z+.
              0010 - fd 1f fd 34 3c bc fb df-af cf ba a2 28 61 ca e0   ...4<.......(a..    Start Time: 1610572328
              Timeout   : 7200 (sec)
              Verify return code: 0 (ok)
              Extended master secret: no
              Max Early Data: 0
          
          $ openssl version
          
          OpenSSL 1.1.1f  31 Mar 2020
          
          $ uname -a
          
          Linux 5.8.0-36-generic #40~20.04.1-Ubuntu SMP Wed Jan 6 10:15:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 

          Given this I assume that there is something very specific to your setup that is causing this behaviour. Perhaps reaching out to that other user might be useful to compare against our three setups to narrow down the cause of this.

          Mez Pahlan added a comment - >Does the plugin code use its own SSL dependencies or the OS ones? You can check the dependencies of the plugin via the pom.xml . OkHttp is the HTTP client that the library uses and you can see how (very little) we customise it in  AppCenterServiceFactory.java .  The plugin is (eventually) using the OS SSL dependencies as far as I am aware. >Have you tried with the openssl command step that I described in the original message  $ openssl s_client -showcerts -connect api.appcenter.ms:443 Server certificate subject=C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = *.appcenter.msissuer=C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 06--- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 5124 bytes and written 729 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: 8A89D7F99D378A44276A91B30EA76C0B99F45AA07F59D990456121744E5DA9FA Session-ID-ctx: Resumption PSK: E64250231E317CE2BBE0B8CC9361C25F6E6123CF031922D956B4F25A03A24041FDA3B8EA5744E6540651110A6A55DAFE PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 600 (seconds) TLS session ticket: 0000 - 68 28 65 08 b2 97 f3 1a-27 2a c4 21 85 95 ef a8 h(e.....'*.!.... 0010 - 0e 05 31 ed ec 71 54 a5-ed 94 96 06 21 9d 57 73 ..1..qT.....!.Ws Start Time: 1610571904 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 $ openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443 Server certificate subject=C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = *.appcenter.msissuer=C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 06--- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 5124 bytes and written 729 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: 7DC6211B0D3B5EAA47A44FAE397E041990C475605CC2794E48B56C643C9056F6 Session-ID-ctx: Resumption PSK: BE3A17FB31E78DFD2292A5BE954E92ED3A1699FD72F6C0F68A80B6EFC48E38309024E643AA52178AD081ACC16D36640F PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 600 (seconds) TLS session ticket: 0000 - 08 21 84 08 67 f2 77 b7-f3 69 12 8a 60 7a 2b bb .!..g.w..i..`z+. 0010 - fd 1f fd 34 3c bc fb df-af cf ba a2 28 61 ca e0 ...4<.......(a.. Start Time: 1610572328 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 $ openssl version OpenSSL 1.1.1f 31 Mar 2020 $ uname -a Linux 5.8.0-36- generic #40~20.04.1-Ubuntu SMP Wed Jan 6 10:15:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Given this I assume that there is something very specific to your setup that is causing this behaviour. Perhaps reaching out to that other user might be useful to compare against our three setups to narrow down the cause of this.

            mezpahlan Mez Pahlan
            michalcrk Em Jot
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: