Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64381

api.appcenter.ms SSL certificate validation issue when uploading files

      It looks like last week (~Dec 2nd 2020), App Center has introduced some changes on their https://api.appcenter.ms endpoint configuration, which result in SSL server certificate validation issue when uploading files:

      ERROR: Build step failed with exception
      sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
      	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
      	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
      	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
      Caused: sun.security.validator.ValidatorException: PKIX path building failed
      	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
      	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
      	at sun.security.validator.Validator.validate(Validator.java:260)
      	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
      	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
      	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
      	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1479)
      Caused: javax.net.ssl.SSLHandshakeException
      

      After doing some debugging, it seems to be related to SNI and the fact that when "servername" is not specified explicitly while requesting the api, api.appcenter.ms presents a wrong (self signed Kubernetes Ingress) certificate.

      The following steps using openssl could be used to confirm that theory:

      • request without specifying the "servername" flag explicitly:
        openssl s_client -showcerts -connect api.appcenter.ms:443

        in the output ("fake" Kubernetes Ingress certificate):

        Server certificate
        subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
        issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
        ---
        No client certificate CA names sent
        ---
        SSL handshake has read 1391 bytes and written 421 bytes
        ---
        New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
        Server public key is 2048 bit
        Secure Renegotiation IS supported
        Compression: NONE
        Expansion: NONE
        SSL-Session:
            Protocol  : TLSv1.2
            Cipher    : ECDHE-RSA-AES128-GCM-SHA256
            Session-ID: 2D139F6A03B9FAF75A61FD854B03CB24CC646C25DBEC66D3808B2BE8E9512FCC
            Session-ID-ctx:
            Master-Key: A4765DD8043CDE8F44B3633E0DB2B78C32073059E232B7F6116622A9F0A0455E71F0AA4A2AC8507F9A346778F275A2AA
            Key-Arg   : None
            PSK identity: None
            PSK identity hint: None
            SRP username: None
            Start Time: 1607349341
            Timeout   : 300 (sec)
            Verify return code: 21 (unable to verify the first certificate)
        
      • request with the "servername" flag explicitly set:
        openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443

        result (correct certificate):

        Server certificate
        subject=/C=US/ST=WA/L=Redmond/O=Microsoft Corporation/CN=*.appcenter.ms
        issuer=/C=US/O=Microsoft Corporation/CN=Microsoft Azure TLS Issuing CA 06
        ---
        No client certificate CA names sent
        ---
        SSL handshake has read 4945 bytes and written 446 bytes
        ---
        New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
        Server public key is 2048 bit
        Secure Renegotiation IS supported
        Compression: NONE
        Expansion: NONE
        SSL-Session:
            Protocol  : TLSv1.2
            Cipher    : ECDHE-RSA-AES128-GCM-SHA256
            Session-ID: 91F7E6F2F13056FF416FB09864FB3331CCF63D43CC07E68B2C5682C897398E5F
            Session-ID-ctx:
            Master-Key: 03487026B3BAF22CA42FD4F037ACD7CA4C132A78767097633E7D0D2642D32EEB0C3D7470369AD569D361674A9893BA4F
            Key-Arg   : None
            PSK identity: None
            PSK identity hint: None
            SRP username: None
            Start Time: 1607349413
            Timeout   : 300 (sec)
            Verify return code: 19 (self signed certificate in certificate chain)
        

      Unfortunately there is no easy way to workaround it for now (I tried with adding the "fake" certificates to Jenkins cert store, but it looks like every time different "fake" certificate gets returned, so that doesn't work).

      I realize that it's not really App Center plugin bug, but more of a configuration oversight from App Center side, so I issued a support ticket for them as well. 

          [JENKINS-64381] api.appcenter.ms SSL certificate validation issue when uploading files

          Em Jot added a comment -

          Linking GitHub Issue created for App Center:

          https://github.com/microsoft/appcenter/issues/2070

          Em Jot added a comment - Linking GitHub Issue created for App Center: https://github.com/microsoft/appcenter/issues/2070

          Em Jot added a comment -

          Any updates/plans on that one mezpahlan?

          Em Jot added a comment - Any updates/plans on that one mezpahlan ?

          Mez Pahlan added a comment -

          Hi michalcrk thanks for chasing up on this. 

          As you've mentioned on the AppCenter GitHub ticket there wasn't any changes to this Jenkins plugin to do with how we work with SSL. In fact for most (if not all) cases we don't explicitly do anything to either support or not support it. What I can say is that it works on my home laptop, the installation at my place of work, and there haven't been any other similar reports of SSL issues. So at the moment I am inclined to believe this is something unique to your setup.

           

          I appreciate that isn't very reassuring from your point of view but at the moment I don't even know where to start . I can't reproduce this in any environment I have access to and I'm not aware of any changes around this area to double check them. I truly apologise for that I would love to get to the bottom of this if for now other reason but to improve my working knowledge of this stuff.

          What might help me is if you could let me know are you connected directly to the internet from the machine that you have Jenkins installed on or are you connecting via a company proxy / firewall that may have its own self signed certificate? 

           

          Mez Pahlan added a comment - Hi michalcrk  thanks for chasing up on this.  As you've mentioned on the AppCenter GitHub ticket there wasn't any changes to this Jenkins plugin to do with how we work with SSL. In fact for most (if not all) cases we don't explicitly do anything to either support or not support it. What I can say is that it works on my home laptop, the installation at my place of work, and there haven't been any other similar reports of SSL issues. So at the moment I am inclined to believe this is something unique to your setup.   I appreciate that isn't very reassuring from your point of view but at the moment I don't even know  where to start . I can't reproduce this in any environment I have access to and I'm not aware of any changes around this area to double check them. I truly apologise for that I would love to get to the bottom of this if for now other reason but to improve my working knowledge of this stuff. What might help me is if you could let me know are you connected directly to the internet from the machine that you have Jenkins installed on or are you connecting via a company proxy / firewall that may have its own self signed certificate?   

          Em Jot added a comment -

          Thank you for your reply mezpahlan.

          What might help me is if you could let me know are you connected directly to the internet from the machine that you have Jenkins installed on or are you connecting via a company proxy / firewall that may have its own self signed certificate?

          I'm connecting directly from Jenkins master to AppCenter, there is no proxy/firewall on the way. 

          What I can say is that it works on my home laptop, the installation at my place of work

          Have you tried with the openssl command step that I described in the original message (

          openssl s_client -showcerts -connect api.appcenter.ms:443

          ) maybe? 

          Now I'm thinking that it's maybe related to OS libraries? Does the plugin code use its own SSL dependencies or the OS ones?

          and there haven't been any other similar reports of SSL issues

          There is one more reported on the GH thread actually. Maybe I could ask about that person's setup to compare it to mine.

          Em Jot added a comment - Thank you for your reply mezpahlan . What might help me is if you could let me know are you connected directly to the internet from the machine that you have Jenkins installed on or are you connecting via a company proxy / firewall that may have its own self signed certificate? I'm connecting directly from Jenkins master to AppCenter, there is no proxy/firewall on the way.  What I can say is that it works on my home laptop, the installation at my place of work Have you tried with the openssl command step that I described in the original message ( openssl s_client -showcerts -connect api.appcenter.ms:443 ) maybe?  Now I'm thinking that it's maybe related to OS libraries? Does the plugin code use its own SSL dependencies or the OS ones? and there haven't been any other similar reports of SSL issues There is one more reported on the GH thread actually. Maybe I could ask about that person's setup to compare it to mine.

          Mez Pahlan added a comment -

          >Does the plugin code use its own SSL dependencies or the OS ones?

          You can check the dependencies of the plugin via the pom.xml. OkHttp is the HTTP client that the library uses and you can see how (very little) we customise it in  AppCenterServiceFactory.java.  The plugin is (eventually) using the OS SSL dependencies as far as I am aware.

          >Have you tried with the openssl command step that I described in the original message 

          $ openssl s_client -showcerts -connect api.appcenter.ms:443
          
          Server certificate
          subject=C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = *.appcenter.msissuer=C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 06---
          No client certificate CA names sent
          Peer signing digest: SHA256
          Peer signature type: RSA-PSS
          Server Temp Key: ECDH, P-256, 256 bits
          ---
          SSL handshake has read 5124 bytes and written 729 bytes
          Verification: OK
          ---
          New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
          Server public key is 2048 bit
          Secure Renegotiation IS NOT supported
          Compression: NONE
          Expansion: NONE
          No ALPN negotiated
          Early data was not sent
          Verify return code: 0 (ok)
          ---
          ---
          Post-Handshake New Session Ticket arrived:
          SSL-Session:
              Protocol  : TLSv1.3
              Cipher    : TLS_AES_256_GCM_SHA384
              Session-ID: 8A89D7F99D378A44276A91B30EA76C0B99F45AA07F59D990456121744E5DA9FA
              Session-ID-ctx: 
              Resumption PSK: E64250231E317CE2BBE0B8CC9361C25F6E6123CF031922D956B4F25A03A24041FDA3B8EA5744E6540651110A6A55DAFE
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              TLS session ticket lifetime hint: 600 (seconds)
              TLS session ticket:
              0000 - 68 28 65 08 b2 97 f3 1a-27 2a c4 21 85 95 ef a8   h(e.....'*.!....
              0010 - 0e 05 31 ed ec 71 54 a5-ed 94 96 06 21 9d 57 73   ..1..qT.....!.Ws    Start Time: 1610571904
              Timeout   : 7200 (sec)
              Verify return code: 0 (ok)
              Extended master secret: no
              Max Early Data: 0
          $ openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443
          
          Server certificate
          subject=C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = *.appcenter.msissuer=C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 06---
          No client certificate CA names sent
          Peer signing digest: SHA256
          Peer signature type: RSA-PSS
          Server Temp Key: ECDH, P-256, 256 bits
          ---
          SSL handshake has read 5124 bytes and written 729 bytes
          Verification: OK
          ---
          New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
          Server public key is 2048 bit
          Secure Renegotiation IS NOT supported
          Compression: NONE
          Expansion: NONE
          No ALPN negotiated
          Early data was not sent
          Verify return code: 0 (ok)
          ---
          ---
          Post-Handshake New Session Ticket arrived:
          SSL-Session:
              Protocol  : TLSv1.3
              Cipher    : TLS_AES_256_GCM_SHA384
              Session-ID: 7DC6211B0D3B5EAA47A44FAE397E041990C475605CC2794E48B56C643C9056F6
              Session-ID-ctx: 
              Resumption PSK: BE3A17FB31E78DFD2292A5BE954E92ED3A1699FD72F6C0F68A80B6EFC48E38309024E643AA52178AD081ACC16D36640F
              PSK identity: None
              PSK identity hint: None
              SRP username: None
              TLS session ticket lifetime hint: 600 (seconds)
              TLS session ticket:
              0000 - 08 21 84 08 67 f2 77 b7-f3 69 12 8a 60 7a 2b bb   .!..g.w..i..`z+.
              0010 - fd 1f fd 34 3c bc fb df-af cf ba a2 28 61 ca e0   ...4<.......(a..    Start Time: 1610572328
              Timeout   : 7200 (sec)
              Verify return code: 0 (ok)
              Extended master secret: no
              Max Early Data: 0
          
          $ openssl version
          
          OpenSSL 1.1.1f  31 Mar 2020
          
          $ uname -a
          
          Linux 5.8.0-36-generic #40~20.04.1-Ubuntu SMP Wed Jan 6 10:15:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 

          Given this I assume that there is something very specific to your setup that is causing this behaviour. Perhaps reaching out to that other user might be useful to compare against our three setups to narrow down the cause of this.

          Mez Pahlan added a comment - >Does the plugin code use its own SSL dependencies or the OS ones? You can check the dependencies of the plugin via the pom.xml . OkHttp is the HTTP client that the library uses and you can see how (very little) we customise it in  AppCenterServiceFactory.java .  The plugin is (eventually) using the OS SSL dependencies as far as I am aware. >Have you tried with the openssl command step that I described in the original message  $ openssl s_client -showcerts -connect api.appcenter.ms:443 Server certificate subject=C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = *.appcenter.msissuer=C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 06--- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 5124 bytes and written 729 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: 8A89D7F99D378A44276A91B30EA76C0B99F45AA07F59D990456121744E5DA9FA Session-ID-ctx: Resumption PSK: E64250231E317CE2BBE0B8CC9361C25F6E6123CF031922D956B4F25A03A24041FDA3B8EA5744E6540651110A6A55DAFE PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 600 (seconds) TLS session ticket: 0000 - 68 28 65 08 b2 97 f3 1a-27 2a c4 21 85 95 ef a8 h(e.....'*.!.... 0010 - 0e 05 31 ed ec 71 54 a5-ed 94 96 06 21 9d 57 73 ..1..qT.....!.Ws Start Time: 1610571904 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 $ openssl s_client -showcerts -servername api.appcenter.ms -connect api.appcenter.ms:443 Server certificate subject=C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = *.appcenter.msissuer=C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 06--- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 5124 bytes and written 729 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: 7DC6211B0D3B5EAA47A44FAE397E041990C475605CC2794E48B56C643C9056F6 Session-ID-ctx: Resumption PSK: BE3A17FB31E78DFD2292A5BE954E92ED3A1699FD72F6C0F68A80B6EFC48E38309024E643AA52178AD081ACC16D36640F PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 600 (seconds) TLS session ticket: 0000 - 08 21 84 08 67 f2 77 b7-f3 69 12 8a 60 7a 2b bb .!..g.w..i..`z+. 0010 - fd 1f fd 34 3c bc fb df-af cf ba a2 28 61 ca e0 ...4<.......(a.. Start Time: 1610572328 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 $ openssl version OpenSSL 1.1.1f 31 Mar 2020 $ uname -a Linux 5.8.0-36- generic #40~20.04.1-Ubuntu SMP Wed Jan 6 10:15:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Given this I assume that there is something very specific to your setup that is causing this behaviour. Perhaps reaching out to that other user might be useful to compare against our three setups to narrow down the cause of this.

          Em Jot added a comment -

          Thanks for your reply again mezpahlan.
          After many attempts, even though I managed to upgrade openssl to OpenSSL 1.1.1a , the plugin is still giving me the same error.

          In the OkHttp docs I found:

          OkHttp uses your platform's built-in TLS implementation. On Java platforms OkHttp also supports Conscrypt, which integrates BoringSSL with Java. OkHttp will use Conscrypt if it is the first security provider:

          since the "built-in TLS implementation" is not very precise (and it's apparently not libssl ), I gave up for now and decided to use Jenkins shell job step & App Center CLI for now.

          Thanks for looking into the issue anyway.

          Em Jot added a comment - Thanks for your reply again mezpahlan . After many attempts, even though I managed to upgrade openssl to OpenSSL 1.1.1a  , the plugin is still giving me the same error. In the OkHttp docs I found: OkHttp uses your platform's built-in TLS implementation. On Java platforms OkHttp also supports Conscrypt, which integrates BoringSSL with Java. OkHttp will use Conscrypt if it is the first security provider: since the "built-in TLS implementation" is not very precise (and it's apparently not libssl ), I gave up for now and decided to use Jenkins shell job step & App Center CLI for now. Thanks for looking into the issue anyway.

          Mez Pahlan added a comment -

          Thanks for letting me know. Sorry we couldn't find the reason for this. Please feel free to re-open this if you hear of any new information. I'll close this for now.

          Mez Pahlan added a comment - Thanks for letting me know. Sorry we couldn't find the reason for this. Please feel free to re-open this if you hear of any new information. I'll close this for now.

          Mez Pahlan added a comment -

          Postponed until we understand more and can reliably reproduce on my development machines.

          Mez Pahlan added a comment - Postponed until we understand more and can reliably reproduce on my development machines.

          Yevgeniy added a comment -

          Hi there,

          I also faced with this issue and found that issue occurs on machines with LibreSSL installed, for instance macOS, so maybe this will be helpfull with reproducing issue. Also, I could replicate it in Ubuntu with OpenSSL 1.0.2g installed too, but with OpenSSL 1.1.1 everything works great

          ylogachev@macbook projects % uname -a
          Darwin MacBook-Pro.local 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec  2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64 x86_64
          
          ylogachev@macbook projects % openssl version
          LibreSSL 2.8.3
          
          ylogachev@macbook projects % openssl s_client -showcerts -connect api.appcenter.ms:443
          CONNECTED(00000005)
          depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
          verify error:num=20:unable to get local issuer certificate
          verify return:1
          depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
          verify error:num=21:unable to verify the first certificate
          verify return:1
          ---
          Certificate chain
           0 s:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
             i:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate

          Yevgeniy added a comment - Hi there, I also faced with this issue and found that issue occurs on machines with LibreSSL installed, for instance macOS, so maybe this will be helpfull with reproducing issue. Also, I could replicate it in Ubuntu with OpenSSL 1.0.2g installed too, but with OpenSSL 1.1.1 everything works great ylogachev@macbook projects % uname -a Darwin MacBook-Pro.local 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec 2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64 x86_64 ylogachev@macbook projects % openssl version LibreSSL 2.8.3 ylogachev@macbook projects % openssl s_client -showcerts -connect api.appcenter.ms:443 CONNECTED(00000005) depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate verify error:num=20:unable to get local issuer certificate verify return :1 depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate verify error:num=21:unable to verify the first certificate verify return :1 --- Certificate chain 0 s:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate i:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate

          Mez Pahlan added a comment -

          Thans ylogachev for the added information. I think it is fair to say that this is beyond the scope of a simple fix within the plugin and it is system dependent. 

          Mez Pahlan added a comment - Thans ylogachev  for the added information. I think it is fair to say that this is beyond the scope of a simple fix within the plugin and it is system dependent. 

            mezpahlan Mez Pahlan
            michalcrk Em Jot
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: