Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60979

Installation of Plugin fails on LibertyProfile due using the wrong SocketFactory

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Major Major
    • core
    • None
    • - Jenkins any version (broken for me for a few weeks now)
      - Liberty Profile any version
      - Java: adopt-openj9@1.11.0-6 or adopt@1.11.0-6

      When trying to install plugins, Jenkins fails with two errors.

       

      First of all, the connectivity test will throw this exception, which is ignored:

      Connectivity checkConnectivity check java.lang.RuntimeException: com.ibm.ws.ssl.protocol.LibertySSLSocketFactory at com.ibm.ws.kernel.boot.security.SSLSocketFactoryProxy.<init>(SSLSocketFactoryProxy.java:38) at java.base/java.lang.J9VMInternals.newInstanceImpl(Native Method) at java.base/java.lang.Class.newInstance(Class.java:2084) at java.base/javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:110) at java.base/javax.net.ssl.HttpsURLConnection.getDefaultSSLSocketFactory(HttpsURLConnection.java:335) at java.base/javax.net.ssl.HttpsURLConnection.<init>(HttpsURLConnection.java:292) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.<init>(HttpsURLConnectionImpl.java:100) at java.base/sun.net.www.protocol.https.Handler.openConnection(Handler.java:62) at java.base/java.net.URL.openConnection(URL.java:1123) at hudson.ProxyConfiguration.open(ProxyConfiguration.java:260) at hudson.FilePath.installIfNecessaryFrom(FilePath.java:857) at hudson.FilePath.installIfNecessaryFrom(FilePath.java:848) at hudson.tools.ZipExtractionInstaller.performInstallation(ZipExtractionInstaller.java:83) at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:69) at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:109) at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:206) at hudson.model.JDK.forNode(JDK.java:147) at hudson.model.AbstractProject.getEnvironment(AbstractProject.java:341) at hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:1419) at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:400) at hudson.scm.SCM.poll(SCM.java:417) at hudson.model.AbstractProject._poll(AbstractProject.java:1390) at hudson.model.AbstractProject.poll(AbstractProject.java:1293) at jenkins.triggers.SCMTriggerItem$SCMTriggerItems$Bridge.poll(SCMTriggerItem.java:143) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:603) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:649) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)Caused: java.net.SocketException at java.base/javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:263) at java.base/javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:285) at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:444) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515) at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) at hudson.model.UpdateCenter$UpdateCenterConfiguration.testConnection(UpdateCenter.java:1302) at hudson.model.UpdateCenter$UpdateCenterConfiguration.checkUpdateCenter(UpdateCenter.java:1085) at hudson.model.UpdateCenter$ConnectionCheckJob.run(UpdateCenter.java:1534) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.base/java.lang.Thread.run(Thread.java:834)
      

       

      After this, plugins should get downloaded. This will also fail.

      java.net.SocketTimeoutException: Read timed outjava.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292) at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351) at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:754) at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1610) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515) at java.base/sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3094) at java.base/java.net.URLConnection.getHeaderFieldLong(URLConnection.java:636) at java.base/java.net.URLConnection.getContentLengthLong(URLConnection.java:508) at java.base/java.net.URLConnection.getContentLength(URLConnection.java:492) at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1161)Caused: java.net.SocketTimeoutException: Read timed out at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1969) at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1964) at java.base/java.security.AccessController.doPrivileged(AccessController.java:734) at java.base/sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1963) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1531) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515) at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1177)Caused: java.io.IOException: Failed to load http://updates.jenkins-ci.org/download/plugins/warnings/5.0.1/warnings.hpi to $JENKINS_HOME/plugins/warnings.jpi.tmp at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1184)Caused: java.io.IOException: Failed to download from http://updates.jenkins-ci.org/download/plugins/warnings/5.0.1/warnings.hpi (redirected to: http://mirror.serverion.com/jenkins/plugins/warnings/5.0.1/warnings.hpi) at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1218) at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1766) at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2037) at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1740) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.base/java.lang.Thread.run(Thread.java:834)
      

       

       

      Using curl on the same system with the same settings (URL, proxy, etc.) without liberty and java, the download works just fine.

       

      The [documentation for Liberty Profile|https://wiki.jenkins.io/display/JENKINS/Liberty+profile] does not mention such a problem. Please try your test setup with a proxy.

          [JENKINS-60979] Installation of Plugin fails on LibertyProfile due using the wrong SocketFactory

          Ben M added a comment -

          This specific stack trace is from Jenkins v2.190.2.

          The offending line for the actual download is: InputStream in = con.getInputStream();.

          The offending lines for the connection tests are (line "int responseCode = … "):

                  private void testConnection(URL url) throws IOException {
                      try {
                          URLConnection connection = ProxyConfiguration.open(url);                if(connection instanceof HttpURLConnection) {
                              int responseCode = ((HttpURLConnection)connection).getResponseCode();
                              if(HttpURLConnection.HTTP_OK != responseCode) {
                                  throw new HttpRetryException("Invalid response code (" + responseCode + ") from URL: " + url, responseCode);
                              }
                          } else {
                              try (InputStream is = connection.getInputStream()) {
                                  IOUtils.copy(is, new NullOutputStream());
                              }
                          }
                      } catch (SSLHandshakeException e) {
                          if (e.getMessage().contains("PKIX path building failed"))
                             // fix up this crappy error message from JDK
                              throw new IOException("Failed to validate the SSL certificate of "+url,e);
                      }
                  }
          

          Ben M added a comment - This specific stack trace is from Jenkins v2.190.2. The offending line for the actual download is:  InputStream in = con.getInputStream(); . The offending lines for the connection tests are (line "int responseCode = … "): private void testConnection(URL url) throws IOException { try { URLConnection connection = ProxyConfiguration.open(url); if (connection instanceof HttpURLConnection) { int responseCode = ((HttpURLConnection)connection).getResponseCode(); if (HttpURLConnection.HTTP_OK != responseCode) { throw new HttpRetryException( "Invalid response code (" + responseCode + ") from URL: " + url, responseCode); } } else { try (InputStream is = connection.getInputStream()) { IOUtils.copy(is, new NullOutputStream()); } } } catch (SSLHandshakeException e) { if (e.getMessage().contains( "PKIX path building failed" )) // fix up this crappy error message from JDK throw new IOException( "Failed to validate the SSL certificate of " +url,e); } }

          Ben M added a comment - - edited

          This may be related to JDK11 adopt-openj9 and running on liberty profile when enabling the feature "transportsecurity-1.0".

           

          If I turn this feature off, the error from the connectivity check disappears.

           

          The second error seems to disappear when setting very high connectivity values.

          -Dhudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds=240
          -Dhudson.ProxyConfiguration.DEFAULT_CONNECT_TIMEOUT_MILLIS=1200000
          

          Ben M added a comment - - edited This may be related to JDK11 adopt-openj9 and running on liberty profile when enabling the feature "transportsecurity-1.0".   If I turn this feature off, the error from the connectivity check disappears.   The second error seems to disappear when setting very high connectivity values. -Dhudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds=240 -Dhudson.ProxyConfiguration.DEFAULT_CONNECT_TIMEOUT_MILLIS=1200000

          Were you trying to download the plugin using the plugin installation manger tool or is this happening when you're launching Jenkins? If the later, we should change the component to be against core since the `plugin-installation-manager-tool` is a separate thing and under core this would get more attention.

          Kristin Whetstone added a comment - Were you trying to download the plugin using the plugin installation manger tool or is this happening when you're launching Jenkins? If the later, we should change the component to be against core since the `plugin-installation-manager-tool` is a separate thing and under core this would get more attention.

          Ben M added a comment -

          Hello kwhetstone

          IBM found out that the underlying issue is https://issues.jenkins-ci.org/browse/JENKINS-49206. A new Thread is created using a different class loader, overwriting the container's class loader.

          I asked IBM if I can cite their findings to let you know more about the technical detail. I consider this confirmed now, it is broken by design and never should have worked on Java 8 either.

          Ben M added a comment - Hello kwhetstone IBM found out that the underlying issue is  https://issues.jenkins-ci.org/browse/JENKINS-49206 . A new Thread is created using a different class loader, overwriting the container's class loader. I asked IBM if I can cite their findings to let you know more about the technical detail. I consider this confirmed now, it is broken by design and never should have worked on Java 8 either.

          Ben M added a comment - - edited

          IBM has granted me to cite their findings - thanks!

           

          We have traced the modified thread context class loader to a Jenkins class that is explicitly setting a thread context class loader for some of its internal threads, but doing so to a class loader that's inappropriate for thread context operations in Liberty.

          The class in question is ClassLoaderSanityThreadFactory, which appears to have been added to Jenkins 2.105 as a fix to the following issue:https://issues.jenkins-ci.org/browse/JENKINS-49206

           […]

          The Jenkins thread factory mentioned above does an explicit set of the thread context class loader to its own class loader when it creates a new thread, as follows:

          @Override
          public Thread newThread(Runnabler) {
            Thread t = delegate.newThread(r); }}
            t.setContextClassLoader(ClassLoaderSanityThreadFactory.class.getClassLoader());
            return t;
          }

          Liberty's thread context management, however, does not intend that application class loaders (the class loaders that would load these classes) be used as a thread context class loader - rather, it has its own ThreadContextClassLoader class that is granted extra visibility to server-level classes intended to be available to thread context class loader operations. Class loads initiated by other class loader types (such as the AppClassLoader) are not able to access packages exported with threadContext visibility, and as in this case, the load fails.
            
           The solution would be for Jenkins to either provide a switchable implementation (providing a configuration setting or system property to avoid this thread context class loader swap) or to utilize more robust logic in deciding whether to change the thread context class loader - for example, rather than immediately switching the thread context class loader to the current class' loader, it could first check to see if the existing thread context class loader can successfully load it (if it can, then that loader is adequate).

           

          I traced the underlying issue back to the solution of another issue I had, where IBM JVM engineers found a difference between the CommonThreadPool-Implementations of Java 8 and Java 11. I shortened it a bit as well. I created threads using CompletableFuture.supplyAsync(), but I think the behaviour is the very same.

          By specifying the ManagedExecutorService, you are using Libertys EE DefaultContextService, which will propagate the application classloader context

          We (WAS L3: EEConcurrency) can confirm that ManagedExecutorService by default propagates the thread context class loader of the submitter thread to the task/action that is submitted to it. With CompletableFuture.supplyAsync(supplier, executor), the submitter is actually the JDK's CompletableFuture.supplyAsync implementation, which is understood to delegate to executor.execute from within the supplyAsync method, meaning that if the application invoked supplyAsync, its thread context class loader is put onto the thread that runs the Supplier action. This seems like a reasonable way to ensure the application's thread context class loader is used. When CompletableFuture.supplyAsync(supplier) is used without specifying any executor parameter, the supplier is documented, see https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#supplyAsync-java.util.function.Supplier-to run on ForkJoinPool.commonPool() which is provided by the JDK, which is outside the control of EE Concurrency, and outside the control of Liberty as well.  Questions as to what sort of context the JDK's common ForkJoinPool uses, including differences in their implementation between JDK levels, must be directed to the Java team. We cannot answer that for them."
            
           So this narrowed the problem down further and our java L3 have now been able to recreate the issue and also point out where and why the
           change in behaviour was made and why SDK 8 behaves differently to sdk 11 -
            
           "Ok, so the issue that's being reported is that with Java 8, the task threads in the common ForkJoinPool inherit the context class loader from the thread which calls ForkJoinPool.execute(). However, with Java 11 the context class loader is not inherited.

          I then looked at the changelog for java.util.concurrent.ForkJoinWorkerThread in the OpenJDK mercurial repos, which led me to this change:
            
               https://bugs.openjdk.java.net/browse/JDK-8172726

           So this is an intentional change in Java 11 to avoid memory leaks caused by threads in the common ForkJoinPool retaining references to custom class loaders. However, the change has not been backported to Java 8. The net of this is that both Java 8 and Java 11 are working as intended.

          So, it looks like […] we have been able to narrow down the problem to a deliberate change of behaviour  in the java versions and this is considered now working as designed for this reason."

          Takeaway: Do not create threads unless you can set the ThreadContext using a container-provided way.

          Kudos to IBM for their findings and in-depth analysis!

          Ben M added a comment - - edited IBM has granted me to cite their findings - thanks!   We have traced the modified thread context class loader to a Jenkins class that is explicitly setting a thread context class loader for some of its internal threads, but doing so to a class loader that's inappropriate for thread context operations in Liberty. The class in question is ClassLoaderSanityThreadFactory, which appears to have been added to Jenkins 2.105 as a fix to the following issue: https://issues.jenkins-ci.org/browse/JENKINS-49206   […] The Jenkins thread factory mentioned above does an explicit set of the thread context class loader to its own class loader when it creates a new thread, as follows: @Override public Thread newThread(Runnabler) { Thread t = delegate.newThread(r); }}  t.setContextClassLoader(ClassLoaderSanityThreadFactory. class. getClassLoader()); return t; } Liberty's thread context management, however, does not intend that application class loaders (the class loaders that would load these classes) be used as a thread context class loader - rather, it has its own ThreadContextClassLoader class that is granted extra visibility to server-level classes intended to be available to thread context class loader operations. Class loads initiated by other class loader types (such as the AppClassLoader) are not able to access packages exported with threadContext visibility, and as in this case, the load fails.     The solution would be for Jenkins to either provide a switchable implementation (providing a configuration setting or system property to avoid this thread context class loader swap) or to utilize more robust logic in deciding whether to change the thread context class loader - for example, rather than immediately switching the thread context class loader to the current class' loader, it could first check to see if the existing thread context class loader can successfully load it (if it can, then that loader is adequate).   I traced the underlying issue back to the solution of another issue I had, where IBM JVM engineers found a difference between the CommonThreadPool-Implementations of Java 8 and Java 11. I shortened it a bit as well. I created threads using CompletableFuture.supplyAsync() , but I think the behaviour is the very same. By specifying the ManagedExecutorService, you are using Libertys EE DefaultContextService, which will propagate the application classloader context We (WAS L3: EEConcurrency) can confirm that ManagedExecutorService by default propagates the thread context class loader of the submitter thread to the task/action that is submitted to it. With CompletableFuture.supplyAsync(supplier, executor), the submitter is actually the JDK's CompletableFuture.supplyAsync implementation, which is understood to delegate to executor.execute from within the supplyAsync method, meaning that if the application invoked supplyAsync, its thread context class loader is put onto the thread that runs the Supplier action. This seems like a reasonable way to ensure the application's thread context class loader is used. When CompletableFuture.supplyAsync(supplier) is used without specifying any executor parameter, the supplier is documented, see  https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#supplyAsync-java.util.function.Supplier- to run on ForkJoinPool.commonPool() which is provided by the JDK, which is outside the control of EE Concurrency, and outside the control of Liberty as well.  Questions as to what sort of context the JDK's common ForkJoinPool uses, including differences in their implementation between JDK levels, must be directed to the Java team. We cannot answer that for them."     So this narrowed the problem down further and our java L3 have now been able to recreate the issue and also point out where and why the  change in behaviour was made and why SDK 8 behaves differently to sdk 11 -     "Ok, so the issue that's being reported is that with Java 8, the task threads in the common ForkJoinPool inherit the context class loader from the thread which calls ForkJoinPool.execute(). However, with Java 11 the context class loader is not inherited. I then looked at the changelog for java.util.concurrent.ForkJoinWorkerThread in the OpenJDK mercurial repos, which led me to this change:          https://bugs.openjdk.java.net/browse/JDK-8172726  So this is an intentional change in Java 11 to avoid memory leaks caused by threads in the common ForkJoinPool retaining references to custom class loaders. However, the change has not been backported to Java 8. The net of this is that both Java 8 and Java 11 are working as intended. So, it looks like […] we have been able to narrow down the problem to a deliberate change of behaviour  in the java versions and this is considered now working as designed for this reason." Takeaway: Do not create threads unless you can set the ThreadContext using a container-provided way. Kudos to IBM for their findings and in-depth analysis!

          Ben M added a comment -

          stopalopa I think there is now everything you need. Can we start working on this issue at some point? Other users will also  be affected.

          Ben M added a comment - stopalopa I think there is now everything you need. Can we start working on this issue at some point? Other users will also  be affected.

          Unassigning from Natasha since she was working on the plugin-installation-manager-tool and this appears to be the actual PluginManager in Jenkins proper. I've also updated the component to reflect the change. Ben, if you have a solution, feel free to make a PR!

          Kristin Whetstone added a comment - Unassigning from Natasha since she was working on the plugin-installation-manager-tool and this appears to be the actual PluginManager in Jenkins proper. I've also updated the component to reflect the change. Ben, if you have a solution, feel free to make a PR!

          Ben M added a comment -

          kwhetstone is there any update? I do not know how to test this, otherwise I would surely create a PR (probably by reverting an older commit).

          Ben M added a comment - kwhetstone  is there any update? I do not know how to test this, otherwise I would surely create a PR (probably by reverting an older commit).

          Mark Waite added a comment -

          The Jenkins project shifted efforts away from openJ9 a year ago or more. JDK 11 hotspot and JDK 17 hotspot for both available and verified with Jenkins on IBM ppc64le and on IBM s390x.

          We've been running Jenkins on Java 11 for many years in many locations and have not seen this report from any location other than openJ9. Closing as "Won't fix".

          Mark Waite added a comment - The Jenkins project shifted efforts away from openJ9 a year ago or more. JDK 11 hotspot and JDK 17 hotspot for both available and verified with Jenkins on IBM ppc64le and on IBM s390x. We've been running Jenkins on Java 11 for many years in many locations and have not seen this report from any location other than openJ9. Closing as "Won't fix".

            Unassigned Unassigned
            bmarwell Ben M
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: