Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60979

Installation of Plugin fails on LibertyProfile due using the wrong SocketFactory

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Won't Fix
    • core
    • None
    • - Jenkins any version (broken for me for a few weeks now)
      - Liberty Profile any version
      - Java: adopt-openj9@1.11.0-6 or adopt@1.11.0-6

    Description

      When trying to install plugins, Jenkins fails with two errors.

       

      First of all, the connectivity test will throw this exception, which is ignored:

      Connectivity checkConnectivity check java.lang.RuntimeException: com.ibm.ws.ssl.protocol.LibertySSLSocketFactory at com.ibm.ws.kernel.boot.security.SSLSocketFactoryProxy.<init>(SSLSocketFactoryProxy.java:38) at java.base/java.lang.J9VMInternals.newInstanceImpl(Native Method) at java.base/java.lang.Class.newInstance(Class.java:2084) at java.base/javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:110) at java.base/javax.net.ssl.HttpsURLConnection.getDefaultSSLSocketFactory(HttpsURLConnection.java:335) at java.base/javax.net.ssl.HttpsURLConnection.<init>(HttpsURLConnection.java:292) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.<init>(HttpsURLConnectionImpl.java:100) at java.base/sun.net.www.protocol.https.Handler.openConnection(Handler.java:62) at java.base/java.net.URL.openConnection(URL.java:1123) at hudson.ProxyConfiguration.open(ProxyConfiguration.java:260) at hudson.FilePath.installIfNecessaryFrom(FilePath.java:857) at hudson.FilePath.installIfNecessaryFrom(FilePath.java:848) at hudson.tools.ZipExtractionInstaller.performInstallation(ZipExtractionInstaller.java:83) at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:69) at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:109) at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:206) at hudson.model.JDK.forNode(JDK.java:147) at hudson.model.AbstractProject.getEnvironment(AbstractProject.java:341) at hudson.scm.SubversionSCM.compareRemoteRevisionWith(SubversionSCM.java:1419) at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:400) at hudson.scm.SCM.poll(SCM.java:417) at hudson.model.AbstractProject._poll(AbstractProject.java:1390) at hudson.model.AbstractProject.poll(AbstractProject.java:1293) at jenkins.triggers.SCMTriggerItem$SCMTriggerItems$Bridge.poll(SCMTriggerItem.java:143) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:603) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:649) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)Caused: java.net.SocketException at java.base/javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:263) at java.base/javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:285) at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:444) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515) at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) at hudson.model.UpdateCenter$UpdateCenterConfiguration.testConnection(UpdateCenter.java:1302) at hudson.model.UpdateCenter$UpdateCenterConfiguration.checkUpdateCenter(UpdateCenter.java:1085) at hudson.model.UpdateCenter$ConnectionCheckJob.run(UpdateCenter.java:1534) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.base/java.lang.Thread.run(Thread.java:834)
      

       

      After this, plugins should get downloaded. This will also fail.

      java.net.SocketTimeoutException: Read timed outjava.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292) at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351) at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:754) at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1610) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515) at java.base/sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3094) at java.base/java.net.URLConnection.getHeaderFieldLong(URLConnection.java:636) at java.base/java.net.URLConnection.getContentLengthLong(URLConnection.java:508) at java.base/java.net.URLConnection.getContentLength(URLConnection.java:492) at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1161)Caused: java.net.SocketTimeoutException: Read timed out at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1969) at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1964) at java.base/java.security.AccessController.doPrivileged(AccessController.java:734) at java.base/sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1963) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1531) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515) at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1177)Caused: java.io.IOException: Failed to load http://updates.jenkins-ci.org/download/plugins/warnings/5.0.1/warnings.hpi to $JENKINS_HOME/plugins/warnings.jpi.tmp at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1184)Caused: java.io.IOException: Failed to download from http://updates.jenkins-ci.org/download/plugins/warnings/5.0.1/warnings.hpi (redirected to: http://mirror.serverion.com/jenkins/plugins/warnings/5.0.1/warnings.hpi) at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1218) at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1766) at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2037) at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1740) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.base/java.lang.Thread.run(Thread.java:834)
      

       

       

      Using curl on the same system with the same settings (URL, proxy, etc.) without liberty and java, the download works just fine.

       

      The [documentation for Liberty Profile|https://wiki.jenkins.io/display/JENKINS/Liberty+profile] does not mention such a problem. Please try your test setup with a proxy.

      Attachments

        Issue Links

          Activity

            bmarwell Ben M added a comment - - edited

            IBM has granted me to cite their findings - thanks!

             

            We have traced the modified thread context class loader to a Jenkins class that is explicitly setting a thread context class loader for some of its internal threads, but doing so to a class loader that's inappropriate for thread context operations in Liberty.

            The class in question is ClassLoaderSanityThreadFactory, which appears to have been added to Jenkins 2.105 as a fix to the following issue:https://issues.jenkins-ci.org/browse/JENKINS-49206

             […]

            The Jenkins thread factory mentioned above does an explicit set of the thread context class loader to its own class loader when it creates a new thread, as follows:

            @Override
            public Thread newThread(Runnabler) {
              Thread t = delegate.newThread(r); }}
              t.setContextClassLoader(ClassLoaderSanityThreadFactory.class.getClassLoader());
              return t;
            }

            Liberty's thread context management, however, does not intend that application class loaders (the class loaders that would load these classes) be used as a thread context class loader - rather, it has its own ThreadContextClassLoader class that is granted extra visibility to server-level classes intended to be available to thread context class loader operations. Class loads initiated by other class loader types (such as the AppClassLoader) are not able to access packages exported with threadContext visibility, and as in this case, the load fails.
              
             The solution would be for Jenkins to either provide a switchable implementation (providing a configuration setting or system property to avoid this thread context class loader swap) or to utilize more robust logic in deciding whether to change the thread context class loader - for example, rather than immediately switching the thread context class loader to the current class' loader, it could first check to see if the existing thread context class loader can successfully load it (if it can, then that loader is adequate).

             

            I traced the underlying issue back to the solution of another issue I had, where IBM JVM engineers found a difference between the CommonThreadPool-Implementations of Java 8 and Java 11. I shortened it a bit as well. I created threads using CompletableFuture.supplyAsync(), but I think the behaviour is the very same.

            By specifying the ManagedExecutorService, you are using Libertys EE DefaultContextService, which will propagate the application classloader context

            We (WAS L3: EEConcurrency) can confirm that ManagedExecutorService by default propagates the thread context class loader of the submitter thread to the task/action that is submitted to it. With CompletableFuture.supplyAsync(supplier, executor), the submitter is actually the JDK's CompletableFuture.supplyAsync implementation, which is understood to delegate to executor.execute from within the supplyAsync method, meaning that if the application invoked supplyAsync, its thread context class loader is put onto the thread that runs the Supplier action. This seems like a reasonable way to ensure the application's thread context class loader is used. When CompletableFuture.supplyAsync(supplier) is used without specifying any executor parameter, the supplier is documented, see https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#supplyAsync-java.util.function.Supplier-to run on ForkJoinPool.commonPool() which is provided by the JDK, which is outside the control of EE Concurrency, and outside the control of Liberty as well.  Questions as to what sort of context the JDK's common ForkJoinPool uses, including differences in their implementation between JDK levels, must be directed to the Java team. We cannot answer that for them."
              
             So this narrowed the problem down further and our java L3 have now been able to recreate the issue and also point out where and why the
             change in behaviour was made and why SDK 8 behaves differently to sdk 11 -
              
             "Ok, so the issue that's being reported is that with Java 8, the task threads in the common ForkJoinPool inherit the context class loader from the thread which calls ForkJoinPool.execute(). However, with Java 11 the context class loader is not inherited.

            I then looked at the changelog for java.util.concurrent.ForkJoinWorkerThread in the OpenJDK mercurial repos, which led me to this change:
              
                 https://bugs.openjdk.java.net/browse/JDK-8172726

             So this is an intentional change in Java 11 to avoid memory leaks caused by threads in the common ForkJoinPool retaining references to custom class loaders. However, the change has not been backported to Java 8. The net of this is that both Java 8 and Java 11 are working as intended.

            So, it looks like […] we have been able to narrow down the problem to a deliberate change of behaviour  in the java versions and this is considered now working as designed for this reason."

            Takeaway: Do not create threads unless you can set the ThreadContext using a container-provided way.

            Kudos to IBM for their findings and in-depth analysis!

            bmarwell Ben M added a comment - - edited IBM has granted me to cite their findings - thanks!   We have traced the modified thread context class loader to a Jenkins class that is explicitly setting a thread context class loader for some of its internal threads, but doing so to a class loader that's inappropriate for thread context operations in Liberty. The class in question is ClassLoaderSanityThreadFactory, which appears to have been added to Jenkins 2.105 as a fix to the following issue: https://issues.jenkins-ci.org/browse/JENKINS-49206   […] The Jenkins thread factory mentioned above does an explicit set of the thread context class loader to its own class loader when it creates a new thread, as follows: @Override public Thread newThread(Runnabler) { Thread t = delegate.newThread(r); }}  t.setContextClassLoader(ClassLoaderSanityThreadFactory. class. getClassLoader()); return t; } Liberty's thread context management, however, does not intend that application class loaders (the class loaders that would load these classes) be used as a thread context class loader - rather, it has its own ThreadContextClassLoader class that is granted extra visibility to server-level classes intended to be available to thread context class loader operations. Class loads initiated by other class loader types (such as the AppClassLoader) are not able to access packages exported with threadContext visibility, and as in this case, the load fails.     The solution would be for Jenkins to either provide a switchable implementation (providing a configuration setting or system property to avoid this thread context class loader swap) or to utilize more robust logic in deciding whether to change the thread context class loader - for example, rather than immediately switching the thread context class loader to the current class' loader, it could first check to see if the existing thread context class loader can successfully load it (if it can, then that loader is adequate).   I traced the underlying issue back to the solution of another issue I had, where IBM JVM engineers found a difference between the CommonThreadPool-Implementations of Java 8 and Java 11. I shortened it a bit as well. I created threads using CompletableFuture.supplyAsync() , but I think the behaviour is the very same. By specifying the ManagedExecutorService, you are using Libertys EE DefaultContextService, which will propagate the application classloader context We (WAS L3: EEConcurrency) can confirm that ManagedExecutorService by default propagates the thread context class loader of the submitter thread to the task/action that is submitted to it. With CompletableFuture.supplyAsync(supplier, executor), the submitter is actually the JDK's CompletableFuture.supplyAsync implementation, which is understood to delegate to executor.execute from within the supplyAsync method, meaning that if the application invoked supplyAsync, its thread context class loader is put onto the thread that runs the Supplier action. This seems like a reasonable way to ensure the application's thread context class loader is used. When CompletableFuture.supplyAsync(supplier) is used without specifying any executor parameter, the supplier is documented, see  https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html#supplyAsync-java.util.function.Supplier- to run on ForkJoinPool.commonPool() which is provided by the JDK, which is outside the control of EE Concurrency, and outside the control of Liberty as well.  Questions as to what sort of context the JDK's common ForkJoinPool uses, including differences in their implementation between JDK levels, must be directed to the Java team. We cannot answer that for them."     So this narrowed the problem down further and our java L3 have now been able to recreate the issue and also point out where and why the  change in behaviour was made and why SDK 8 behaves differently to sdk 11 -     "Ok, so the issue that's being reported is that with Java 8, the task threads in the common ForkJoinPool inherit the context class loader from the thread which calls ForkJoinPool.execute(). However, with Java 11 the context class loader is not inherited. I then looked at the changelog for java.util.concurrent.ForkJoinWorkerThread in the OpenJDK mercurial repos, which led me to this change:          https://bugs.openjdk.java.net/browse/JDK-8172726  So this is an intentional change in Java 11 to avoid memory leaks caused by threads in the common ForkJoinPool retaining references to custom class loaders. However, the change has not been backported to Java 8. The net of this is that both Java 8 and Java 11 are working as intended. So, it looks like […] we have been able to narrow down the problem to a deliberate change of behaviour  in the java versions and this is considered now working as designed for this reason." Takeaway: Do not create threads unless you can set the ThreadContext using a container-provided way. Kudos to IBM for their findings and in-depth analysis!
            bmarwell Ben M added a comment -

            stopalopa I think there is now everything you need. Can we start working on this issue at some point? Other users will also  be affected.

            bmarwell Ben M added a comment - stopalopa I think there is now everything you need. Can we start working on this issue at some point? Other users will also  be affected.

            Unassigning from Natasha since she was working on the plugin-installation-manager-tool and this appears to be the actual PluginManager in Jenkins proper. I've also updated the component to reflect the change. Ben, if you have a solution, feel free to make a PR!

            kwhetstone Kristin Whetstone added a comment - Unassigning from Natasha since she was working on the plugin-installation-manager-tool and this appears to be the actual PluginManager in Jenkins proper. I've also updated the component to reflect the change. Ben, if you have a solution, feel free to make a PR!
            bmarwell Ben M added a comment -

            kwhetstone is there any update? I do not know how to test this, otherwise I would surely create a PR (probably by reverting an older commit).

            bmarwell Ben M added a comment - kwhetstone  is there any update? I do not know how to test this, otherwise I would surely create a PR (probably by reverting an older commit).
            markewaite Mark Waite added a comment -

            The Jenkins project shifted efforts away from openJ9 a year ago or more. JDK 11 hotspot and JDK 17 hotspot for both available and verified with Jenkins on IBM ppc64le and on IBM s390x.

            We've been running Jenkins on Java 11 for many years in many locations and have not seen this report from any location other than openJ9. Closing as "Won't fix".

            markewaite Mark Waite added a comment - The Jenkins project shifted efforts away from openJ9 a year ago or more. JDK 11 hotspot and JDK 17 hotspot for both available and verified with Jenkins on IBM ppc64le and on IBM s390x. We've been running Jenkins on Java 11 for many years in many locations and have not seen this report from any location other than openJ9. Closing as "Won't fix".

            People

              Unassigned Unassigned
              bmarwell Ben M
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: