Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40990

Jenkins 2.27 with remoting 3.0 causes Java 5/6 Maven builds to fail

      Seems the switching mechanism does not kick in again. On a local build all locks fine:

      [maven-jdk-test] $ /export/sbs/jenkins/home/tools/hudson.model.JDK/JDK_1.6.0_latest_/bin/java -cp /export/sbs/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven32-agent-1.8.1.jar:/export/sbs/jenkins/home/tools/hudson.tasks.Maven_MavenInstallation/maven-3.2.x/boot/plexus-classworlds-2.5.2.jar:/export/sbs/jenkins/home/tools/hudson.tasks.Maven_MavenInstallation/maven-3.2.x/conf/logging jenkins.maven3.agent.Maven32Main /export/sbs/jenkins/home/tools/hudson.tasks.Maven_MavenInstallation/maven-3.2.x /run/jenkins/war/WEB-INF/lib/remoting-3.2.jar /export/sbs/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven32-interceptor-1.8.1.jar /export/sbs/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-commons-1.8.1.jar 41978
      Exception in thread "main" java.lang.UnsupportedClassVersionError: hudson/remoting/Launcher : Unsupported major.minor version 51.0
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
      	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
      	at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
      	at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClassFromSelf(ClassRealm.java:401)
      	at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:42)
      	at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
      	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:254)
      	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
      	at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:143)
      	at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:74)
      ERROR: ================================================================================
      ERROR: Invalid project setup: Connection reset
      ERROR: [JENKINS-18403][JENKINS-28294] JDK 'JDK 1.6.0 (latest)' not supported to run Maven projects.
      ERROR: Maven projects have to be launched with a Java version greater or equal to the minimum version required by the master.
      ERROR: Use the Maven JDK Toolchains (plugin) to build your maven project with an older JDK.
      ERROR: Retrying with slave Java and setting compile/test properties to point to /export/sbs/jenkins/home/tools/hudson.model.JDK/JDK_1.6.0_latest_.
      ERROR: ================================================================================
      Established TCP socket on 34326
      [maven-jdk-test] $ /export0/sbs/jenkins-runtime-jdk1.8.0/jre/bin/java -cp /export/sbs/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven32-agent-1.8.1.jar:/export/sbs/jenkins/home/tools/hudson.tasks.Maven_MavenInstallation/maven-3.2.x/boot/plexus-classworlds-2.5.2.jar:/export/sbs/jenkins/home/tools/hudson.tasks.Maven_MavenInstallation/maven-3.2.x/conf/logging jenkins.maven3.agent.Maven32Main /export/sbs/jenkins/home/tools/hudson.tasks.Maven_MavenInstallation/maven-3.2.x /run/jenkins/war/WEB-INF/lib/remoting-3.2.jar /export/sbs/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven32-interceptor-1.8.1.jar /export/sbs/jenkins/home/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-commons-1.8.1.jar 34326
      <===[JENKINS REMOTING CAPACITY]===>channel started
      BUILD Starts
      

      but on a remote node this (same job) fails:

      Parsing POMs
      Established TCP socket on 34109
      maven32-agent.jar already up to date
      maven32-interceptor.jar already up to date
      maven3-interceptor-commons.jar already up to date
      [-maven-jdk-test] $ /export/build/jenkins-localfs/tools/hudson.model.JDK/JDK_1.6.0_latest_/bin/java -cp /export/build/jenkins-localfs/maven32-agent.jar:/export/build/jenkins-localfs/tools/hudson.tasks.Maven_MavenInstallation/maven-3.2.x/boot/plexus-classworlds-2.5.2.jar:/export/build/jenkins-localfs/tools/hudson.tasks.Maven_MavenInstallation/maven-3.2.x/conf/logging jenkins.maven3.agent.Maven32Main /export/build/jenkins-localfs/tools/hudson.tasks.Maven_MavenInstallation/maven-3.2.x /export/build/jenkins-localfs/slave.jar /export/build/jenkins-localfs/maven32-interceptor.jar /export/build/jenkins-localfs/maven3-interceptor-commons.jar 34109
      Exception in thread "main" java.lang.UnsupportedClassVersionError: hudson/remoting/Launcher : Unsupported major.minor version 51.0
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
      	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
      	at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
      	at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClassFromSelf(ClassRealm.java:401)
      	at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:42)
      	at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
      	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:254)
      	at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
      	at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:143)
      	at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:74)
      ERROR: Failed to parse POMs
      java.io.EOFException: unexpected stream termination
      	at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:365)
      	at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:310)
      	at hudson.slaves.Channels.forProcess(Channels.java:115)
      	at hudson.maven.AbstractMavenProcessFactory.newProcess(AbstractMavenProcessFactory.java:294)
      	at hudson.maven.ProcessCache.get(ProcessCache.java:236)
      	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:798)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
      	at hudson.model.Run.execute(Run.java:1728)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:544)
      	at hudson.model.ResourceController.execute(ResourceController.java:98)
      	at hudson.model.Executor.run(Executor.java:404)
      Finished: FAILURE
      

      Again I'm not sure if this is a remoting or maven plugin issue. I switched back to Jenkins 2.19.3 and all worked fine again.

      I now there is a workaround (Will insert the link asap) but this is manual fragile work on all affected Jobs (which is a lot here).

      With the old remoting (2.62) the caught exception that successfully triggers the switch is as follows:

      ERROR: Caught java.io.IOException: Remote call on Channel to Maven [/home/jenkins/ws/tools/hudson.model.JDK/jdk1.6.x/bin/java, -cp, /home/jenkins/ws/maven32-agent.jar:/home/jenkins/ws/tools/hudson.tasks.Maven_MavenInstallation/maven_3.2.x/boot/plexus-classworlds-2.5.2.jar:\home\jenkins\ws\tools\hudson.tasks.Maven_MavenInstallation\maven_3.2.x/conf/logging, jenkins.maven3.agent.Maven32Main, /home/jenkins/ws/tools/hudson.tasks.Maven_MavenInstallation/maven_3.2.x, /home/jenkins/ws/slave.jar, /home/jenkins/ws/maven32-interceptor.jar, /home/jenkins/ws/maven3-interceptor-commons.jar, 41311] failed
      java.io.IOException: Remote call on Channel to Maven [/home/jenkins/ws/tools/hudson.model.JDK/jdk1.6.x/bin/java, -cp, /home/jenkins/ws/maven32-agent.jar:/home/jenkins/ws/tools/hudson.tasks.Maven_MavenInstallation/maven_3.2.x/boot/plexus-classworlds-2.5.2.jar:\home\jenkins\ws\tools\hudson.tasks.Maven_MavenInstallation\maven_3.2.x/conf/logging, jenkins.maven3.agent.Maven32Main, /home/jenkins/ws/tools/hudson.tasks.Maven_MavenInstallation/maven_3.2.x, /home/jenkins/ws/slave.jar, /home/jenkins/ws/maven32-interceptor.jar, /home/jenkins/ws/maven3-interceptor-commons.jar, 41311] failed
      	at hudson.remoting.Channel.call(Channel.java:805)
      	at hudson.maven.AbstractMavenProcessFactory.newProcess(AbstractMavenProcessFactory.java:297)
      	at hudson.maven.ProcessCache.get(ProcessCache.java:236)
      	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:798)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
      	at hudson.model.Run.execute(Run.java:1720)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:544)
      	at hudson.model.ResourceController.execute(ResourceController.java:98)
      	at hudson.model.Executor.run(Executor.java:404)
      Caused by: java.lang.ClassFormatError: Failed to load hudson.maven.AbstractMavenProcessFactory$ConfigureOriginalJDK
      	at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:375)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:285)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:249)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:124)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1589)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1494)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1748)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:131)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      	at hudson.remoting.Request$2.run(Request.java:332)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      	at java.lang.Thread.run(Thread.java:662)
      	at ......remote call to Channel to Maven [/home/jenkins/ws/tools/hudson.model.JDK/jdk1.6.x/bin/java, -cp, /home/jenkins/ws/maven32-agent.jar:/home/jenkins/ws/tools/hudson.tasks.Maven_MavenInstallation/maven_3.2.x/boot/plexus-classworlds-2.5.2.jar:\home\jenkins\ws\tools\hudson.tasks.Maven_MavenInstallation\maven_3.2.x/conf/logging, jenkins.maven3.agent.Maven32Main, /home/jenkins/ws/tools/hudson.tasks.Maven_MavenInstallation/maven_3.2.x, /home/jenkins/ws/slave.jar, /home/jenkins/ws/maven32-interceptor.jar, /home/jenkins/ws/maven3-interceptor-commons.jar, 41311](Native Method)
      	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1433)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
      	at hudson.remoting.Channel.call(Channel.java:797)
      	... 8 more
      Caused by: java.lang.UnsupportedClassVersionError: hudson/maven/AbstractMavenProcessFactory$ConfigureOriginalJDK : Unsupported major.minor version 51.0
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:465)
      	at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:373)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:285)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:249)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:124)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1589)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1494)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1748)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:131)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      	at hudson.remoting.Request$2.run(Request.java:332)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      	at java.lang.Thread.run(Thread.java:662)
      

          [JENKINS-40990] Jenkins 2.27 with remoting 3.0 causes Java 5/6 Maven builds to fail

          oleg_nenashev thanks for the feedback.
          Basically I've explained it all above already, so I try to give it some more detail.

          I see 2 things here the one is the JDK which is required to run Jenkins master or the nodes and the JDK required to execute a job. This is very well solved for freestyle jobs, where I can even execute a JDK1.4 build with a recent Jenkins running on Java 8. But it is broken now with the maven job type (which I know is evil, not only for that reason). IMHO it is essential that the JDK that powers Jenkins in unrelated to the JDK that performs the build. We can build any old Visual Basic software but fail with a old Java maven build?

          The maven plugin requires JDK1.6 and 1.7 since a long time - before the change to remoting 3.0. This is the reason why the maven plugin switches the execution JDK to the platform JDK and points compiler etc. to the old JDK (see the linked code change in my comment above). This switch now does not work any more and must be repaired to function again. The main question is why the root cause exception UnsupportedClassVersionError is not visible any more to the caller of Channel.call(...), and how we can make this visible again. If this is not possible there should be a other solution,

          Please note that this issue will hit projects bound to Java 7 also, once Jenkins requires Java 8.

          Andreas Mandel added a comment - oleg_nenashev thanks for the feedback. Basically I've explained it all above already, so I try to give it some more detail. I see 2 things here the one is the JDK which is required to run Jenkins master or the nodes and the JDK required to execute a job. This is very well solved for freestyle jobs, where I can even execute a JDK1.4 build with a recent Jenkins running on Java 8. But it is broken now with the maven job type (which I know is evil, not only for that reason). IMHO it is essential that the JDK that powers Jenkins in unrelated to the JDK that performs the build. We can build any old Visual Basic software but fail with a old Java maven build? The maven plugin requires JDK1.6 and 1.7 since a long time - before the change to remoting 3.0. This is the reason why the maven plugin switches the execution JDK to the platform JDK and points compiler etc. to the old JDK (see the linked code change in my comment above). This switch now does not work any more and must be repaired to function again. The main question is why the root cause exception UnsupportedClassVersionError is not visible any more to the caller of Channel.call(...) , and how we can make this visible again. If this is not possible there should be a other solution, Please note that this issue will hit projects bound to Java 7 also, once Jenkins requires Java 8.

          Oleg Nenashev added a comment -

          andreasmandel

          I see no way how a fix could be applied on the remoting side. The Java 7 upgrade in the core has been announced long long ago, and actually it was a responsibility of (non-existent?) Maven Project Plugin maintainers to make it work accordingly if they wanted to retain the compatibility with Java 6. CC aheritier, who is listed as a plugin maintainer.

          Fix Approach 1: Compatibility layer in the Maven Project Plugin

          1. Maven plugin should include the Shaded version of remoting 2.x with isolated classloader & Co
          2. Maven plugin should be able to use this remoting version transparently in the entire Maven Interceptors logic

          Fix Approach 2: Complete rework of the Maven Project Plugin

          Workarounds: For Java 6 use native Maven build steps instead of the Maven plugin.

          Regarding the Java 8 migration concern, I have raised the topic here The good news is that I have no middle-term plans to deprecate Java 7 in Remoting.

          Oleg Nenashev added a comment - andreasmandel I see no way how a fix could be applied on the remoting side. The Java 7 upgrade in the core has been announced long long ago, and actually it was a responsibility of (non-existent?) Maven Project Plugin maintainers to make it work accordingly if they wanted to retain the compatibility with Java 6. CC aheritier , who is listed as a plugin maintainer. Fix Approach 1: Compatibility layer in the Maven Project Plugin Maven plugin should include the Shaded version of remoting 2.x with isolated classloader & Co Maven plugin should be able to use this remoting version transparently in the entire Maven Interceptors logic Fix Approach 2: Complete rework of the Maven Project Plugin Workarounds: For Java 6 use native Maven build steps instead of the Maven plugin. Regarding the Java 8 migration concern, I have raised the topic here The good news is that I have no middle-term plans to deprecate Java 7 in Remoting.

          Jesse Glick added a comment -

          There is a standing mitigation which just happened to get broken by accident. We should fix the breakage and we will be back where we always were. I do not think this blocks a Java 8 migration in any way.

          Jesse Glick added a comment - There is a standing mitigation which just happened to get broken by accident. We should fix the breakage and we will be back where we always were. I do not think this blocks a Java 8 migration in any way.

          Oleg Nenashev added a comment -

          aheritier From what I understood from your recent tweet, you are not going to support Maven Project plugin for Java versions older than the cores one. If yes, could you please clarify it here and maybe in the plugin's Wiki?

          Oleg Nenashev added a comment - aheritier From what I understood from your recent tweet, you are not going to support Maven Project plugin for Java versions older than the cores one. If yes, could you please clarify it here and maybe in the plugin's Wiki?

          It never worked as documented in https://wiki.jenkins-ci.org/display/JENKINS/Maven+Project+Plugin

          All improvements to make it more understable are welcomed

          Jenkins was "silently" changing the JDK selected by the customer to the one used to launch the agent (which I find completely crappy)

          This is this "silent" change which is now broken here...

          Arnaud Héritier added a comment - It never worked as documented in https://wiki.jenkins-ci.org/display/JENKINS/Maven+Project+Plugin All improvements to make it more understable are welcomed Jenkins was "silently" changing the JDK selected by the customer to the one used to launch the agent (which I find completely crappy) This is this "silent" change which is now broken here...

          Jesse Glick added a comment -

          Jenkins was "silently" changing the JDK selected by the customer to the one used to launch the agent

          IIRC it was not silent, and of course this was paired with setting Maven properties to achieve the same effect in most cases.

          Jesse Glick added a comment - Jenkins was "silently" changing the JDK selected by the customer to the one used to launch the agent IIRC it was not silent, and of course this was paired with setting Maven properties to achieve the same effect in most cases.

          Yes jglick "silently" because it was "hidden" in your build logs (at the beginning of your build thus often very far from your build failure) and a large part of users didn't took care of it and thus yes in most cases it was transparent by setting only the source/target (if your code isn't hitting a JDK compat issue, your build isn't hardcoding the source/target, ...)

          Arnaud Héritier added a comment - Yes jglick "silently" because it was "hidden" in your build logs (at the beginning of your build thus often very far from your build failure) and a large part of users didn't took care of it and thus yes in most cases it was transparent by setting only the source/target (if your code isn't hitting a JDK compat issue, your build isn't hardcoding the source/target, ...)

          Oleg Nenashev added a comment -

          Currently the plugin Wiki explicitly documents Java compatibility.

          I would suggest closing this ticket aheritier . Otherwise we will end up having Java 10+ queries here as well.

           

           

          Oleg Nenashev added a comment - Currently the plugin Wiki explicitly documents Java compatibility. I would suggest closing this ticket aheritier . Otherwise we will end up having Java 10+ queries here as well.    

          yes makes sense oleg_nenashev

          Here the problem is having the hack which is auto-switching the JVM configured by the project to the Agent JVM which is broken because of remoting Java version 

          It's not something we can fix easily.

          Arnaud Héritier added a comment - yes makes sense oleg_nenashev Here the problem is having the hack which is auto-switching the JVM configured by the project to the Agent JVM which is broken because of remoting Java version  It's not something we can fix easily.

          Basil Crow added a comment -

          I would suggest closing this ticket Arnaud Héritier . Otherwise we will end up having Java 10+ queries here as well.

          Hello from the future! I think the same problem is happening again in JENKINS-68878.

          Basil Crow added a comment - I would suggest closing this ticket Arnaud Héritier . Otherwise we will end up having Java 10+ queries here as well. Hello from the future! I think the same problem is happening again in JENKINS-68878 .

            Unassigned Unassigned
            andreasmandel Andreas Mandel
            Votes:
            7 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: