Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6604

Possible race condition in RemoteClassLoader renders slave unusable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core
    • CentOS 5.3, Sun JDK 1.6.0_19 64-bit

      We are restarting hudson each Sunday afternoon to evade problems with memory leaks and have a couple of nightly builds that kick in at midnight. The scenario is that Hudson is fresh when multiple builds kick in, that is its remote class loader did not have a chance to read any classes yet. We have 3 executors defined. I suppose that the SCM poll action that is sent in many build procedures causes multiple requests to load classes for the SCM (we use slightly hacked version of CVS SCM). We are getting the following exception:
      java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"

      I have looked around on the web and found this (http://jira.codehaus.org/browse/JETTY-418) that lead me to believe that lack of synchronization while loading classes in remote class loader is the cause.

      Full stack trace:

      Started on May 24, 2010 12:00:54 AM
      FATAL: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
      hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
      	at hudson.FilePath.act(FilePath.java:743)
      	at hudson.FilePath.act(FilePath.java:729)
      	at com.syncron.hudson.cvs2.CVS2.isUpdatable(CVS2.java:813)
      	at com.syncron.hudson.cvs2.CVS2.pollChanges(CVS2.java:310)
      	at hudson.scm.SCM.poll(SCM.java:370)
      	at hudson.model.AbstractProject.poll(AbstractProject.java:1153)
      	at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:330)
      	at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:359)
      	at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:619)
      Caused by: java.io.IOException: Remote call on slave-81 failed
      	at hudson.remoting.Channel.call(Channel.java:560)
      	at hudson.FilePath.act(FilePath.java:736)
      	... 14 more
      Caused by: java.lang.LinkageError: loader (instance of  hudson/remoting/RemoteClassLoader): attempted  duplicate class definition for name: "hudson/model/ModelObject"
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
      	at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
      	at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      	at java.lang.Class.getDeclaredFields0(Native Method)
      	at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
      	at java.lang.Class.getDeclaredField(Class.java:1880)
      	at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610)
      	at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52)
      	at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
      	at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
      	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:178)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:98)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      	at hudson.remoting.Request$2.run(Request.java:270)
      	... 6 more
      Done. Took 63 ms
      No changes
      

      If we start single job manually after restart it executes properly. Any consecutive jobs will also run fine. However if we get that exception once, no other jobs that use the class mentioned in exception (pretty much all) will execute anymore until slave is restarted.

          [JENKINS-6604] Possible race condition in RemoteClassLoader renders slave unusable

          michal_grzejszczak created issue -
          michal_grzejszczak made changes -
          Description Original: We are restarting hudson each Sunday afternoon to evade problems with memory leaks and have a couple of nightly builds that kick in at midnight. The scenario is that Hudson is fresh when multiple builds kick in, that is its remote class loader did not have a chance to read any classes yet. We have 3 executors defined. I suppose that the SCM poll action that is sent in many build procedures causes multiple requests to load classes for the SCM (we use slightly hacked version of CVS SCM). We are getting the following exception:
          java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"

          I have looked around on the web and found this (http://jira.codehaus.org/browse/JETTY-418) that lead me to believe that lack of synchronization while loading classes in remote class loader is the cause.

          Full stack trace:
          {code}
          Started on May 24, 2010 12:00:54 AM
          FATAL: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
          hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
          at hudson.FilePath.act(FilePath.java:743)
          at hudson.FilePath.act(FilePath.java:729)
          at com.syncron.hudson.cvs2.CVS2.isUpdatable(CVS2.java:813)
          at com.syncron.hudson.cvs2.CVS2.pollChanges(CVS2.java:310)
          at hudson.scm.SCM.poll(SCM.java:370)
          at hudson.model.AbstractProject.poll(AbstractProject.java:1153)
          at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:330)
          at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:359)
          at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
          Caused by: java.io.IOException: Remote call on slave-81 failed
          at hudson.remoting.Channel.call(Channel.java:560)
          at hudson.FilePath.act(FilePath.java:736)
          ... 14 more
          Caused by: java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
          at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
          at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
          at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
          at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
          at java.lang.Class.getDeclaredFields0(Native Method)
          at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
          at java.lang.Class.getDeclaredField(Class.java:1880)
          at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610)
          at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52)
          at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
          at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
          at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
          at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
          at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
          at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
          at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:178)
          at hudson.remoting.UserRequest.perform(UserRequest.java:98)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:270)
          ... 6 more
          Done. Took 63 ms
          No changes
          {code}
          New: We are restarting hudson each Sunday afternoon to evade problems with memory leaks and have a couple of nightly builds that kick in at midnight. The scenario is that Hudson is fresh when multiple builds kick in, that is its remote class loader did not have a chance to read any classes yet. We have 3 executors defined. I suppose that the SCM poll action that is sent in many build procedures causes multiple requests to load classes for the SCM (we use slightly hacked version of CVS SCM). We are getting the following exception:
          java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"

          I have looked around on the web and found this (http://jira.codehaus.org/browse/JETTY-418) that lead me to believe that lack of synchronization while loading classes in remote class loader is the cause.

          Full stack trace:
          {code}
          Started on May 24, 2010 12:00:54 AM
          FATAL: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
          hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
          at hudson.FilePath.act(FilePath.java:743)
          at hudson.FilePath.act(FilePath.java:729)
          at com.syncron.hudson.cvs2.CVS2.isUpdatable(CVS2.java:813)
          at com.syncron.hudson.cvs2.CVS2.pollChanges(CVS2.java:310)
          at hudson.scm.SCM.poll(SCM.java:370)
          at hudson.model.AbstractProject.poll(AbstractProject.java:1153)
          at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:330)
          at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:359)
          at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
          Caused by: java.io.IOException: Remote call on slave-81 failed
          at hudson.remoting.Channel.call(Channel.java:560)
          at hudson.FilePath.act(FilePath.java:736)
          ... 14 more
          Caused by: java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
          at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
          at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
          at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
          at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
          at java.lang.Class.getDeclaredFields0(Native Method)
          at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
          at java.lang.Class.getDeclaredField(Class.java:1880)
          at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610)
          at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52)
          at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
          at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
          at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
          at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
          at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
          at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
          at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:178)
          at hudson.remoting.UserRequest.perform(UserRequest.java:98)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:270)
          ... 6 more
          Done. Took 63 ms
          No changes
          {code}

          If we start single job manually after restart it executes properly. Any consecutive jobs will also run fine. However if we get that exception once, no other jobs that use the class mentioned in exception (pretty much all) will execute anymore until slave is restarted.

          Albert So added a comment -

          We're seeing a similar issue. We're on Jenkins 1.424 which is using Remoting-2.4.
          It seems like this happened shortly after the slave was restarted and 2 builds started simultaneously.

          Here's our stacktrace with some names obfuscated:

          Started by timer
          Building remotely on rhel-XXXX-012
          hudson.util.IOException2: remote file operation failed: /home/XXXXX/hudson-ws/workspace/XXXXXX at hudson.remoting.Channel@234711c1:rhel-XXXX-012
          	at hudson.FilePath.act(FilePath.java:754)
          	at hudson.FilePath.act(FilePath.java:740)
          	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:737)
          	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:681)
          	at hudson.model.AbstractProject.checkout(AbstractProject.java:1193)
          	at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:566)
          	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:454)
          	at hudson.model.Run.run(Run.java:1376)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          	at hudson.model.ResourceController.execute(ResourceController.java:88)
          	at hudson.model.Executor.run(Executor.java:175)
          Caused by: java.io.IOException: Remote call on rhel-XXXX-012 failed
          	at hudson.remoting.Channel.call(Channel.java:677)
          	at hudson.FilePath.act(FilePath.java:747)
          	... 10 more
          Caused by: java.lang.LinkageError: loader (instance of  hudson/remoting/RemoteClassLoader): attempted  duplicate class definition for name: "hudson/model/Descriptor"
          	at java.lang.ClassLoader.defineClass1(Native Method)
          	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
          	at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
          	at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
          	at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
          	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
          	at java.lang.Class.getDeclaredFields0(Native Method)
          	at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
          	at java.lang.Class.getDeclaredField(Class.java:1880)
          	at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610)
          	at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52)
          	at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
          	at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
          	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547)
          	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
          	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
          	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
          	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
          	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
          	at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:98)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          	at hudson.remoting.Request$2.run(Request.java:287)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:662)
          Analyzing test results with LabeledTestResultGroupPublisher...
          error processing LabeledTestResultGroup [unit] : No test report files were found. Configuration error?
          hudson.AbortException: No test report files were found. Configuration error?
          	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:111)
          	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:87)
          	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1994)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          	at hudson.remoting.Request$2.run(Request.java:287)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:662)
          Notifying upstream projects of job completion
          Finished: FAILURE
          

          Albert So added a comment - We're seeing a similar issue. We're on Jenkins 1.424 which is using Remoting-2.4. It seems like this happened shortly after the slave was restarted and 2 builds started simultaneously. Here's our stacktrace with some names obfuscated: Started by timer Building remotely on rhel-XXXX-012 hudson.util.IOException2: remote file operation failed: /home/XXXXX/hudson-ws/workspace/XXXXXX at hudson.remoting.Channel@234711c1:rhel-XXXX-012 at hudson.FilePath.act(FilePath.java:754) at hudson.FilePath.act(FilePath.java:740) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:737) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:681) at hudson.model.AbstractProject.checkout(AbstractProject.java:1193) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:566) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:454) at hudson.model.Run.run(Run.java:1376) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:175) Caused by: java.io.IOException: Remote call on rhel-XXXX-012 failed at hudson.remoting.Channel.call(Channel.java:677) at hudson.FilePath.act(FilePath.java:747) ... 10 more Caused by: java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/Descriptor" at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.lang.ClassLoader.defineClass(ClassLoader.java:466) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2291) at java.lang.Class.getDeclaredField(Class.java:1880) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610) at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at hudson.remoting.UserRequest.deserialize(UserRequest.java:182) at hudson.remoting.UserRequest.perform(UserRequest.java:98) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:287) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Analyzing test results with LabeledTestResultGroupPublisher... error processing LabeledTestResultGroup [unit] : No test report files were found. Configuration error? hudson.AbortException: No test report files were found. Configuration error? at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:111) at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:87) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1994) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:287) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Notifying upstream projects of job completion Finished: FAILURE

          Simon Gamma added a comment -

          Same Issue here. Running Jenkins ver. 1.437

          Simon Gamma added a comment - Same Issue here. Running Jenkins ver. 1.437

          We started to see these issues today with EC2 build slaves, haven't ever seen errors like this before for a year we run this infrastructure. The only change happened before today is that some additional plugins were installed. We're still on Jenkins 1.419.

          Paul Sokolovsky added a comment - We started to see these issues today with EC2 build slaves, haven't ever seen errors like this before for a year we run this infrastructure. The only change happened before today is that some additional plugins were installed. We're still on Jenkins 1.419.

          Our proposed fix is in the Jenkins' "remoting" library, the code is available here https://code.launchpad.net/~linaro-infrastructure/jenkins-remoting/linaro-trunk . It's branched from 2.12 jenkins-remoting lib so that's something to take into consideration.
          We weren't able to reproduce this issue on cloned test instances but after deploying this fix, our production didn't have any more of these issues (for around 4 days now). We'll post again in a while if it doesn't happen again, and I encourage the same if someone try this fix of ours.

          Stevan Radakovic added a comment - Our proposed fix is in the Jenkins' "remoting" library, the code is available here https://code.launchpad.net/~linaro-infrastructure/jenkins-remoting/linaro-trunk . It's branched from 2.12 jenkins-remoting lib so that's something to take into consideration. We weren't able to reproduce this issue on cloned test instances but after deploying this fix, our production didn't have any more of these issues (for around 4 days now). We'll post again in a while if it doesn't happen again, and I encourage the same if someone try this fix of ours.

          Here's the patch for the jenkins-remoting 2.12 library which should deal with this bug.

          Stevan Radakovic added a comment - Here's the patch for the jenkins-remoting 2.12 library which should deal with this bug.
          Stevan Radakovic made changes -
          Attachment New: jenkins-6604-remoting-2.12.patch [ 22000 ]

          Erik Lovlie added a comment -

          Can someone explain why this bug is of "minor" priority?

          Comparing the consequence of this bug (dynamic slave start/stop is broken) with the cost of fixing it (applying the attached patch) I am taking the liberty of increasing the priority of this bug from "minor" to "critical".

          Erik Lovlie added a comment - Can someone explain why this bug is of "minor" priority? Comparing the consequence of this bug (dynamic slave start/stop is broken) with the cost of fixing it (applying the attached patch) I am taking the liberty of increasing the priority of this bug from "minor" to "critical".
          Erik Lovlie made changes -
          Priority Original: Minor [ 4 ] New: Critical [ 2 ]

            jglick Jesse Glick
            michal_grzejszczak michal_grzejszczak
            Votes:
            5 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: