Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6604

Possible race condition in RemoteClassLoader renders slave unusable

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      CentOS 5.3, Sun JDK 1.6.0_19 64-bit
    • Similar Issues:

      Description

      We are restarting hudson each Sunday afternoon to evade problems with memory leaks and have a couple of nightly builds that kick in at midnight. The scenario is that Hudson is fresh when multiple builds kick in, that is its remote class loader did not have a chance to read any classes yet. We have 3 executors defined. I suppose that the SCM poll action that is sent in many build procedures causes multiple requests to load classes for the SCM (we use slightly hacked version of CVS SCM). We are getting the following exception:
      java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"

      I have looked around on the web and found this (http://jira.codehaus.org/browse/JETTY-418) that lead me to believe that lack of synchronization while loading classes in remote class loader is the cause.

      Full stack trace:

      Started on May 24, 2010 12:00:54 AM
      FATAL: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
      hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
      	at hudson.FilePath.act(FilePath.java:743)
      	at hudson.FilePath.act(FilePath.java:729)
      	at com.syncron.hudson.cvs2.CVS2.isUpdatable(CVS2.java:813)
      	at com.syncron.hudson.cvs2.CVS2.pollChanges(CVS2.java:310)
      	at hudson.scm.SCM.poll(SCM.java:370)
      	at hudson.model.AbstractProject.poll(AbstractProject.java:1153)
      	at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:330)
      	at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:359)
      	at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:619)
      Caused by: java.io.IOException: Remote call on slave-81 failed
      	at hudson.remoting.Channel.call(Channel.java:560)
      	at hudson.FilePath.act(FilePath.java:736)
      	... 14 more
      Caused by: java.lang.LinkageError: loader (instance of  hudson/remoting/RemoteClassLoader): attempted  duplicate class definition for name: "hudson/model/ModelObject"
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
      	at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
      	at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      	at java.lang.Class.getDeclaredFields0(Native Method)
      	at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
      	at java.lang.Class.getDeclaredField(Class.java:1880)
      	at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610)
      	at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52)
      	at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
      	at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
      	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:178)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:98)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      	at hudson.remoting.Request$2.run(Request.java:270)
      	... 6 more
      Done. Took 63 ms
      No changes
      

      If we start single job manually after restart it executes properly. Any consecutive jobs will also run fine. However if we get that exception once, no other jobs that use the class mentioned in exception (pretty much all) will execute anymore until slave is restarted.

        Attachments

          Issue Links

            Activity

            michal_grzejszczak michal_grzejszczak created issue -
            michal_grzejszczak michal_grzejszczak made changes -
            Field Original Value New Value
            Description We are restarting hudson each Sunday afternoon to evade problems with memory leaks and have a couple of nightly builds that kick in at midnight. The scenario is that Hudson is fresh when multiple builds kick in, that is its remote class loader did not have a chance to read any classes yet. We have 3 executors defined. I suppose that the SCM poll action that is sent in many build procedures causes multiple requests to load classes for the SCM (we use slightly hacked version of CVS SCM). We are getting the following exception:
            java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"

            I have looked around on the web and found this (http://jira.codehaus.org/browse/JETTY-418) that lead me to believe that lack of synchronization while loading classes in remote class loader is the cause.

            Full stack trace:
            {code}
            Started on May 24, 2010 12:00:54 AM
            FATAL: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
            hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
            at hudson.FilePath.act(FilePath.java:743)
            at hudson.FilePath.act(FilePath.java:729)
            at com.syncron.hudson.cvs2.CVS2.isUpdatable(CVS2.java:813)
            at com.syncron.hudson.cvs2.CVS2.pollChanges(CVS2.java:310)
            at hudson.scm.SCM.poll(SCM.java:370)
            at hudson.model.AbstractProject.poll(AbstractProject.java:1153)
            at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:330)
            at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:359)
            at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            at java.util.concurrent.FutureTask.run(FutureTask.java:138)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
            at java.lang.Thread.run(Thread.java:619)
            Caused by: java.io.IOException: Remote call on slave-81 failed
            at hudson.remoting.Channel.call(Channel.java:560)
            at hudson.FilePath.act(FilePath.java:736)
            ... 14 more
            Caused by: java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"
            at java.lang.ClassLoader.defineClass1(Native Method)
            at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
            at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
            at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
            at java.lang.ClassLoader.defineClass1(Native Method)
            at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
            at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
            at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
            at java.lang.Class.getDeclaredFields0(Native Method)
            at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
            at java.lang.Class.getDeclaredField(Class.java:1880)
            at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610)
            at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52)
            at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
            at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
            at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547)
            at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
            at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
            at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
            at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
            at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
            at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
            at hudson.remoting.UserRequest.deserialize(UserRequest.java:178)
            at hudson.remoting.UserRequest.perform(UserRequest.java:98)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:270)
            ... 6 more
            Done. Took 63 ms
            No changes
            {code}
            We are restarting hudson each Sunday afternoon to evade problems with memory leaks and have a couple of nightly builds that kick in at midnight. The scenario is that Hudson is fresh when multiple builds kick in, that is its remote class loader did not have a chance to read any classes yet. We have 3 executors defined. I suppose that the SCM poll action that is sent in many build procedures causes multiple requests to load classes for the SCM (we use slightly hacked version of CVS SCM). We are getting the following exception:
            java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"

            I have looked around on the web and found this (http://jira.codehaus.org/browse/JETTY-418) that lead me to believe that lack of synchronization while loading classes in remote class loader is the cause.

            Full stack trace:
            {code}
            Started on May 24, 2010 12:00:54 AM
            FATAL: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
            hudson.util.IOException2: remote file operation failed: /home/hudson-slave/workspace/BPE_8.1SR at hudson.remoting.Channel@1219b8c:slave-81
            at hudson.FilePath.act(FilePath.java:743)
            at hudson.FilePath.act(FilePath.java:729)
            at com.syncron.hudson.cvs2.CVS2.isUpdatable(CVS2.java:813)
            at com.syncron.hudson.cvs2.CVS2.pollChanges(CVS2.java:310)
            at hudson.scm.SCM.poll(SCM.java:370)
            at hudson.model.AbstractProject.poll(AbstractProject.java:1153)
            at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:330)
            at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:359)
            at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
            at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            at java.util.concurrent.FutureTask.run(FutureTask.java:138)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
            at java.lang.Thread.run(Thread.java:619)
            Caused by: java.io.IOException: Remote call on slave-81 failed
            at hudson.remoting.Channel.call(Channel.java:560)
            at hudson.FilePath.act(FilePath.java:736)
            ... 14 more
            Caused by: java.lang.LinkageError: loader (instance of hudson/remoting/RemoteClassLoader): attempted duplicate class definition for name: "hudson/model/ModelObject"
            at java.lang.ClassLoader.defineClass1(Native Method)
            at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
            at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
            at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
            at java.lang.ClassLoader.defineClass1(Native Method)
            at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:466)
            at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:151)
            at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:131)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
            at java.lang.Class.getDeclaredFields0(Native Method)
            at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
            at java.lang.Class.getDeclaredField(Class.java:1880)
            at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1610)
            at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:52)
            at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:425)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
            at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
            at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:547)
            at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
            at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
            at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
            at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
            at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
            at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
            at hudson.remoting.UserRequest.deserialize(UserRequest.java:178)
            at hudson.remoting.UserRequest.perform(UserRequest.java:98)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:270)
            ... 6 more
            Done. Took 63 ms
            No changes
            {code}

            If we start single job manually after restart it executes properly. Any consecutive jobs will also run fine. However if we get that exception once, no other jobs that use the class mentioned in exception (pretty much all) will execute anymore until slave is restarted.
            stevanr Stevan Radakovic made changes -
            Attachment jenkins-6604-remoting-2.12.patch [ 22000 ]
            eriklovlie Erik Lovlie made changes -
            Priority Minor [ 4 ] Critical [ 2 ]
            eriklovlie Erik Lovlie made changes -
            Priority Critical [ 2 ] Blocker [ 1 ]
            eriklovlie Erik Lovlie made changes -
            Assignee Andrey Myatlyuk [ kukido ]
            eriklovlie Erik Lovlie made changes -
            Assignee Andrey Myatlyuk [ kukido ] Kohsuke Kawaguchi [ kktest10 ]
            eriklovlie Erik Lovlie made changes -
            Link This issue is related to JENKINS-15716 [ JENKINS-15716 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-15716 [ JENKINS-15716 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-15716 [ JENKINS-15716 ]
            jglick Jesse Glick made changes -
            Assignee Kohsuke Kawaguchi [ kktest10 ] Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            olamy Olivier Lamy made changes -
            Link This issue is duplicated by JENKINS-15845 [ JENKINS-15845 ]
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            jglick Jesse Glick made changes -
            Labels 1.480.3-fixed
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 136699 ] JNJira + In-Review [ 187268 ]

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              michal_grzejszczak michal_grzejszczak
              Votes:
              5 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: