• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None

      The way the remoting is written, a certain race condition can result in a dead lock.

      The root cause of this is that Channel.ReaderThread performs blocking write operation to OutputStream when it's executing the ProxyOutputStream.Chunk command. This synchronous behavior was necessary to ensure the proper in-order data arrival guarantee.

      Now, the situation is, the master initiates a FilePath.copyRecursiveTo from the master to a slave. This creates a piped stream pair over Channel, where the sender on the master are directly sending data over Channel, which gets to FastPipedOutputStream on the slave, then the forwarded Callable on the slave is reading from the corresponding FastPipedInputStream.

      The callable executing on the slave needs to load a class from the master, which results in this callable making an callback to the master to fetch a class file. But while this class file arrives as a Command, the master can send enough byte sequence to FastPipedOutputStream to the point that the buffer fills up and the write operation blocks.

      At this point, the class file is stuck in the network waiting to be read by the command reader thread, but that won't be able to do that until there'll be some space in the buffer, and there won't be any space in the buffer until the class file is retrieved, hence the dead lock.

        1. log.zip
          15 kB
        2. HUDSON-5977.stack
          17 kB
        3. deadlockpatch.txt
          3 kB
        4. 34838_hang_on_emma_publisher.txt
          11 kB
        5. 34838_hang_on_archiving.txt
          18 kB

          [JENKINS-5977] Dead lock condition due to pipe clogging

          Kohsuke Kawaguchi created issue -

          This is the thread executing Callable of FilePath.copyRecursiveTo.

          Thread t@15: (state = BLOCKED)
           - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
           - hudson.remoting.Request.call(hudson.remoting.Channel) @bci=120, line=122 (Interpreted frame)
           - hudson.remoting.RemoteInvocationHandler.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=135, line=160 (Interpreted frame)
           - hudson.remoting.$Proxy5.fetch2(java.lang.String) @bci=16 (Interpreted frame)
           - hudson.remoting.RemoteClassLoader.findClass(java.lang.String) @bci=41, line=121 (Interpreted frame)
           - java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=43, line=307 (Interpreted frame)
           - java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=252 (Interpreted frame)
           - java.lang.ClassLoader.loadClassInternal(java.lang.String) @bci=2, line=320 (Interpreted frame)
           - hudson.FilePath$32.invoke(java.io.File, hudson.remoting.VirtualChannel) @bci=33, line=1403 (Interpreted frame)
           - hudson.FilePath$32.invoke(java.io.File, hudson.remoting.VirtualChannel) @bci=3, line=1400 (Interpreted frame)
           - hudson.FilePath$FileCallableWrapper.call() @bci=21, line=1962 (Interpreted frame)
           - hudson.remoting.UserRequest.perform(hudson.remoting.Channel) @bci=123, line=104 (Interpreted frame)
           - hudson.remoting.UserRequest.perform(hudson.remoting.Channel) @bci=2, line=48 (Interpreted frame)
           - hudson.remoting.Request$2.run() @bci=8, line=270 (Interpreted frame)
           - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame)
           - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame)
           - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame)
           - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
           - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
           - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
          

          This is the channel reader thread blocking on FastPipedOutputStream:

          Thread t@13: (state = BLOCKED)
           - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
           - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
           - hudson.remoting.FastPipedOutputStream.write(byte[], int, int) @bci=106, line=136 (Interpreted frame)
           - hudson.remoting.FastPipedOutputStream.write(byte[]) @bci=5, line=115 (Interpreted frame)
           - hudson.remoting.ProxyOutputStream$Chunk.execute(hudson.remoting.Channel) @bci=17, line=164 (Interpreted frame)
           - hudson.remoting.Channel$ReaderThread.run() @bci=119, line=867 (Interpreted frame)
          

          Kohsuke Kawaguchi added a comment - This is the thread executing Callable of FilePath.copyRecursiveTo. Thread t@15: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - hudson.remoting.Request.call(hudson.remoting.Channel) @bci=120, line=122 (Interpreted frame) - hudson.remoting.RemoteInvocationHandler.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=135, line=160 (Interpreted frame) - hudson.remoting.$Proxy5.fetch2(java.lang.String) @bci=16 (Interpreted frame) - hudson.remoting.RemoteClassLoader.findClass(java.lang.String) @bci=41, line=121 (Interpreted frame) - java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=43, line=307 (Interpreted frame) - java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=252 (Interpreted frame) - java.lang.ClassLoader.loadClassInternal(java.lang.String) @bci=2, line=320 (Interpreted frame) - hudson.FilePath$32.invoke(java.io.File, hudson.remoting.VirtualChannel) @bci=33, line=1403 (Interpreted frame) - hudson.FilePath$32.invoke(java.io.File, hudson.remoting.VirtualChannel) @bci=3, line=1400 (Interpreted frame) - hudson.FilePath$FileCallableWrapper.call() @bci=21, line=1962 (Interpreted frame) - hudson.remoting.UserRequest.perform(hudson.remoting.Channel) @bci=123, line=104 (Interpreted frame) - hudson.remoting.UserRequest.perform(hudson.remoting.Channel) @bci=2, line=48 (Interpreted frame) - hudson.remoting.Request$2.run() @bci=8, line=270 (Interpreted frame) - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame) - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) This is the channel reader thread blocking on FastPipedOutputStream: Thread t@13: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - hudson.remoting.FastPipedOutputStream.write(byte[], int, int) @bci=106, line=136 (Interpreted frame) - hudson.remoting.FastPipedOutputStream.write(byte[]) @bci=5, line=115 (Interpreted frame) - hudson.remoting.ProxyOutputStream$Chunk.execute(hudson.remoting.Channel) @bci=17, line=164 (Interpreted frame) - hudson.remoting.Channel$ReaderThread.run() @bci=119, line=867 (Interpreted frame)

          The reverse chronological sequence of the log showing what the master has sent. Looking at the slave stack trace, it hasn't read a single byte yet, but the master has sent in the 370K data in "Pipe.Chunk(6,371226)", and the class file image afterward, which is never read.

          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7900)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7988)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7908)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7804)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7740)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7996)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7888)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7856)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7964)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7984)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Response[retVal=hudson.remoting.RemoteClassLoader$ClassFile@5f471136,exception=null]
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Received RPCRequest(2,fetch2)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7768)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,8192)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,7928)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send Pipe.Chunk(6,371226)
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Received hudson.remoting.Pipe$ConnectCommand@47d7e1fb
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send UserRequest:hudson.FilePath$FileCallableWrapper@40af3623
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Received Response[retVal=hudson.remoting.UserResponse@145e62e7,exception=null]
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Send UserRequest:hudson.EnvVars$GetEnvVars@652dc559
          
          Mar 17, 2010 6:13:02 PM hudson.remoting.Channel
          FINE: Received Response[retVal=hudson.remoting.UserResponse@762c3f0,exception=null]
          
          Mar 17, 2010 6:13:01 PM hudson.remoting.Channel
          FINE: Send UserRequest:hudson.EnvVars$GetEnvVars@490606c0
          
          Mar 17, 2010 6:13:01 PM hudson.remoting.Channel
          FINE: Received Response[retVal=hudson.remoting.UserResponse@3fd15873,exception=null]
          
          Mar 17, 2010 6:13:01 PM hudson.remoting.Channel
          FINE: Send UserRequest:hudson.FilePath$FileCallableWrapper@26c76ec2
          

          Kohsuke Kawaguchi added a comment - The reverse chronological sequence of the log showing what the master has sent. Looking at the slave stack trace, it hasn't read a single byte yet, but the master has sent in the 370K data in "Pipe.Chunk(6,371226)", and the class file image afterward, which is never read. Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7900) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7988) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7908) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7804) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7740) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7996) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7888) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7856) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7964) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7984) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Response[retVal=hudson.remoting.RemoteClassLoader$ClassFile@5f471136,exception=null] Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Received RPCRequest(2,fetch2) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7768) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,8192) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,7928) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send Pipe.Chunk(6,371226) Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Received hudson.remoting.Pipe$ConnectCommand@47d7e1fb Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send UserRequest:hudson.FilePath$FileCallableWrapper@40af3623 Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Received Response[retVal=hudson.remoting.UserResponse@145e62e7,exception=null] Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Send UserRequest:hudson.EnvVars$GetEnvVars@652dc559 Mar 17, 2010 6:13:02 PM hudson.remoting.Channel FINE: Received Response[retVal=hudson.remoting.UserResponse@762c3f0,exception=null] Mar 17, 2010 6:13:01 PM hudson.remoting.Channel FINE: Send UserRequest:hudson.EnvVars$GetEnvVars@490606c0 Mar 17, 2010 6:13:01 PM hudson.remoting.Channel FINE: Received Response[retVal=hudson.remoting.UserResponse@3fd15873,exception=null] Mar 17, 2010 6:13:01 PM hudson.remoting.Channel FINE: Send UserRequest:hudson.FilePath$FileCallableWrapper@26c76ec2

          Code changed in hudson
          User: : mindless
          Path:
          trunk/hudson/plugins/copyartifact/src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
          trunk/hudson/plugins/copyartifact/src/main/webapp/JENKINS-5977/deleteme.tmp
          http://jenkins-ci.org/commit/28825
          Log:
          [copyartifact] [FIXED JENKINS-5934] Add workaround for JENKINS-5977
          so slaves won't hang on copy of larger artifacts.

          SCM/JIRA link daemon added a comment - Code changed in hudson User: : mindless Path: trunk/hudson/plugins/copyartifact/src/main/java/hudson/plugins/copyartifact/CopyArtifact.java trunk/hudson/plugins/copyartifact/src/main/webapp/ JENKINS-5977 /deleteme.tmp http://jenkins-ci.org/commit/28825 Log: [copyartifact] [FIXED JENKINS-5934] Add workaround for JENKINS-5977 so slaves won't hang on copy of larger artifacts.

          Another possibly related pipe clogging problem, this time the reader thread on the master is clogging:

          This is the thread on a slave that's sending stream. This is FilePath.read()

          "pool-1-thread-415" prio=6 tid=0x031df800 nid=0x604 runnable [0x02e6f000]
             java.lang.Thread.State: RUNNABLE
          	at java.net.SocketOutputStream.socketWrite0(Native Method)
          	at java.net.SocketOutputStream.socketWrite(Unknown Source)
          	at java.net.SocketOutputStream.write(Unknown Source)
          	at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
          	at java.io.BufferedOutputStream.write(Unknown Source)
          	- locked <0x22fe3c60> (a java.io.BufferedOutputStream)
          	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(Unknown Source)
          	at java.io.ObjectOutputStream$BlockDataOutputStream.write(Unknown Source)
          	at java.io.ObjectOutputStream.writeArray(Unknown Source)
          	at java.io.ObjectOutputStream.writeObject0(Unknown Source)
          	at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source)
          	at java.io.ObjectOutputStream.writeSerialData(Unknown Source)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
          	at java.io.ObjectOutputStream.writeObject0(Unknown Source)
          	at java.io.ObjectOutputStream.writeObject(Unknown Source)
          	at hudson.remoting.Channel.send(Channel.java:417)
          	- locked <0x22fda8d0> (a hudson.remoting.Channel)
          	at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:112)
          	- locked <0x2303de50> (a hudson.remoting.ProxyOutputStream)
          	at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:96)
          	at hudson.Util.copyStream(Util.java:406)
          	at hudson.FilePath$24.call(FilePath.java:1202)
          	at hudson.FilePath$24.call(FilePath.java:1197)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:114)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          	at hudson.remoting.Request$2.run(Request.java:270)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          	at java.util.concurrent.FutureTask.run(Unknown Source)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          	at hudson.remoting.Engine$1$1.run(Engine.java:58)
          	at java.lang.Thread.run(Unknown Source)
          

          This is the thread on the master hanging because of this:

          "Channel reader thread: hb-slave-trunk3" Id=53 TIMED_WAITING on [B@18c8d4d
          	at java.lang.Object.wait(Native Method)
          	-  waiting on [B@18c8d4d
          	at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:147)
          	at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:122)
          	at hudson.remoting.ProxyOutputStream$Chunk.execute(ProxyOutputStream.java:164)
          	at hudson.remoting.Channel$ReaderThread.run(Channel.java:867)
          

          Kohsuke Kawaguchi added a comment - Another possibly related pipe clogging problem, this time the reader thread on the master is clogging: This is the thread on a slave that's sending stream. This is FilePath.read() "pool-1-thread-415" prio=6 tid=0x031df800 nid=0x604 runnable [0x02e6f000] java.lang.Thread.State: RUNNABLE at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at java.io.BufferedOutputStream.flushBuffer(Unknown Source) at java.io.BufferedOutputStream.write(Unknown Source) - locked <0x22fe3c60> (a java.io.BufferedOutputStream) at java.io.ObjectOutputStream$BlockDataOutputStream.drain(Unknown Source) at java.io.ObjectOutputStream$BlockDataOutputStream.write(Unknown Source) at java.io.ObjectOutputStream.writeArray(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source) at java.io.ObjectOutputStream.writeSerialData(Unknown Source) at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeObject(Unknown Source) at hudson.remoting.Channel.send(Channel.java:417) - locked <0x22fda8d0> (a hudson.remoting.Channel) at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:112) - locked <0x2303de50> (a hudson.remoting.ProxyOutputStream) at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:96) at hudson.Util.copyStream(Util.java:406) at hudson.FilePath$24.call(FilePath.java:1202) at hudson.FilePath$24.call(FilePath.java:1197) at hudson.remoting.UserRequest.perform(UserRequest.java:114) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:270) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:58) at java.lang.Thread.run(Unknown Source) This is the thread on the master hanging because of this: "Channel reader thread: hb-slave-trunk3" Id=53 TIMED_WAITING on [B@18c8d4d at java.lang.Object.wait(Native Method) - waiting on [B@18c8d4d at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:147) at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:122) at hudson.remoting.ProxyOutputStream$Chunk.execute(ProxyOutputStream.java:164) at hudson.remoting.Channel$ReaderThread.run(Channel.java:867)

          Details for my previous comment.

          Kohsuke Kawaguchi added a comment - Details for my previous comment.
          Kohsuke Kawaguchi made changes -
          Attachment New: log.zip [ 19251 ]

          Full log for my previous comment is at http://issues.jenkins-ci.org/secure/attachment/19251/log.zip

          Kohsuke Kawaguchi added a comment - Full log for my previous comment is at http://issues.jenkins-ci.org/secure/attachment/19251/log.zip

          glundh added a comment -

          glundh added a comment - Proof of concept fix: http://hudson.361315.n4.nabble.com/Discussion-regarding-JENKINS-5977-Dead-lock-condition-due-to-pipe-clogging-And-more-td2322497.html Please review.

          glundh added a comment -

          Proof-of-concept fix.

          glundh added a comment - Proof-of-concept fix.
          glundh made changes -
          Attachment New: deadlockpatch.txt [ 19736 ]

            kohsuke Kohsuke Kawaguchi
            kohsuke Kohsuke Kawaguchi
            Votes:
            27 Vote for this issue
            Watchers:
            26 Start watching this issue

              Created:
              Updated:
              Resolved: