Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14909

Deadlock between SlaveComputer$2.onClosed and printStackTrace from CLICommand.main

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core

      A deadlock involving the remoting layer:

      "Channel reader thread: HTTP full-duplex channel ...":
        waiting to lock monitor 0x00007f12c4d469b8 (object 0x00000000dd36af48, a java.io.PrintStream),
        which is held by "pool-3-thread-2602"
      "pool-3-thread-2602":
        waiting to lock monitor 0x00007f12acb570f8 (object 0x00000000dd3657b0, a hudson.remoting.Channel),
        which is held by "Channel reader thread: HTTP full-duplex channel ..."
      "Channel reader thread: HTTP full-duplex channel ...":
      	at java.io.PrintStream.write(PrintStream.java:429)
      	- waiting to lock <0x00000000dd36af48> (a java.io.PrintStream)
      	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
      	at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
      	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
      	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
      	- locked <0x00000000dd5e84e0> (a java.io.OutputStreamWriter)
      	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
      	at java.io.PrintWriter.newLine(PrintWriter.java:438)
      	- locked <0x00000000dd5e84e0> (a java.io.OutputStreamWriter)
      	at java.io.PrintWriter.println(PrintWriter.java:585)
      	at java.io.PrintWriter.println(PrintWriter.java:696)
      	- locked <0x00000000dd5e84e0> (a java.io.OutputStreamWriter)
      	at java.lang.Throwable.printStackTraceAsCause(Throwable.java:538)
      	at java.lang.Throwable.printStackTrace(Throwable.java:516)
      	- locked <0x00000000dd5e8538> (a java.io.PrintWriter)
      	at hudson.slaves.SlaveComputer$2.onClosed(SlaveComputer.java:339)
      	at hudson.remoting.Channel.terminate(Channel.java:747)
      	- locked <0x00000000dd3657b0> (a hudson.remoting.Channel)
      	at hudson.remoting.Channel$ReaderThread.run(Channel.java:1139)
      "pool-3-thread-2602":
      	at hudson.remoting.Channel.send(Channel.java:498)
      	- waiting to lock <0x00000000dd3657b0> (a hudson.remoting.Channel)
      	at hudson.remoting.ProxyOutputStream._write(ProxyOutputStream.java:123)
      	- locked <0x00000000dd36af78> (a hudson.remoting.ProxyOutputStream)
      	at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:103)
      	at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:110)
      	at java.io.PrintStream.write(PrintStream.java:430)
      	- locked <0x00000000dd36af48> (a java.io.PrintStream)
      	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
      	at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
      	at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)
      	- locked <0x00000000dd36afc8> (a java.io.OutputStreamWriter)
      	at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)
      	at java.io.PrintStream.write(PrintStream.java:477)
      	- locked <0x00000000dd36af48> (a java.io.PrintStream)
      	at java.io.PrintStream.print(PrintStream.java:619)
      	at java.io.PrintStream.println(PrintStream.java:773)
      	- locked <0x00000000dd36af48> (a java.io.PrintStream)
      	at java.lang.Throwable.printStackTrace(Throwable.java:461)
      	- locked <0x00000000dd36af48> (a java.io.PrintStream)
      	at hudson.cli.CLICommand.main(CLICommand.java:215)
      	at hudson.cli.CliManagerImpl.main(CliManagerImpl.java:66)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:274)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:255)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      	at hudson.remoting.Request$2.run(Request.java:287)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      

          [JENKINS-14909] Deadlock between SlaveComputer$2.onClosed and printStackTrace from CLICommand.main

          Jesse Glick added a comment -

          The immediate problem is that terminate is synchronized yet invokes unknown listener callbacks, which is always a poor idea.

          It is hard to tell from the thread dump what really is happening. Is there just one Channel involved, or several (slave + CLI)? How can SlaveComputer expect to print to a closed channel, or is it trying to print to a different channel?

          Jesse Glick added a comment - The immediate problem is that terminate is synchronized yet invokes unknown listener callbacks, which is always a poor idea. It is hard to tell from the thread dump what really is happening. Is there just one Channel involved, or several (slave + CLI)? How can SlaveComputer expect to print to a closed channel, or is it trying to print to a different channel?

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/hudson/remoting/Channel.java
          http://jenkins-ci.org/commit/remoting/f28b79dfd26853d7e97ad3037afa69c689cf5ded
          Log:
          JENKINS-14909 Deadlock involving onClosed.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/remoting/Channel.java http://jenkins-ci.org/commit/remoting/f28b79dfd26853d7e97ad3037afa69c689cf5ded Log: JENKINS-14909 Deadlock involving onClosed.

          Jesse Glick added a comment -

          Tried to fix in remoting layer; Jenkins would need to incorporate 2.17 to pick up fix.

          Jesse Glick added a comment - Tried to fix in remoting layer; Jenkins would need to incorporate 2.17 to pick up fix.

          Yes, the stack trace indicates that the slave log that SlaveComputer is writing to is over the very same channel. This fix should fix that.

          The actual attempt to write to such a stream in hudson.slaves.SlaveComputer$2.onClosed(SlaveComputer.java:339) would fail silently, but that's OK.

          Kohsuke Kawaguchi added a comment - Yes, the stack trace indicates that the slave log that SlaveComputer is writing to is over the very same channel. This fix should fix that. The actual attempt to write to such a stream in hudson.slaves.SlaveComputer$2.onClosed(SlaveComputer.java:339) would fail silently, but that's OK.

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          pom.xml
          http://jenkins-ci.org/commit/jenkins/c733b24be79d4c6b12110107fda832bce2e6f08d
          Log:
          [FIXED JENKINS-14909]

          integrating a fix that's made in remoting.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/c733b24be79d4c6b12110107fda832bce2e6f08d Log: [FIXED JENKINS-14909] integrating a fix that's made in remoting.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #1870
          [FIXED JENKINS-14909] (Revision c733b24be79d4c6b12110107fda832bce2e6f08d)

          Result = SUCCESS
          Kohsuke Kawaguchi : c733b24be79d4c6b12110107fda832bce2e6f08d
          Files :

          • changelog.html
          • pom.xml

          dogfood added a comment - Integrated in jenkins_main_trunk #1870 [FIXED JENKINS-14909] (Revision c733b24be79d4c6b12110107fda832bce2e6f08d) Result = SUCCESS Kohsuke Kawaguchi : c733b24be79d4c6b12110107fda832bce2e6f08d Files : changelog.html pom.xml

            kohsuke Kohsuke Kawaguchi
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: