Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-3077

Hudson 1.284 streams many NullPointerExceptions

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • other
    • None
    • Platform: All, OS: All

      After Hudson 1.284 executes a couple of builds (on a slave node?), it starts
      streaming 25-30 NullPointerExceptions that look like:

      Feb 18, 2009 3:24:58 PM hudson.remoting.Channel$ReaderThread run
      SEVERE: Failed to execute command Pipe.EOF(0)
      java.lang.NullPointerException
      at hudson.remoting.ProxyOutputStream$EOF.execute(ProxyOutputStream.java:229)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:670)
      Feb 18, 2009 3:24:58 PM hudson.remoting.Channel$ReaderThread run
      SEVERE: This command is created here
      Command Pipe.EOF(0) created at
      at hudson.remoting.Command.<init>(Command.java:24)
      at hudson.remoting.ProxyOutputStream$EOF.<init>(ProxyOutputStream.java:169)
      at hudson.remoting.ProxyOutputStream.doClose(ProxyOutputStream.java:104)
      at hudson.remoting.ProxyOutputStream.close(ProxyOutputStream.java:100)
      at hudson.remoting.RemoteOutputStream.close(RemoteOutputStream.java:93)
      at hudson.FilePath$23.invoke(FilePath.java:874)
      at hudson.FilePath$23.invoke(FilePath.java:869)
      at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1278)
      at hudson.remoting.UserRequest.perform(UserRequest.java:69)
      at hudson.remoting.UserRequest.perform(UserRequest.java:23)
      at hudson.remoting.Request$2.run(Request.java:213)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
      at java.util.concurrent.FutureTask.run(FutureTask.java:123)
      at
      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
      at
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:595)

      I'm not sure what, if any, effect this has on Hudson or build results besides
      spamming the server log (although my Hudson server currently only lives about 2
      hours anyway due to issue 3076.) Both master and slaves are RHEL5 machines.

          [JENKINS-3077] Hudson 1.284 streams many NullPointerExceptions

          bbarlow added a comment -

          (ignore my previous comment - I intended to post that to issue 3076 instead, sorry)

          bbarlow added a comment - (ignore my previous comment - I intended to post that to issue 3076 instead, sorry)

          Code changed in hudson
          User: : kohsuke
          Path:
          trunk/hudson/main/core/src/test/java/hudson/FilePathTest.java
          trunk/hudson/main/remoting/src/main/java/hudson/remoting/ProxyOutputStream.java
          trunk/hudson/main/remoting/src/main/java/hudson/remoting/RemoteOutputStream.java
          http://fisheye4.cenqua.com/changelog/hudson/?cs=15446
          Log:
          JENKINS-3077
          Adding more probes to understand where the invalid data creeps in

          SCM/JIRA link daemon added a comment - Code changed in hudson User: : kohsuke Path: trunk/hudson/main/core/src/test/java/hudson/FilePathTest.java trunk/hudson/main/remoting/src/main/java/hudson/remoting/ProxyOutputStream.java trunk/hudson/main/remoting/src/main/java/hudson/remoting/RemoteOutputStream.java http://fisheye4.cenqua.com/changelog/hudson/?cs=15446 Log: JENKINS-3077 Adding more probes to understand where the invalid data creeps in

          vijayj added a comment -

          I am running into the same issue. I have the following information all over my
          log file. Builds are running just fine. One thing I have noticed, I am not able
          to view any files when click on the workspace icon. Not sure these are all
          related this issue...

          Feb 18, 2009 11:08:23 AM hudson.remoting.Channel$ReaderThread run
          SEVERE: Failed to execute command Pipe.EOF(0)
          java.lang.NullPointerException
          at hudson.remoting.ProxyOutputStream$EOF.execute(ProxyOutputStream.java:229)
          at hudson.remoting.Channel$ReaderThread.run(Channel.java:670)
          Feb 18, 2009 11:08:23 AM hudson.remoting.Channel$ReaderThread run
          SEVERE: This command is created here
          Command Pipe.EOF(0) created at
          at hudson.remoting.Command.<init>(Command.java:24)
          at hudson.remoting.ProxyOutputStream$EOF.<init>(ProxyOutputStream.java:169)
          at hudson.remoting.ProxyOutputStream.doClose(ProxyOutputStream.java:104)
          at hudson.remoting.ProxyOutputStream.close(ProxyOutputStream.java:100)
          at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:196)
          at hudson.FilePath$18.call(FilePath.java:771)
          at hudson.FilePath$18.call(FilePath.java:762)
          at hudson.remoting.UserRequest.perform(UserRequest.java:69)
          at hudson.remoting.UserRequest.perform(UserRequest.java:23)
          at hudson.remoting.Request$2.run(Request.java:213)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at
          java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
          Feb 18, 2009 11:08:31 AM hudson.remoting.Channel$ReaderThread run
          SEVERE: Failed to execute command Pipe.EOF(0)
          java.lang.NullPointerException
          at hudson.remoting.ProxyOutputStream$EOF.execute(ProxyOutputStream.java:229)
          at hudson.remoting.Channel$ReaderThread.run(Channel.java:670)
          Feb 18, 2009 11:08:31 AM hudson.remoting.Channel$ReaderThread run

          vijayj added a comment - I am running into the same issue. I have the following information all over my log file. Builds are running just fine. One thing I have noticed, I am not able to view any files when click on the workspace icon. Not sure these are all related this issue... Feb 18, 2009 11:08:23 AM hudson.remoting.Channel$ReaderThread run SEVERE: Failed to execute command Pipe.EOF(0) java.lang.NullPointerException at hudson.remoting.ProxyOutputStream$EOF.execute(ProxyOutputStream.java:229) at hudson.remoting.Channel$ReaderThread.run(Channel.java:670) Feb 18, 2009 11:08:23 AM hudson.remoting.Channel$ReaderThread run SEVERE: This command is created here Command Pipe.EOF(0) created at at hudson.remoting.Command.<init>(Command.java:24) at hudson.remoting.ProxyOutputStream$EOF.<init>(ProxyOutputStream.java:169) at hudson.remoting.ProxyOutputStream.doClose(ProxyOutputStream.java:104) at hudson.remoting.ProxyOutputStream.close(ProxyOutputStream.java:100) at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:196) at hudson.FilePath$18.call(FilePath.java:771) at hudson.FilePath$18.call(FilePath.java:762) at hudson.remoting.UserRequest.perform(UserRequest.java:69) at hudson.remoting.UserRequest.perform(UserRequest.java:23) at hudson.remoting.Request$2.run(Request.java:213) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Feb 18, 2009 11:08:31 AM hudson.remoting.Channel$ReaderThread run SEVERE: Failed to execute command Pipe.EOF(0) java.lang.NullPointerException at hudson.remoting.ProxyOutputStream$EOF.execute(ProxyOutputStream.java:229) at hudson.remoting.Channel$ReaderThread.run(Channel.java:670) Feb 18, 2009 11:08:31 AM hudson.remoting.Channel$ReaderThread run

          sydpolk added a comment -

          I am being hit by this bug, so I want to see progress.

          sydpolk added a comment - I am being hit by this bug, so I want to see progress.

          Jesse Glick added a comment -

          BTW the newly created FilePathTest seems to fail randomly on my machine.
          Sometimes passes with no messages. Sometimes prints:

          Feb 19, 2009 9:10:39 PM hudson.remoting.Channel$CloseCommand execute
          SEVERE: close command failed on The other side of the channel
          java.io.IOException: Pipe closed
          at java.io.PipedInputStream.checkStateForReceive(PipedInputStream.java:244)
          at java.io.PipedInputStream.receive(PipedInputStream.java:210)
          at java.io.PipedOutputStream.write(PipedOutputStream.java:132)
          at
          java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1838)
          at
          java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1747)
          at java.io.ObjectOutputStream.reset(ObjectOutputStream.java:475)
          at hudson.remoting.Channel.send(Channel.java:352)
          at hudson.remoting.Channel.close(Channel.java:617)
          at hudson.remoting.Channel$CloseCommand.execute(Channel.java:576)
          at hudson.remoting.Channel$ReaderThread.run(Channel.java:670)
          Feb 19, 2009 9:10:39 PM hudson.remoting.Channel$CloseCommand execute
          INFO: close command created at
          Command close created at
          at hudson.remoting.Command.<init>(Command.java:47)
          at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:573)
          at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:573)
          at hudson.remoting.Channel.close(Channel.java:617)
          at hudson.FilePathTest.tearDown(FilePathTest.java:73)
          at junit.framework.TestCase.runBare(TestCase.java:140)
          at junit.framework.TestResult$1.protect(TestResult.java:110)
          at junit.framework.TestResult.runProtected(TestResult.java:128)
          at junit.framework.TestResult.run(TestResult.java:113)
          at junit.framework.TestCase.run(TestCase.java:124)
          at junit.framework.TestSuite.runTest(TestSuite.java:232)
          at junit.framework.TestSuite.run(TestSuite.java:227)
          at
          org.junit.internal.runners.OldTestClassRunner.run(OldTestClassRunner.java:76)
          at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:32)
          at
          org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:421)
          Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.34 sec

          and usually also gets an error which makes the test fail:

          SEVERE: I/O error in channel This side of the channel
          java.io.EOFException
          at
          java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
          at hudson.remoting.Channel$ReaderThread.run(Channel.java:660)

          sometimes also something in tearDown during french.close which I cannot now
          reproduce.

          If I comment out the body of tearDown then the test passes.

          Jesse Glick added a comment - BTW the newly created FilePathTest seems to fail randomly on my machine. Sometimes passes with no messages. Sometimes prints: Feb 19, 2009 9:10:39 PM hudson.remoting.Channel$CloseCommand execute SEVERE: close command failed on The other side of the channel java.io.IOException: Pipe closed at java.io.PipedInputStream.checkStateForReceive(PipedInputStream.java:244) at java.io.PipedInputStream.receive(PipedInputStream.java:210) at java.io.PipedOutputStream.write(PipedOutputStream.java:132) at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1838) at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1747) at java.io.ObjectOutputStream.reset(ObjectOutputStream.java:475) at hudson.remoting.Channel.send(Channel.java:352) at hudson.remoting.Channel.close(Channel.java:617) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:576) at hudson.remoting.Channel$ReaderThread.run(Channel.java:670) Feb 19, 2009 9:10:39 PM hudson.remoting.Channel$CloseCommand execute INFO: close command created at Command close created at at hudson.remoting.Command.<init>(Command.java:47) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:573) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:573) at hudson.remoting.Channel.close(Channel.java:617) at hudson.FilePathTest.tearDown(FilePathTest.java:73) at junit.framework.TestCase.runBare(TestCase.java:140) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.junit.internal.runners.OldTestClassRunner.run(OldTestClassRunner.java:76) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:32) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:421) Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.34 sec and usually also gets an error which makes the test fail: SEVERE: I/O error in channel This side of the channel java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at hudson.remoting.Channel$ReaderThread.run(Channel.java:660) sometimes also something in tearDown during french.close which I cannot now reproduce. If I comment out the body of tearDown then the test passes.

          Code changed in hudson
          User: : kohsuke
          Path:
          trunk/hudson/main/remoting/src/main/java/hudson/remoting/Channel.java
          http://fisheye4.cenqua.com/changelog/hudson/?cs=15498
          Log:
          JENKINS-3077 Fixed a random failure.

          SCM/JIRA link daemon added a comment - Code changed in hudson User: : kohsuke Path: trunk/hudson/main/remoting/src/main/java/hudson/remoting/Channel.java http://fisheye4.cenqua.com/changelog/hudson/?cs=15498 Log: JENKINS-3077 Fixed a random failure.

          Code changed in hudson
          User: : kohsuke
          Path:
          trunk/hudson/main/remoting/src/main/java/hudson/remoting/ProxyOutputStream.java
          trunk/www/changelog.html
          http://fisheye4.cenqua.com/changelog/hudson/?cs=15767
          Log:
          [FIXED JENKINS-3077]
          This was caused by the serialization format change, masked by the manual serialVersionUID for the given class. This is only an issue when the version of slave.jar on the slave and the master disagrees, but then, that is often a common case when people upgrade the master.

          SCM/JIRA link daemon added a comment - Code changed in hudson User: : kohsuke Path: trunk/hudson/main/remoting/src/main/java/hudson/remoting/ProxyOutputStream.java trunk/www/changelog.html http://fisheye4.cenqua.com/changelog/hudson/?cs=15767 Log: [FIXED JENKINS-3077] This was caused by the serialization format change, masked by the manual serialVersionUID for the given class. This is only an issue when the version of slave.jar on the slave and the master disagrees, but then, that is often a common case when people upgrade the master.

          Let me do a recap on what happened. Since some people may see the aftermath of this.

          As far as this issue is concerned, there are three variations of
          ProxyOutputStream implementations:

          Everything until 1.283, we'll call it as "version A"
          From 1.284 till 1.285, we'll call it as "version B"
          From 1.286 onward, we'll call it as "version C"

          ProxyOutputStream class is in slave.jar, and therefore in a common distributed
          build set up, this file manually needs to be copied over to slaves.

          The compatibility between these versions are as follows:

          A B C
          A o x o
          B x o x
          C o x o

          "o" indicates "it works without this problem", and "x" indicates it exhibits the
          problem outlined in this issue.

          So for example, if you have copied slave.jar to your slave a long time ago and
          upgraded master to 1.284, then you have B in master and A in slave, so you hit
          this problem.

          Notice that it is still possible to hit this problem after 1.286 release — for
          example, you might have B in your slave and if the master is C, you get this
          problem again.

          So if you continue to see this problem after 1.286, please first make sure that
          you got the slave.jar updated to the version matching your master. If you are
          positive that it still doesn't fix your problem, please feel free to reopen the
          issue.

          Kohsuke Kawaguchi added a comment - Let me do a recap on what happened. Since some people may see the aftermath of this. As far as this issue is concerned, there are three variations of ProxyOutputStream implementations: Everything until 1.283, we'll call it as "version A" From 1.284 till 1.285, we'll call it as "version B" From 1.286 onward, we'll call it as "version C" ProxyOutputStream class is in slave.jar, and therefore in a common distributed build set up, this file manually needs to be copied over to slaves. The compatibility between these versions are as follows: A B C A o x o B x o x C o x o "o" indicates "it works without this problem", and "x" indicates it exhibits the problem outlined in this issue. So for example, if you have copied slave.jar to your slave a long time ago and upgraded master to 1.284, then you have B in master and A in slave, so you hit this problem. Notice that it is still possible to hit this problem after 1.286 release — for example, you might have B in your slave and if the master is C, you get this problem again. So if you continue to see this problem after 1.286, please first make sure that you got the slave.jar updated to the version matching your master. If you are positive that it still doesn't fix your problem, please feel free to reopen the issue.

          bbarlow added a comment -

          Note: I am no longer seeing these exceptions with 1.287.

          bbarlow added a comment - Note: I am no longer seeing these exceptions with 1.287.

          erikengerd added a comment -
              • Issue 3825 has been marked as a duplicate of this issue. ***

          erikengerd added a comment - Issue 3825 has been marked as a duplicate of this issue. ***

            Unassigned Unassigned
            bbarlow bbarlow
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: