-
Bug
-
Resolution: Fixed
-
Major
-
Powered by SuggestiMate
Errors from log, occurs from time to time:
Nov 26, 2013 4:31:58 PM hudson.remoting.Channel$2 handle SEVERE: Failed to execute command Pipe.EOF(49) (channel qa3) java.lang.NullPointerException at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Nov 26, 2013 4:31:58 PM hudson.remoting.Channel$2 handle SEVERE: This command is created here Command Pipe.EOF(49) created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.ProxyWriter$EOF.<init>(ProxyWriter.java:169) at hudson.remoting.ProxyWriter.close(ProxyWriter.java:124) at hudson.remoting.ProxyWriter.finalize(ProxyWriter.java:132) at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101) at java.lang.ref.Finalizer.access$100(Finalizer.java:32) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
[JENKINS-20769] Failed to execute command Pipe.EOF
I committed https://github.com/jenkinsci/remoting/commit/622af099d6b94569c873b8fcd2d1bc65860a8cc6 but am not sure of the root cause here. Possibly a race condition while trying to shut down a remoting channel.
Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/hudson/remoting/ProxyInputStream.java
src/main/java/hudson/remoting/ProxyOutputStream.java
http://jenkins-ci.org/commit/remoting/eb3a4c380e695580174ce381a8330e8e2b8a9b2e
Log:
JENKINS-20769 Trying to fix/suppress some related observed or actual NPEs.
java.lang.NullPointerException
at hudson.remoting.ProxyOutputStream$Flush$1.run(ProxyOutputStream.java:305)
at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:111)
I have it also with 1.532.1 and 1.532.2-SNAPSHOT (rc-01/13/2014 23:54 GMT-kohsuke)
I'm also seeing this in 1.549:
Feb 4, 2014 9:10:08 AM com.cloudbees.jenkins.GitHubPushTrigger$1 run INFO: SCM changes detected in xwiki-platform. Triggering #4954 Feb 4, 2014 9:10:28 AM hudson.remoting.Channel$2 handle SEVERE: Failed to execute command Pipe.EOF(9) (channel agent-2-2) java.lang.NullPointerException at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)Feb 4, 2014 9:10:28 AM hudson.remoting.Channel$2 handle SEVERE: This command is created here Command Pipe.EOF(9) created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.ProxyWriter$EOF.<init>(ProxyWriter.java:169) at hudson.remoting.ProxyWriter.close(ProxyWriter.java:124) at hudson.remoting.ProxyWriter.finalize(ProxyWriter.java:132) at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83) at java.lang.ref.Finalizer.access$100(Finalizer.java:14) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:172) Feb 4, 2014 9:10:31 AM hudson.remoting.Channel$2 handle SEVERE: Failed to execute command Pipe.EOF(9) (channel agent-2-3) java.lang.NullPointerException at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Feb 4, 2014 9:10:31 AM hudson.remoting.Channel$2 handle SEVERE: This command is created here Command Pipe.EOF(9) created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.ProxyWriter$EOF.<init>(ProxyWriter.java:169) at hudson.remoting.ProxyWriter.close(ProxyWriter.java:124) at hudson.remoting.ProxyWriter.finalize(ProxyWriter.java:132) at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83) at java.lang.ref.Finalizer.access$100(Finalizer.java:14) ...
Is it a problem? What can I do?
I've recently upgraded from fedora 14 to fedora 20 and am suddenly getting this one quite often. for me it seems to occur after it provisioned a new EC2 node.
Mar 04, 2014 8:03:49 AM hudson.slaves.NodeProvisioner update INFO: pg, mysql, etc (ami-d53f32bc) provisioningE successfully completed. We have now 2 computer(s) Mar 04, 2014 8:04:09 AM hudson.remoting.Channel$2 handle SEVERE: Failed to execute command Pipe.EOF(10) (channel pg, mysql, etc (i-e5638cc6)) java.lang.NullPointerException at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
I have this issue though I may be able to provide a more reliable repeatable method. My setup is as follows.
-> Jenkins running in JBoss as jboss system user (RHEL 6.5) - jboss-eap-5.1.2 - Jenkins ver. 1.532.2
-> slave.jar running on RHEL6.5 slave as jenkins system user which is started via SSH keys.
-> Eclipse Standard 4.3.2 - eclipse-standard-kepler-SR2-win32-x86_64
-> Tried Mylyn build connector for Jenkins using both latest or weekly
-> Execute a build from the Builds view.
When I run a build a couple of times every once in a while the JBoss server will crash completely. Here's the log output.
13:34:33,581 SEVERE [Channel] Failed to execute command Pipe.EOF(22) (channel <host omitted>) java.lang.NullPointerException at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) 13:34:33,581 SEVERE [Channel] This command is created here Command Pipe.EOF(22) created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.ProxyWriter$EOF.<init>(ProxyWriter.java:169) at hudson.remoting.ProxyWriter.close(ProxyWriter.java:124) at hudson.remoting.ProxyWriter.finalize(ProxyWriter.java:132) at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83) at java.lang.ref.Finalizer.access$100(Finalizer.java:14) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160) 13:36:39,627 INFO [Run] Coeus-Local #31 main build action completed: SUCCESS /app/jboss/bin/run.sh: line 284: 20872 Killed "/app/java/bin/java" -Dprogram.name=run.sh -server -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs="/app/jboss/lib/endorsed" -classpath "/app/jboss/bin/run.jar:/app/java/lib/tools.jar" org.jboss.Main -c jenkins-ci -b <interface omitted>
In Eclipse, I get errors like the following. The Eclipse error happens every time but the Jenkins/JBoss crash doesn't always happen.
Message: Unexpected error: Unexpected response from Hudson server for '/jenkins/job/Coeus-Local/build': Created Exception Stack Trace: org.eclipse.mylyn.internal.hudson.core.client.HudsonException: Unexpected response from Hudson server for '/jenkins/job/Coeus-Local/build': Created at org.eclipse.mylyn.internal.hudson.core.client.HudsonOperation.validate(HudsonOperation.java:235) at org.eclipse.mylyn.internal.hudson.core.client.RestfulHudsonClient$7.doValidate(RestfulHudsonClient.java:359) at org.eclipse.mylyn.internal.hudson.core.client.HudsonOperation.processAndRelease(HudsonOperation.java:220) at org.eclipse.mylyn.internal.hudson.core.client.RestfulHudsonClient$7.execute(RestfulHudsonClient.java:353) at org.eclipse.mylyn.internal.hudson.core.client.HudsonOperation.run(HudsonOperation.java:167) at org.eclipse.mylyn.internal.hudson.core.client.RestfulHudsonClient.runBuild(RestfulHudsonClient.java:361) at org.eclipse.mylyn.internal.hudson.core.HudsonServerBehaviour.runBuild(HudsonServerBehaviour.java:791) at org.eclipse.mylyn.builds.internal.core.operations.RunBuildOperation$1.doExecute(RunBuildOperation.java:55) at org.eclipse.mylyn.builds.internal.core.operations.BuildJob.run(BuildJob.java:79) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53) Session Data: eclipse.buildId=4.3.2.M20140221-1700 java.version=1.7.0_45 java.vendor=Oracle Corporation BootLoader constants: OS=win32, ARCH=x86_64, WS=win32, NL=en_US Framework arguments: -product org.eclipse.epp.package.standard.product Command-line arguments: -os win32 -ws win32 -arch x86_64 -product org.eclipse.epp.package.standard.product
I don't know what the plugin is doing over the API to cause this but since it works sometimes but not others I would agree it is likely a race condition.
Regarding my previous comment. It was not because of a race condition. It was due to my VM being out of memory. I increased the VM memory from 2GB -> 16GB and it was resolved.
It happens to me as well:
- Jenkins ver. 1.532.2
- Ubuntu 12.04
SEVERE: Failed to execute command Pipe.EOF(165) (channel bob-10.69) java.lang.NullPointerException at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Mar 26, 2014 1:04:48 PM hudson.remoting.Channel$2 handle SEVERE: This command is created here Command Pipe.EOF(165) created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.ProxyWriter$EOF.<init>(ProxyWriter.java:169) at hudson.remoting.ProxyWriter.close(ProxyWriter.java:124) at hudson.remoting.ProxyWriter.finalize(ProxyWriter.java:132) at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101) at java.lang.ref.Finalizer.access$100(Finalizer.java:32) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190) Mar 26, 2014 1:04:48 PM hudson.remoting.Channel$2 handle SEVERE: Failed to execute command Pipe.EOF(386) (channel bob-10.16) java.lang.NullPointerException at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Mar 26, 2014 1:04:48 PM hudson.remoting.Channel$2 handle SEVERE: This command is created here Command Pipe.EOF(386) created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.ProxyWriter$EOF.<init>(ProxyWriter.java:169) at hudson.remoting.ProxyWriter.close(ProxyWriter.java:124) at hudson.remoting.ProxyWriter.finalize(ProxyWriter.java:132) at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101) at java.lang.ref.Finalizer.access$100(Finalizer.java:32) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
I found it interesting that this problem appears to be only seen with ProxyWriter, while more or less equivalent and more prevelant ProxyOutputStream do not show up in the stack.
+1
I am also experiencing the same issue with the ProxyWriter only mentioned in the stack, the same way as the ones in the earlier comments.
@kohsuke as noted in my eb3a4c3 commit comment I have observed this also in ProxyOutputStream.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/main/java/hudson/remoting/ProxyInputStream.java
src/main/java/hudson/remoting/ProxyOutputStream.java
http://jenkins-ci.org/commit/remoting/ec1a7c8e982d6cad5cbdd3d4b9ca3c06404f037c
Log:
Revert "JENKINS-20769 Trying to fix/suppress some related observed or actual NPEs."
This reverts commit eb3a4c380e695580174ce381a8330e8e2b8a9b2e.
Reverting this commit to unmask the problem.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/main/java/hudson/remoting/ProxyWriter.java
http://jenkins-ci.org/commit/remoting/d1e0fe07d8f72d37d9064404f94f7ae3bd6b223a
Log:
Revert "NPE observed in log while trying to reconnect a slave."
This reverts commit 622af099d6b94569c873b8fcd2d1bc65860a8cc6.
Simiarly, reverting to unmask the root cause of JENKINS-20769.
These problems are indicative of a deeper problem, and shouldn't
be just swept under the rug.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/test/java/hudson/remoting/ProxyWriterTest.java
http://jenkins-ci.org/commit/remoting/e498f5aa99af88d5d2e621fbb1e85b2d6306e61f
Log:
JENKINS-20769 adding a test case that reproduces the problem.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/main/java/hudson/remoting/ProxyOutputStream.java
src/main/java/hudson/remoting/ProxyWriter.java
http://jenkins-ci.org/commit/remoting/96c670de534de269bcde729a932655842a3aea2a
Log:
[FIXED JENKINS-20769]
When ProxyWriter is finalized, it should attempt to release a remote reference count,
but it shouldn't try to close the stream. For one, the other side that had exported
the writer might be still using it (imagine exporting System.out), and secondly, the
said object could have already been unexported, for example if the closure that brought
ProxyWriter to this side has already returned, since RemoteWriter is auto-unexporting.
Compare ProxyOutputStream impl vs ProxyWriter impl and the difference becomes clearer.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/main/java/hudson/remoting/ProxyWriter.java
http://jenkins-ci.org/commit/remoting/57c3f1ab9e866ae237a568bd57382b6fe900ab98
Log:
JENKINS-20769
Unexport is new, so for older versions fall back to EOF.
Better to have a bogus error message than memory leak.
I have fixed this problem with ProxyWriter. As noted in my comment of 96c670de534de269bcde729a932655842a3aea2a, the problem is that this EOF command was bogus. ProxyOutputStream handles this correctly, which explains why we see this on ProxyWriter but not on ProxyOutputStream (or at least less of, given jglick comment above that he has seen it with ProxyOutputStream.
The stack trace Jesse cites in eb3a4c3 is the following, and this is a very different pattern. This is not from EOF but for Flush, and it wasn't induced by a finalizer. This one smells a genuine reference counting error, and should be tracked separately:
java.lang.NullPointerException at hudson.remoting.ProxyOutputStream$Flush$1.run(ProxyOutputStream.java:305) at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:111)
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
pom.xml
http://jenkins-ci.org/commit/jenkins/11458c956e64673d99a9dc2c05cfd5f9533b4e1b
Log:
[JENKINS-20769 JENKINS-20204] integrated remoting 2.35 in the core.
just to update, I'm running "Jenkins ver. 1.558-SNAPSHOT (rc-04/01/2014 04:22 GMT-kohsuke)" which I assume includes this fix, I now see this in my logs:
SEVERE: Failed to execute command Pipe.Flush(10) (channel general (i-34cf7b17)) java.lang.ClassCastException: java.io.OutputStreamWriter cannot be cast to java.io.OutputStream at hudson.remoting.ProxyWriter$Flush.execute(ProxyWriter.java:285) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Apr 01, 2014 3:55:54 PM hudson.remoting.Channel$2 handle SEVERE: This command is created here Apr 01, 2014 3:57:13 PM hudson.remoting.Channel$2 handle SEVERE: Failed to execute command Pipe.Flush(10) (channel general (i-6ac87c49)) java.lang.ClassCastException: java.io.OutputStreamWriter cannot be cast to java.io.OutputStream at hudson.remoting.ProxyWriter$Flush.execute(ProxyWriter.java:285) at hudson.remoting.Channel$2.handle(Channel.java:461) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
not sure if this is related to the fix or not.
@zzzeek added a note to @kohsuke’s commit: https://github.com/jenkinsci/remoting/commit/cdbc40b8980d3ea528473daf7e44475c956a9a03#commitcomment-5870985
Resolved integrating remoting:2.36 in 75447b57b655e38fdb48f1e854a4b287071342cf.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
pom.xml
http://jenkins-ci.org/commit/jenkins/a89aa713b2ba5c325cd296d5733a565aa5cc63ec
Log:
[JENKINS-20769 JENKINS-20204] Integrating remoting 2.36
Fixed a bug Jesse found in remoting
(cherry picked from commit 75447b57b655e38fdb48f1e854a4b287071342cf)
Conflicts:
pom.xml
This will make it in the next LTS version, right?
Disregard - I found out what 1.554.1-fixed means Awaiting next LTS build release.
From scanning the changelog, 1.558 and later have the fix. LTS does not have it (not sure if it needs to be labeled lts-candidate to be picked up or not).
Update: added the label
Was this really fixed in 1.554.1 (that's what the label means per https://wiki.jenkins-ci.org/display/JENKINS/LTS+Release+Line)? The changelog doesn't mention it.
aje: The LTS changelogs are not reliable in this regard. You are right to say that fix went into 1.554.1. See cloudbees changlog that tends to be accurate: http://release-notes.cloudbees.com/release/Jenkins+OSS+LTS/1.554.1
Noticing the same here (Jenkins v1.541, using swarm plugin):