Details
-
Bug
-
Status: Closed (View Workflow)
-
Blocker
-
Resolution: Fixed
-
None
-
* Mac OS X
* Slave node running on same machine as Jenkins (connected via SSH to localhost, same 'jenkins' user running everything)
* Unity Cache server (running on same machine as Jenkins and Unity slave node)
Description
We get a pipe broken exception in the Jenkins Unity plugin about 50% of the time. The workspace is cleaned before every build, and compiled assets are fetched from a Unity Cache server running on the same machine.
The error happens ALWAYS at the exact same step:
[...]
Refresh: trashing asset 2197dcffa7f1e4295a0f8e83fed9f5bf
Refresh: trashing asset bca6eeff7c4924a429da9a1b3450b036
Refresh: trashing asset 6b4a9fff255e844c999a4bcd6105d894
Refresh: trashing asset d63cefffccda34624843a84caddbb852
Failure on remote
java.io.IOException: Pipe broken
at java.io.PipedInputStream.read(PipedInputStream.java:322)
at java.io.PipedInputStream.read(PipedInputStream.java:378)
at java.io.InputStream.read(InputStream.java:101)
at org.jenkinsci.plugins.unity3d.io.StreamCopyThread.run(StreamCopyThread.java:64)
When it succeeds, the next line is ALWAYS
Updating Assets - GUID: 00000000000000001000000000000000...
done. [Time: 18.743246 ms]
So IF the build fails, then it is ALWAYS right after the last asset being trashed. If it makes it to the next line (updating asset with GUID 00001000), then the build will ALWAYS succeed.
I think that the frequency of the error is dependent on how large the project is, as we didn't get the error nearly as much in the beginning phases of the project. Now that the project is large we get the pipe broken error about 50% of the time. This is of course conjecture and the frequency of the error could be caused by something else entirely.
Also, errors tend to happen multiple times in a row. Conversely, once it succeeds it tends to succeed for several consecutive builds.
This is of course very much a showstopper for us, so I'd be extremely grateful if someone could have a look at this issue.
Thanks!
- Tommi Kiviniemi
VP Technology, Co-founder
Seriously Digital Entertainment Ltd.
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-
Activity
Good news, I managed to find at least one reproducible scenario in which this error happens. It required a distributed built setup. More info later!
lacostej
added a comment - - edited Good news, I managed to find at least one reproducible scenario in which this error happens. It required a distributed built setup. More info later! lacostej
added a comment - I know the root cause and I have a fix that seems to pass my tests.
The root cause is when I wrote the plugin, I created my own piping mechanism because I was having issues with the Jenkins remoting one. When doing so, I went with using the remoting and java.io.Piped*Stream.
While the java.io.Piped*Stream classes work fine most of the time, they have some issues, including thread unfriendliness. See http://www.live-graph.org/javadoc/complete/org/LiveGraph/dataFile/read/PipedInputStream.html for some description.
As stated in the Javadoc for the java.io.PipedInputStream: A pipe is said to be broken if a thread that was providing data bytes to the connected piped output stream is no longer alive.
In the case of jenkins used in a distributed scenario, those threads are pooled (search threadPoolForRemoting in jenkins core). In particular, if the remote output isn't producing output for a while, this pooling seems to have more chance of happening. In this case, the writeSide thread is recycled/replaced. The next call to read detects the "pipe is broken".
So we cannot use the java.io.Piped*Stream in conjunction with Jenkins.
Now it seems that I can use the jenkins remoting FastPiped*Stream classes and everything work fine. So easy...
lacostej
added a comment - I know the root cause and I have a fix that seems to pass my tests.
The root cause is when I wrote the plugin, I created my own piping mechanism because I was having issues with the Jenkins remoting one. When doing so, I went with using the remoting and java.io.Piped*Stream.
See https://github.com/jenkinsci/unity3d-plugin/blob/b4d78b6fee8752c02eacb35c1b0416c55d9d76f5/src/main/java/org/jenkinsci/plugins/unity3d/io/Pipe.java#L35
While the java.io.Piped*Stream classes work fine most of the time, they have some issues, including thread unfriendliness. See http://www.live-graph.org/javadoc/complete/org/LiveGraph/dataFile/read/PipedInputStream.html for some description.
As stated in the Javadoc for the java.io.PipedInputStream: A pipe is said to be broken if a thread that was providing data bytes to the connected piped output stream is no longer alive.
In the case of jenkins used in a distributed scenario, those threads are pooled (search threadPoolForRemoting in jenkins core). In particular, if the remote output isn't producing output for a while, this pooling seems to have more chance of happening. In this case, the writeSide thread is recycled/replaced. The next call to read detects the "pipe is broken".
So we cannot use the java.io.Piped*Stream in conjunction with Jenkins.
Now it seems that I can use the jenkins remoting FastPiped*Stream classes and everything work fine. So easy... Code changed in jenkins
User: Jerome Lacoste
Path:
src/main/java/org/jenkinsci/plugins/unity3d/io/Pipe.java
http://jenkins-ci.org/commit/unity3d-plugin/dff4212ee1f3c9ec1ed54dda0f2788d0126d2fc1
Log:
[FIXED JENKINS-23958] use jenkins remoting piping streams for piping. java.io.Piped*Stream are not threads friendly and cause 'Pipe is broken' issue when jenkins pool the writing threads
Thanks Charles for the logs.
I don't know what's happening yet but this should help us a bit. I am surprised to not see more communication.
What can I see:
Here's a patch to apply to your remoting.jar https://github.com/lacostej/remoting/commit/02969e1271443afc933dcb8e0d472c1dfc9e8856 that could help knowing why the Death. Maybe worth a shot ?
This also should help a bit: https://wiki.jenkins-ci.org/display/JENKINS/Remoting+issue although Charles said he couldn't find any specific info in his slaves logs.