-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
Platform: All, OS: All
The artifact transfer is currently a 3-4x penalty for the project that I am
working on. I have reproduced the issue with a simple test pom that does
nothing but jar hudson.war. I performed this test on a heterogeneous
environment. Both master and slave are running Fedora 10, but the master is a
faster machine. Still, it highlights the issue.
Here are some stats (all stats are after caching dependencies in the local repos):
Master build through Hudson: 19s
Master build from command line (no Hudson): 9s
Slave build through Hudson: 1m46s
Slave build from command line (no Hudson): 16s
To be fair we should at least add time to do a straight scp of the artifact from
slave to master. The two nodes share a 100 Mbit switch:
$ scp target/slow-rider-1.0.0-SNAPSHOT.jar master_node:
slow-rider-1.0.0NAPSHOT.jar 100% 25MB 12.7MB/s 00:02
Of course this example exaggerates the issue to make it more clear but not by
too much. I originally noticed this in a completely separate environment that
was all virtual. I reproduced this on two physical machines using a different
switch and different ethernet drivers (both virtual and physical). The
reproducibility plus the comparison against command line + scp leads me to
suspect eager flushing.
- is related to
-
JENKINS-3799 Slave-to-master copies can be extremely slow
-
- Resolved
-
-
JENKINS-7813 Archiving artifacts very slow
-
- Resolved
-
-
JENKINS-7921 Archiving very slow between slave and master in unix driven by ssh
-
- Resolved
-
-
JENKINS-3524 sending artifacts to master is very slow
-
- Closed
-
For me (FreeBSD), it is locked in next stack trace:
"Channel reader thread: Channel to Maven [/usr/local/openjdk6//bin/java, -cp, /usr/home/builder/jenkins/builder/maven-agent.jar:/usr/home/builder/jenkins/builder/classworlds.jar, hudson.maven.agent.Main, /usr/local/share/java/maven2, /usr/home/builder/jenkins/builder/slave.jar, /usr/home/builder/jenkins/builder/maven-interceptor.jar, 40088, /usr/home/builder/jenkins/builder/maven2.1-interceptor.jar] / waiting for hudson.remoting.Channel@77cd18d:builder" prio=5 tid=0x0000000851778000 nid=0x84e460740 in Object.wait() [0x00007ffffa9ac000..0x00007ffffa9ac920]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at hudson.remoting.Request.call(Request.java:127)
at hudson.remoting.ProxyInputStream._read(ProxyInputStream.java:74)
at hudson.remoting.ProxyInputStream.read(ProxyInputStream.java:80)
at hudson.remoting.RemoteInputStream.read(RemoteInputStream.java:91)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2264)
at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2666)
at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2696)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1648)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1945)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1869)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087)
Looking at the code, there is a comment here:
// I don't know exactly when this can happen, as pendingCalls are cleaned up by Channel,
// but in production I've observed that in rare occasion it can block forever, even after a channel
// is gone. So be defensive against that.
wait(30*1000);
It seems that it is time to get when this does occur