* Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
* Tomcat 7.0.22
* JDK 1.6.0_14
* Only ONE Master configuration - no slaves are configured
* 3 Executors - (one less than the max number of CPUs)
We reported an issue some time back that was also listed as fixed in Jenkins 1.441:
Log: [FIXED JENKINS-11130] SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar
Perhaps another bug should NOT be submitted so I have added the following comments below the line to the original defect 11130 comments section in case it can be reviewed/re-opened.
We did NOT try to make any adjustments to the Tomcat configuration:
Tomcat Connector connectionUploadTimeout
but we are also now seeing the same problem with Winstone when at this same 1.441 level. We did revert to the 1.438 version of the CLI (leaving the WAR at 1.441 running in Winstone) and that is serving asthe current workaround.
We have downloaded and installed the LATEST 1.441 release that lists the fix for this problem. Currently we were running 1.438 on Winstone only (since with Tomcat 6 or 7, we had experienced the error HOWEVER yet under Winstone, it worked OK so that was our workaround - while running 1.438).
Now with Jenkins 1.441 - we are getting the ERROR again and NOW WITH BOTH Winstone and the Tomcat configurations). We have left the Jenkins 1.441 WAR file in place running on Winstone, and reverted the CLI jar file back to the 1.438 version for now and that appears to work again with Winstone.
Checked Manifest of CLI jar downloaded with the 1.441 WAR installation:
Started by command line [workspace] $ /bin/bash -xe /opt/apache-tomcat-7.0.22_jenkins/temp/hudson32817888834817830.sh
+ /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-cli.jar -s http://11.22.33.44:8082/jenkins/ build XYZ_Project-SharedLibs -s -p SVN_PATH=trunk
Dec 5, 2011 12:59:11 PM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel Chunked connection to http://11.22.33.44:8082/jenkins/cli
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1115)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
Exception in thread "main" hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.call(Request.java:149)
at hudson.remoting.Channel.call(Channel.java:681)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
at $Proxy2.main(Unknown Source)
at hudson.cli.CLI.execute(CLI.java:200)
at hudson.cli.CLI._main(CLI.java:330)
at hudson.cli.CLI.main(CLI.java:245)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:273)
at hudson.remoting.Channel.terminate(Channel.java:732)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1139)
Caused by: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1115)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
Build step 'Execute shell' marked build as failure
Notifying upstream projects of job completion
Finished: FAILURE
Under Winstone, we get this stacktrace - it's somewhat different:
Started by command line [workspace] $ /bin/bash -xe /tmp/hudson10791816374444704.sh
+ /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-cli.jar -s http://11.22.33.44:8082/jenkins/ build XYZ_Project-SharedLibs -s -p SVN_PATH=trunk
Dec 5, 2011 1:18:22 PM hudson.remoting.Channel$ReaderThread run
SEVERE: I/O error in channel Chunked connection to http://11.22.33.44:8082/jenkins/cli
java.io.IOException: Premature EOF
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538) (http://www.http.ChunkedInputStream.readAheadBlocking%28ChunkedInputStream.java:538%29)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582) (http://www.http.ChunkedInputStream.readAhead%28ChunkedInputStream.java:582%29)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669) (http://www.http.ChunkedInputStream.read%28ChunkedInputStream.java:669%29)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2504) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2504%29)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2499) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2499%29)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2488) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2488%29)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
Exception in thread "main" hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Premature EOF
at hudson.remoting.Request.call(Request.java:149)
at hudson.remoting.Channel.call(Channel.java:681)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
at $Proxy2.main(Unknown Source)
at hudson.cli.CLI.execute(CLI.java:200)
at hudson.cli.CLI._main(CLI.java:330)
at hudson.cli.CLI.main(CLI.java:245)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Premature EOF
at hudson.remoting.Request.abort(Request.java:273)
at hudson.remoting.Channel.terminate(Channel.java:732)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1139)
Caused by: java.io.IOException: Premature EOF
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538) (http://www.http.ChunkedInputStream.readAheadBlocking%28ChunkedInputStream.java:538%29)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582) (http://www.http.ChunkedInputStream.readAhead%28ChunkedInputStream.java:582%29)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669) (http://www.http.ChunkedInputStream.read%28ChunkedInputStream.java:669%29)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2504) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2504%29)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2499) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2499%29)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2488) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2488%29)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
Build step 'Execute shell' marked build as failure
Notifying upstream projects of job completion
Finished: FAILURE
Use a ping timeout of 3/4 of the ping interval instead of the default
4 minute timeout.
This happens to workaround an issue in the remoting PingThread
implementation which causes the ping to busy wait for the timeout
period instead of keeping the channel alive.
SCM/JIRA link daemon
added a comment - Code changed in jenkins
User: Richard Mortimer
Path:
changelog.html
cli/src/main/java/hudson/cli/CLI.java
http://jenkins-ci.org/commit/jenkins/ea1b80aebe85a9414d5befd58e976d85818a258d
Log:
[FIXED JENKINS-12037] CLI - I/O error in channel Chunked connection
Use a ping timeout of 3/4 of the ping interval instead of the default
4 minute timeout.
This happens to workaround an issue in the remoting PingThread
implementation which causes the ping to busy wait for the timeout
period instead of keeping the channel alive.
I've committed a change to the CLI code that brings the execution timeout in line with the ping period. This has the side effect of fixing the problem in my test environment (however the cli ping thread will busy wait roughly 75% of the time).
Richard Mortimer
added a comment - I've committed a change to the CLI code that brings the execution timeout in line with the ping period. This has the side effect of fixing the problem in my test environment (however the cli ping thread will busy wait roughly 75% of the time).
In addition I've submitted pull request https://github.com/jenkinsci/remoting/pull/3 which will get rid of the busy waiting.
Richard Mortimer
added a comment - A snapshot build of 1.459 with the fix can be downloaded from
cli-1.459-SNAPSHOT-jar-with-dependencies.jar this should allow testing against recent version of Jenkins.
THANK YOU for this... it appears to be WORKING NOW . We are using your snapshot build below and the GA Jenkins 1.457 version (WAR) and running on Tomcat 7.0.22. (not Winstone anymore). The connection is not dropped and the jobs sequence runs to completion.
Is it safe to assume the fixed version will be part of the official 1.459 release?
We really appreciate your time and help with getting this fixed.
mark streit
added a comment - Richard
THANK YOU for this... it appears to be WORKING NOW . We are using your snapshot build below and the GA Jenkins 1.457 version (WAR) and running on Tomcat 7.0.22. (not Winstone anymore). The connection is not dropped and the jobs sequence runs to completion.
Is it safe to assume the fixed version will be part of the official 1.459 release?
We really appreciate your time and help with getting this fixed.
Yes it will be part of 1.459. The snapshot build that you tested is direct from Jenkins on Jenkins builds.
I mentioned it earlier but the current fix actually busy-waits during the ping cycle so you may see high cpu usage from the cli command. The busy-wait is fixed by the change that I have submitted to the underlying "remoting" library. That might take a release or two to filter through.
Finally we should really say thank you to George who managed to get a packet trace with the failure so that we could work out what was going on.
Richard Mortimer
added a comment - Yes it will be part of 1.459. The snapshot build that you tested is direct from Jenkins on Jenkins builds.
I mentioned it earlier but the current fix actually busy-waits during the ping cycle so you may see high cpu usage from the cli command. The busy-wait is fixed by the change that I have submitted to the underlying "remoting" library. That might take a release or two to filter through.
Finally we should really say thank you to George who managed to get a packet trace with the failure so that we could work out what was going on.
Hi,
We have just downloaded Jenkins 1.461 (because we had this problem with 1.443), and I still see such failures in catalina logs. we are using tomcat 6.0.29. will upgrade to Tomcat 7.0.22 help?
Moshe Sucaz
added a comment - Hi,
We have just downloaded Jenkins 1.461 (because we had this problem with 1.443), and I still see such failures in catalina logs. we are using tomcat 6.0.29. will upgrade to Tomcat 7.0.22 help?
1) you need to make sure you download and install the latest jenkins-cli.jar that comes with 1.460 or higher version of the WAR download...and have it in your classpath wherever you are utilizing the CLI jar.
2) It's the CLI jar code that was fixed as part of this. Simply updating the WAR file on your Tomcat instance is only part of the upgrade.
3) I checked our catalina.out logs on our Tomcat 7.0.22 installation and we don't see these anymore since the fix was delivered in 1.460 which is what we now run. We made sure wherever we use CLI commands that the 1.460 version of the jenkins-cli.jar is on the classpath.
Hope this helps.
mark streit
added a comment - A couple of things with this:
1) you need to make sure you download and install the latest jenkins-cli.jar that comes with 1.460 or higher version of the WAR download...and have it in your classpath wherever you are utilizing the CLI jar.
2) It's the CLI jar code that was fixed as part of this. Simply updating the WAR file on your Tomcat instance is only part of the upgrade.
3) I checked our catalina.out logs on our Tomcat 7.0.22 installation and we don't see these anymore since the fix was delivered in 1.460 which is what we now run. We made sure wherever we use CLI commands that the 1.460 version of the jenkins-cli.jar is on the classpath.
Hope this helps.
Code changed in jenkins
User: Richard Mortimer
Path:
src/main/java/hudson/remoting/PingThread.java http://jenkins-ci.org/commit/remoting/463b5427d272f5d0c2c97eb6c41ef95757eb6c15
Log:
Do not attempt to retry a channel get after an ExecutionException.
Without this the pinger busy waits retrying the same operation for the
timeout period. The timeout/retry mechanism was only intended to retry
pings that were interrupted.
This is part of the failure seen in JENKINS-12037 where the PingThread
busy waits for 4 minutes when the Ping class fails to execute on a restricted
channel.
SCM/JIRA link daemon
added a comment - Code changed in jenkins
User: Richard Mortimer
Path:
src/main/java/hudson/remoting/PingThread.java
http://jenkins-ci.org/commit/remoting/463b5427d272f5d0c2c97eb6c41ef95757eb6c15
Log:
Do not attempt to retry a channel get after an ExecutionException.
Without this the pinger busy waits retrying the same operation for the
timeout period. The timeout/retry mechanism was only intended to retry
pings that were interrupted.
This is part of the failure seen in JENKINS-12037 where the PingThread
busy waits for 4 minutes when the Ping class fails to execute on a restricted
channel.
[{"id":-1,"name":"My open issues","jql":"assignee = currentUser() AND resolution = Unresolved order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-2,"name":"Reported by me","jql":"reporter = currentUser() order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-4,"name":"All issues","jql":"order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-5,"name":"Open issues","jql":"resolution = Unresolved order by priority DESC,updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-9,"name":"Done issues","jql":"statusCategory = Done order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-3,"name":"Viewed recently","jql":"issuekey in issueHistory() order by lastViewed DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-6,"name":"Created recently","jql":"created >= -1w order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-7,"name":"Resolved recently","jql":"resolutiondate >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-8,"name":"Updated recently","jql":"updated >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false}]
Bug
Critical
Richard Mortimer
added a comment - 
jenkins_main_trunk #1624
mark streit
added a comment - 
. We are using your snapshot build below and the GA Jenkins 1.457 version (WAR) and running on Tomcat 7.0.22. (not Winstone anymore). The connection is not dropped and the jobs sequence runs to completion.
Moshe Sucaz
added a comment - 
Code changed in jenkins
User: Richard Mortimer
Path:
changelog.html
cli/src/main/java/hudson/cli/CLI.java
http://jenkins-ci.org/commit/jenkins/ea1b80aebe85a9414d5befd58e976d85818a258d
Log:
[FIXED JENKINS-12037] CLI - I/O error in channel Chunked connection
Use a ping timeout of 3/4 of the ping interval instead of the default
4 minute timeout.
This happens to workaround an issue in the remoting PingThread
implementation which causes the ping to busy wait for the timeout
period instead of keeping the channel alive.