Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12037

CLI - I/O error in channel Chunked connection/Unexpected termination of the channel - still occurring in Jenkins 1.449

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • cli
    • None
    • * Running on SLES9 Linux server with 4 CPUs and plenty of diskspace.
      * Tomcat 7.0.22
      * JDK 1.6.0_14
      * Only ONE Master configuration - no slaves are configured
      * 3 Executors - (one less than the max number of CPUs)

      We reported an issue some time back that was also listed as fixed in Jenkins 1.441:
      Log:
      [FIXED JENKINS-11130] SEVERE: I/O error in channel Chunked connection when using jenkins-cli.jar

      Perhaps another bug should NOT be submitted so I have added the following comments below the line to the original defect 11130 comments section in case it can be reviewed/re-opened.

      We did NOT try to make any adjustments to the Tomcat configuration:

      Tomcat Connector connectionUploadTimeout

      but we are also now seeing the same problem with Winstone when at this same 1.441 level. We did revert to the 1.438 version of the CLI (leaving the WAR at 1.441 running in Winstone) and that is serving asthe current workaround.

      ================================================================================================

      We have downloaded and installed the LATEST 1.441 release that lists the fix for this problem. Currently we were running 1.438 on Winstone only (since with Tomcat 6 or 7, we had experienced the error HOWEVER yet under Winstone, it worked OK so that was our workaround - while running 1.438).

      Now with Jenkins 1.441 - we are getting the ERROR again and NOW WITH BOTH Winstone and the Tomcat configurations). We have left the Jenkins 1.441 WAR file in place running on Winstone, and reverted the CLI jar file back to the 1.438 version for now and that appears to work again with Winstone.

      Checked Manifest of CLI jar downloaded with the 1.441 WAR installation:

      Manifest-Version: 1.0
      Archiver-Version: Plexus Archiver
      Created-By: Apache Maven
      Built-By: kohsuke
      Build-Jdk: 1.6.0_26
      Main-Class: hudson.cli.CLI
      Jenkins-CLI-Version: 1.441

      Under Tomcat 7, we get this stacktrace:

      Started by command line
      [workspace] $ /bin/bash -xe /opt/apache-tomcat-7.0.22_jenkins/temp/hudson32817888834817830.sh
      + /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-cli.jar -s http://11.22.33.44:8082/jenkins/ build XYZ_Project-SharedLibs -s -p SVN_PATH=trunk
      Dec 5, 2011 12:59:11 PM hudson.remoting.Channel$ReaderThread run
      SEVERE: I/O error in channel Chunked connection to http://11.22.33.44:8082/jenkins/cli
      java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1115)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
      Exception in thread "main" hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Request.call(Request.java:149)
      at hudson.remoting.Channel.call(Channel.java:681)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
      at $Proxy2.main(Unknown Source)
      at hudson.cli.CLI.execute(CLI.java:200)
      at hudson.cli.CLI._main(CLI.java:330)
      at hudson.cli.CLI.main(CLI.java:245)
      Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Request.abort(Request.java:273)
      at hudson.remoting.Channel.terminate(Channel.java:732)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1139)
      Caused by: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1115)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
      Build step 'Execute shell' marked build as failure
      Notifying upstream projects of job completion
      Finished: FAILURE

      Under Winstone, we get this stacktrace - it's somewhat different:

      Started by command line
      [workspace] $ /bin/bash -xe /tmp/hudson10791816374444704.sh
      + /opt/Sun/jdk1.6.0_14/bin/java -jar /opt/Sun/jdk1.6.0_14/lib/jenkins-cli.jar -s http://11.22.33.44:8082/jenkins/ build XYZ_Project-SharedLibs -s -p SVN_PATH=trunk
      Dec 5, 2011 1:18:22 PM hudson.remoting.Channel$ReaderThread run
      SEVERE: I/O error in channel Chunked connection to http://11.22.33.44:8082/jenkins/cli
      java.io.IOException: Premature EOF
      at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538) (http://www.http.ChunkedInputStream.readAheadBlocking%28ChunkedInputStream.java:538%29)
      at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582) (http://www.http.ChunkedInputStream.readAhead%28ChunkedInputStream.java:582%29)
      at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669) (http://www.http.ChunkedInputStream.read%28ChunkedInputStream.java:669%29)
      at java.io.FilterInputStream.read(FilterInputStream.java:116)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2504) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2504%29)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2499) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2499%29)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2488) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2488%29)
      at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
      at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
      Exception in thread "main" hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Premature EOF
      at hudson.remoting.Request.call(Request.java:149)
      at hudson.remoting.Channel.call(Channel.java:681)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
      at $Proxy2.main(Unknown Source)
      at hudson.cli.CLI.execute(CLI.java:200)
      at hudson.cli.CLI._main(CLI.java:330)
      at hudson.cli.CLI.main(CLI.java:245)
      Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Premature EOF
      at hudson.remoting.Request.abort(Request.java:273)
      at hudson.remoting.Channel.terminate(Channel.java:732)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1139)
      Caused by: java.io.IOException: Premature EOF
      at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:538) (http://www.http.ChunkedInputStream.readAheadBlocking%28ChunkedInputStream.java:538%29)
      at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:582) (http://www.http.ChunkedInputStream.readAhead%28ChunkedInputStream.java:582%29)
      at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:669) (http://www.http.ChunkedInputStream.read%28ChunkedInputStream.java:669%29)
      at java.io.FilterInputStream.read(FilterInputStream.java:116)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2504) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2504%29)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2499) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2499%29)
      at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2488) (http://www.protocol.http.HttpURLConnection$HttpInputStream.read%28HttpURLConnection.java:2488%29)
      at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
      at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2542)
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2552)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at hudson.remoting.Channel$ReaderThread.run(Channel.java:1109)
      Build step 'Execute shell' marked build as failure
      Notifying upstream projects of job completion
      Finished: FAILURE

        1. jenkins-timeout
          39 kB
          George Panayotov

          [JENKINS-12037] CLI - I/O error in channel Chunked connection/Unexpected termination of the channel - still occurring in Jenkins 1.449

          Code changed in jenkins
          User: Richard Mortimer
          Path:
          changelog.html
          cli/src/main/java/hudson/cli/CLI.java
          http://jenkins-ci.org/commit/jenkins/ea1b80aebe85a9414d5befd58e976d85818a258d
          Log:
          [FIXED JENKINS-12037] CLI - I/O error in channel Chunked connection

          Use a ping timeout of 3/4 of the ping interval instead of the default
          4 minute timeout.
          This happens to workaround an issue in the remoting PingThread
          implementation which causes the ping to busy wait for the timeout
          period instead of keeping the channel alive.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: changelog.html cli/src/main/java/hudson/cli/CLI.java http://jenkins-ci.org/commit/jenkins/ea1b80aebe85a9414d5befd58e976d85818a258d Log: [FIXED JENKINS-12037] CLI - I/O error in channel Chunked connection Use a ping timeout of 3/4 of the ping interval instead of the default 4 minute timeout. This happens to workaround an issue in the remoting PingThread implementation which causes the ping to busy wait for the timeout period instead of keeping the channel alive.

          I've committed a change to the CLI code that brings the execution timeout in line with the ping period. This has the side effect of fixing the problem in my test environment (however the cli ping thread will busy wait roughly 75% of the time).

          In addition I've submitted pull request https://github.com/jenkinsci/remoting/pull/3 which will get rid of the busy waiting.

          Richard Mortimer added a comment - I've committed a change to the CLI code that brings the execution timeout in line with the ping period. This has the side effect of fixing the problem in my test environment (however the cli ping thread will busy wait roughly 75% of the time). In addition I've submitted pull request https://github.com/jenkinsci/remoting/pull/3 which will get rid of the busy waiting.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #1624
          [FIXED JENKINS-12037] CLI - I/O error in channel Chunked connection (Revision ea1b80aebe85a9414d5befd58e976d85818a258d)

          Result = SUCCESS
          richm : ea1b80aebe85a9414d5befd58e976d85818a258d
          Files :

          • cli/src/main/java/hudson/cli/CLI.java
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #1624 [FIXED JENKINS-12037] CLI - I/O error in channel Chunked connection (Revision ea1b80aebe85a9414d5befd58e976d85818a258d) Result = SUCCESS richm : ea1b80aebe85a9414d5befd58e976d85818a258d Files : cli/src/main/java/hudson/cli/CLI.java changelog.html

          A snapshot build of 1.459 with the fix can be downloaded from
          cli-1.459-SNAPSHOT-jar-with-dependencies.jar this should allow testing against recent version of Jenkins.

          Richard Mortimer added a comment - A snapshot build of 1.459 with the fix can be downloaded from cli-1.459-SNAPSHOT-jar-with-dependencies.jar this should allow testing against recent version of Jenkins.

          mark streit added a comment -

          Richard

          THANK YOU for this... it appears to be WORKING NOW . We are using your snapshot build below and the GA Jenkins 1.457 version (WAR) and running on Tomcat 7.0.22. (not Winstone anymore). The connection is not dropped and the jobs sequence runs to completion.

          Is it safe to assume the fixed version will be part of the official 1.459 release?

          We really appreciate your time and help with getting this fixed.

          mark streit added a comment - Richard THANK YOU for this... it appears to be WORKING NOW . We are using your snapshot build below and the GA Jenkins 1.457 version (WAR) and running on Tomcat 7.0.22. (not Winstone anymore). The connection is not dropped and the jobs sequence runs to completion. Is it safe to assume the fixed version will be part of the official 1.459 release? We really appreciate your time and help with getting this fixed.

          Yes it will be part of 1.459. The snapshot build that you tested is direct from Jenkins on Jenkins builds.

          I mentioned it earlier but the current fix actually busy-waits during the ping cycle so you may see high cpu usage from the cli command. The busy-wait is fixed by the change that I have submitted to the underlying "remoting" library. That might take a release or two to filter through.

          Finally we should really say thank you to George who managed to get a packet trace with the failure so that we could work out what was going on.

          Richard Mortimer added a comment - Yes it will be part of 1.459. The snapshot build that you tested is direct from Jenkins on Jenkins builds. I mentioned it earlier but the current fix actually busy-waits during the ping cycle so you may see high cpu usage from the cli command. The busy-wait is fixed by the change that I have submitted to the underlying "remoting" library. That might take a release or two to filter through. Finally we should really say thank you to George who managed to get a packet trace with the failure so that we could work out what was going on.

          dogfood added a comment -

          Integrated in jenkins_ui-changes_branch #21
          [FIXED JENKINS-12037] CLI - I/O error in channel Chunked connection (Revision ea1b80aebe85a9414d5befd58e976d85818a258d)

          Result = SUCCESS
          richm : ea1b80aebe85a9414d5befd58e976d85818a258d
          Files :

          • cli/src/main/java/hudson/cli/CLI.java
          • changelog.html

          dogfood added a comment - Integrated in jenkins_ui-changes_branch #21 [FIXED JENKINS-12037] CLI - I/O error in channel Chunked connection (Revision ea1b80aebe85a9414d5befd58e976d85818a258d) Result = SUCCESS richm : ea1b80aebe85a9414d5befd58e976d85818a258d Files : cli/src/main/java/hudson/cli/CLI.java changelog.html

          Moshe Sucaz added a comment -

          Hi,
          We have just downloaded Jenkins 1.461 (because we had this problem with 1.443), and I still see such failures in catalina logs. we are using tomcat 6.0.29. will upgrade to Tomcat 7.0.22 help?

          Moshe Sucaz added a comment - Hi, We have just downloaded Jenkins 1.461 (because we had this problem with 1.443), and I still see such failures in catalina logs. we are using tomcat 6.0.29. will upgrade to Tomcat 7.0.22 help?

          mark streit added a comment -

          A couple of things with this:

          1) you need to make sure you download and install the latest jenkins-cli.jar that comes with 1.460 or higher version of the WAR download...and have it in your classpath wherever you are utilizing the CLI jar.
          2) It's the CLI jar code that was fixed as part of this. Simply updating the WAR file on your Tomcat instance is only part of the upgrade.
          3) I checked our catalina.out logs on our Tomcat 7.0.22 installation and we don't see these anymore since the fix was delivered in 1.460 which is what we now run. We made sure wherever we use CLI commands that the 1.460 version of the jenkins-cli.jar is on the classpath.

          Hope this helps.

          mark streit added a comment - A couple of things with this: 1) you need to make sure you download and install the latest jenkins-cli.jar that comes with 1.460 or higher version of the WAR download...and have it in your classpath wherever you are utilizing the CLI jar. 2) It's the CLI jar code that was fixed as part of this. Simply updating the WAR file on your Tomcat instance is only part of the upgrade. 3) I checked our catalina.out logs on our Tomcat 7.0.22 installation and we don't see these anymore since the fix was delivered in 1.460 which is what we now run. We made sure wherever we use CLI commands that the 1.460 version of the jenkins-cli.jar is on the classpath. Hope this helps.

          Code changed in jenkins
          User: Richard Mortimer
          Path:
          src/main/java/hudson/remoting/PingThread.java
          http://jenkins-ci.org/commit/remoting/463b5427d272f5d0c2c97eb6c41ef95757eb6c15
          Log:
          Do not attempt to retry a channel get after an ExecutionException.
          Without this the pinger busy waits retrying the same operation for the
          timeout period. The timeout/retry mechanism was only intended to retry
          pings that were interrupted.

          This is part of the failure seen in JENKINS-12037 where the PingThread
          busy waits for 4 minutes when the Ping class fails to execute on a restricted
          channel.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Richard Mortimer Path: src/main/java/hudson/remoting/PingThread.java http://jenkins-ci.org/commit/remoting/463b5427d272f5d0c2c97eb6c41ef95757eb6c15 Log: Do not attempt to retry a channel get after an ExecutionException. Without this the pinger busy waits retrying the same operation for the timeout period. The timeout/retry mechanism was only intended to retry pings that were interrupted. This is part of the failure seen in JENKINS-12037 where the PingThread busy waits for 4 minutes when the Ping class fails to execute on a restricted channel.

            Unassigned Unassigned
            mcs13099 mark streit
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: