Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23232

Jenkins CLI: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A

      We use the jenkins-cli.jar to attach new slaves and this has been working successfully in v1.534, however when upgrading to v1.554.1, we are unable to do so and a "SEVERE: I/O error in channel Chunked connection" error occurs.

      Tested on linux slaves using java version "1.6.0_35" and "1.7.0_45" and the same error occurs.

      Attached console output to error.log

      Workaround: -Dhudson.diyChunking=false on both slave/agent and master sides

          [JENKINS-23232] Jenkins CLI: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A

          My guess is that this is because hudson.diyChunking is still set to true, and between Jetty and Jenkins we end up double-decoding chunked encoding. Needs to verify.

          Kohsuke Kawaguchi added a comment - My guess is that this is because hudson.diyChunking is still set to true, and between Jetty and Jenkins we end up double-decoding chunked encoding. Needs to verify.

          Ashok Manji added a comment -

          Thanks for the update Kohsuke. Looking forward to the resolution

          Ashok Manji added a comment - Thanks for the update Kohsuke. Looking forward to the resolution

          Daniel Beck added a comment -

          Ashok: What's the output of

          System.getProperty('hudson.diyChunking')

          in Manage Jenkins » Script Console?

          Daniel Beck added a comment - Ashok: What's the output of System .getProperty( 'hudson.diyChunking' ) in Manage Jenkins » Script Console?

          Ashok Manji added a comment -

          @Daniel Beck - Result: true

          Ashok Manji added a comment - @Daniel Beck - Result: true

          Peter Kline added a comment - - edited

          Possibly related – JENKINS-23223

          Peter Kline added a comment - - edited Possibly related – JENKINS-23223

          Daniel Beck added a comment -

          Ashok: Check your startup scripts (e.g. /etc/sysconfig/jenkins, /etc/init.d/jenkins, ... when using the RPM installer) for any occurrence of

          -Dhudson.diyChunking=true

          and remove it. Restart Jenkins using

          service restart jenkins

          (rather than via e.g. /safeRestart) so it picks up the changes. See whether the issue disappears.

          Daniel Beck added a comment - Ashok: Check your startup scripts (e.g. /etc/sysconfig/jenkins, /etc/init.d/jenkins, ... when using the RPM installer) for any occurrence of -Dhudson.diyChunking= true and remove it. Restart Jenkins using service restart jenkins (rather than via e.g. /safeRestart) so it picks up the changes. See whether the issue disappears.

          Peter Kline added a comment -

          By setting

          -Dhudson.diyChunking=false

          in /etc/sysconfig/jenkins and restarting I am now able to run remote build calls. Not sure if this fixes the remote slave issue but it certainly solves my issue when running the jenkins-cli.jar problems with Chunking

          Peter Kline added a comment - By setting -Dhudson.diyChunking=false in /etc/sysconfig/jenkins and restarting I am now able to run remote build calls. Not sure if this fixes the remote slave issue but it certainly solves my issue when running the jenkins-cli.jar problems with Chunking

          Daniel Beck added a comment -

          slave-setup is a plugin, removing that component.

          Daniel Beck added a comment - slave-setup is a plugin, removing that component.

          Ashok Manji added a comment -

          @Daniel - There were no occurrences of "-Dhudson.diyChunking=true" in my startup scripts, yet this is set as true (even though the default is false)
          I followed Peter's comment above to set this property to 'false' in /etc/sysconfig/jenkins and this has resolved the reported problem.

          Ashok Manji added a comment - @Daniel - There were no occurrences of "-Dhudson.diyChunking=true" in my startup scripts, yet this is set as true (even though the default is false) I followed Peter's comment above to set this property to 'false' in /etc/sysconfig/jenkins and this has resolved the reported problem.

          Jeff Geerling added a comment -

          In my case (1.574, CentOS 6.5), I was getting the same error, but it seems it was a case of DNS resolution failure with my URL. I had to explicitly map the IP of the server to the server's hostname (as used in the jenkins-cli command) and this error went away. The diyChunking property was false when I checked.

          Jeff Geerling added a comment - In my case (1.574, CentOS 6.5), I was getting the same error, but it seems it was a case of DNS resolution failure with my URL. I had to explicitly map the IP of the server to the server's hostname (as used in the jenkins-cli command) and this error went away. The diyChunking property was false when I checked.

          Daniel Beck added a comment -

          Given the recent responses, reducing priority.

          kohsuke: Is there still a bug left to be fixed? Or is this a doc issue? Or not a defect at all?

          Daniel Beck added a comment - Given the recent responses, reducing priority. kohsuke : Is there still a bug left to be fixed? Or is this a doc issue? Or not a defect at all?

          I've seen this error message when trying to use jenkins-cli pointing to a Jenkins master where the JNLP port is blocked by a firewall

          Daniel Serodio added a comment - I've seen this error message when trying to use jenkins-cli pointing to a Jenkins master where the JNLP port is blocked by a firewall

          Steve Jansen added a comment - - edited

          Not sure if this helps or not, but I faced a similar issue with this network topology:

          • LAN A (OpenStack cluster with a 192.168.2.0/24 internal network, NAT'd to the rest of the LAN via "floating" public IPs on 10.166.140.0/24)
          • Jenkins master @ 192.168.2.10 or 10.166.140.10
          • Jenkins slave 1 @ 192.168.2.11
          • Jenkins slave 2 @ 192.168.2.12
          • LAN B (3rd party data center, connected to LAN A via site-to-site VPN which NAT to 10.21.26.0/24 on LAN A)
          • Jenkins slave 3 @ 10.21.26.50
          • Jenkins slave 4 @ 10.21.26.51

          The CLI worked as expected when using the master node's private ip address but not the master node's public ip address.

          For example `java -jar jenkins-cli.jar -s http://192.168.2.10:8080/ help` worked, but `java -jar jenkins-cli.jar -s http://10.166.140.10:8080/ help` raised the error "SEVERE: I/O error in channel Chunked connection" on any slave.

          The problem was ultimately the software firewall in the OpenStack cluster that only applied to the floating IPs (e.g., 10.166.140.0/24).

          The resolution was to allowing UDP ingress on ports 1024-65535 and TCP ingress on ports 1024-65535 where the destination was the Jenkins master node and the source was my slaves public IPs (10.21.26.0/24 and 10.166.140.0/24).

          After the firewall change, executing `java -jar jenkins-cli.jar -s http://10.166.140.10:8080/ help` worked as expected on all slaves in both data centers.

          My $0.02 - there is some complex networking for CLI connections: there seems to be ICMP, UDP, and TCP in use at the same time, along with random (JNLP?) ports on the server. The CLI client could use connectivity checks and/or clearcut error messages to better explain when connectivity fails for a specific protocol/address/port.

          Steve Jansen added a comment - - edited Not sure if this helps or not, but I faced a similar issue with this network topology: LAN A (OpenStack cluster with a 192.168.2.0/24 internal network, NAT'd to the rest of the LAN via "floating" public IPs on 10.166.140.0/24) Jenkins master @ 192.168.2.10 or 10.166.140.10 Jenkins slave 1 @ 192.168.2.11 Jenkins slave 2 @ 192.168.2.12 LAN B (3rd party data center, connected to LAN A via site-to-site VPN which NAT to 10.21.26.0/24 on LAN A) Jenkins slave 3 @ 10.21.26.50 Jenkins slave 4 @ 10.21.26.51 The CLI worked as expected when using the master node's private ip address but not the master node's public ip address. For example `java -jar jenkins-cli.jar -s http://192.168.2.10:8080/ help` worked, but `java -jar jenkins-cli.jar -s http://10.166.140.10:8080/ help` raised the error "SEVERE: I/O error in channel Chunked connection" on any slave. The problem was ultimately the software firewall in the OpenStack cluster that only applied to the floating IPs (e.g., 10.166.140.0/24). The resolution was to allowing UDP ingress on ports 1024-65535 and TCP ingress on ports 1024-65535 where the destination was the Jenkins master node and the source was my slaves public IPs (10.21.26.0/24 and 10.166.140.0/24). After the firewall change, executing `java -jar jenkins-cli.jar -s http://10.166.140.10:8080/ help` worked as expected on all slaves in both data centers. My $0.02 - there is some complex networking for CLI connections: there seems to be ICMP, UDP, and TCP in use at the same time, along with random (JNLP?) ports on the server. The CLI client could use connectivity checks and/or clearcut error messages to better explain when connectivity fails for a specific protocol/address/port.

          Thanks Steve, it does help. However, we'd have a hard time asking our security team to open a firewall rule allowing so many ports (1024-65535). I don't understand the details of Jenkins CLI networking, is it possible to narrow the range of ports, or are they this much dynamic?

          Daniel Serodio added a comment - Thanks Steve, it does help. However, we'd have a hard time asking our security team to open a firewall rule allowing so many ports (1024-65535). I don't understand the details of Jenkins CLI networking, is it possible to narrow the range of ports, or are they this much dynamic?

          Daniel Beck added a comment -

          Doesn't it just use the port configured on the Configure Global Security page? (needs security enabled)

          Daniel Beck added a comment - Doesn't it just use the port configured on the Configure Global Security page? (needs security enabled)

          Steve Jansen added a comment -

          Hi Daniel & Daniel,

          I'd love to use a more specific firewall rule. Re: the Configure Global Security port, I think that port is only for JNLP. I have the default setting to use a random port for JNLP. Does a static port for JNLP also effect the CLI? That probably makes sense for the CLI to use JNLP...

          Cheers,
          Steve

          Steve Jansen added a comment - Hi Daniel & Daniel, I'd love to use a more specific firewall rule. Re: the Configure Global Security port, I think that port is only for JNLP. I have the default setting to use a random port for JNLP. Does a static port for JNLP also effect the CLI? That probably makes sense for the CLI to use JNLP... Cheers, Steve

          Daniel Beck added a comment - https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/Functions.java#L1869 https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/TcpSlaveAgentListener.java#L42

          cepnyci added a comment - - edited

          We experience the same issue with this network topology:

          Jenkins master @ 10.10.100.204
          Jenkins slave 1 @ 10.10.100.205
          Jenkins slave 2 @ 10.10.100.207
          Jenkins slave 3 @ 10.10.100.193

          on the same LAN.

          Jenkins ver. 1.584

          Oct 16, 2014 12:18:27 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
          SEVERE: I/O error in channel 10.10.100.207
          java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
                  at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782)
                  at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279)
                  at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
                  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          
          09:18:06 FATAL: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
          09:18:06 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
          09:18:06 	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
          09:18:06 	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
          09:18:06 	at hudson.remoting.Request.call(Request.java:174)
          09:18:06 	at hudson.remoting.Channel.call(Channel.java:742)
          09:18:06 	at hudson.FilePath.act(FilePath.java:907)
          09:18:06 	at hudson.FilePath.act(FilePath.java:896)
          09:18:06 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:910)
          09:18:06 	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:845)
          09:18:06 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1253)
          09:18:06 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
          09:18:06 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
          09:18:06 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
          09:18:06 	at hudson.model.Run.execute(Run.java:1754)
          09:18:06 	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529)
          09:18:06 	at hudson.model.ResourceController.execute(ResourceController.java:89)
          09:18:06 	at hudson.model.Executor.run(Executor.java:240)
          09:18:06 Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
          09:18:06 	at hudson.remoting.Request.abort(Request.java:299)
          09:18:06 	at hudson.remoting.Channel.terminate(Channel.java:805)
          09:18:06 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
          09:18:06 Caused by: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
          09:18:06 	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782)
          09:18:06 	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279)
          09:18:06 	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
          09:18:06 	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          09:18:06 	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          

          cepnyci added a comment - - edited We experience the same issue with this network topology: Jenkins master @ 10.10.100.204 Jenkins slave 1 @ 10.10.100.205 Jenkins slave 2 @ 10.10.100.207 Jenkins slave 3 @ 10.10.100.193 on the same LAN. Jenkins ver. 1.584 Oct 16, 2014 12:18:27 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel 10.10.100.207 java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) 09:18:06 FATAL: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41) 09:18:06 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34) 09:18:06 at hudson.remoting.Request.call(Request.java:174) 09:18:06 at hudson.remoting.Channel.call(Channel.java:742) 09:18:06 at hudson.FilePath.act(FilePath.java:907) 09:18:06 at hudson.FilePath.act(FilePath.java:896) 09:18:06 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:910) 09:18:06 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:845) 09:18:06 at hudson.model.AbstractProject.checkout(AbstractProject.java:1253) 09:18:06 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622) 09:18:06 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) 09:18:06 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528) 09:18:06 at hudson.model.Run.execute(Run.java:1754) 09:18:06 at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529) 09:18:06 at hudson.model.ResourceController.execute(ResourceController.java:89) 09:18:06 at hudson.model.Executor.run(Executor.java:240) 09:18:06 Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 at hudson.remoting.Request.abort(Request.java:299) 09:18:06 at hudson.remoting.Channel.terminate(Channel.java:805) 09:18:06 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) 09:18:06 Caused by: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782) 09:18:06 at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279) 09:18:06 at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) 09:18:06 at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) 09:18:06 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          Jan Hollevoet added a comment -

          We are at v1.559 and intensively use the dist-fork plugin. This error hit us severely. Since a fixed port is used for JNLP slave agents (Global Security), the error did not occur anymore.

          Jan Hollevoet added a comment - We are at v1.559 and intensively use the dist-fork plugin. This error hit us severely. Since a fixed port is used for JNLP slave agents (Global Security), the error did not occur anymore.

          Daniel Beck added a comment -

          cepnyci: The stream header is in hex digits and equals ASCII 'nloa'. Does that tell you anything?

          terrapin: What did you use before? Disabled or dynamic port?

          Daniel Beck added a comment - cepnyci : The stream header is in hex digits and equals ASCII 'nloa'. Does that tell you anything? terrapin : What did you use before? Disabled or dynamic port?

          Jan Hollevoet added a comment - - edited

          Daniel, I was using the dynamic port setting.

          Yesterday, I saw it once again on 480 launches of the cli:

          Nov 7, 2014 7:06:12 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
          SEVERE: I/O error in channel Chunked connection to http://192.168.100.61:8080/cli
          java.io.IOException: Unexpected termination of the channel

          I am also using the ip address instead of the hostname for the Jenkins server. Not sure whether that helped.

          And hudson.diyChunking = false

          Jan Hollevoet added a comment - - edited Daniel, I was using the dynamic port setting. Yesterday, I saw it once again on 480 launches of the cli: Nov 7, 2014 7:06:12 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel Chunked connection to http://192.168.100.61:8080/cli java.io.IOException: Unexpected termination of the channel I am also using the ip address instead of the hostname for the Jenkins server. Not sure whether that helped. And hudson.diyChunking = false

          Using FQDN fails and using 127.0.0.1 works for me too.

          Same error the others reported.

          Diego Woitasen added a comment - Using FQDN fails and using 127.0.0.1 works for me too. Same error the others reported.

          cepnyci added a comment - - edited

          Daniel, I think you meant ASCII "nload": unknown host. We will update the configuration accordingly and monitor the slaves.Thanks for the help.

          cepnyci added a comment - - edited Daniel, I think you meant ASCII "nload": unknown host. We will update the configuration accordingly and monitor the slaves.Thanks for the help.

          Daniel Beck added a comment -

          I got it straight from the error message you provided:

          Caused by: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61

          That hex part translates to 'nloa'. Unfortunately it only ever provides four characters.

          Daniel Beck added a comment - I got it straight from the error message you provided: Caused by: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 That hex part translates to 'nloa'. Unfortunately it only ever provides four characters.

          Patrick Wong added a comment - - edited

          I get the same issue with the following steps:

          • Client machine is CentOS 6.5 with Java 1.7.0-b71 (from CentOS package java-1.7.0-openjdk-devel)
          • Jenkins VM is CentOS 7 with Java 1.7.0-b71 (from CentOS package java-1.7.0-openjdk-devel)
            • port 8080 forwarded to host machine port 8081
            • Vagrant basebox is jayunit/centos7 with Oracle Virtualbox provider
          • jenkins-cli.jar comes from wget localhost:8081/jnlpjars/jenkins-cli.jar and it is then renamed to have the version string, e.g. mv jenkins-cli.jar jenkins-cli-1.580.2.jar
          • Command is java -jar jenkins-cli-1.580.2.jar -s http://localhost:8081/ list-plugins

          Have tried with both Jenkins 1.580.2 stable and 1.596 latest.

          Patrick Wong added a comment - - edited I get the same issue with the following steps: Client machine is CentOS 6.5 with Java 1.7.0-b71 (from CentOS package java-1.7.0-openjdk-devel) Jenkins VM is CentOS 7 with Java 1.7.0-b71 (from CentOS package java-1.7.0-openjdk-devel) port 8080 forwarded to host machine port 8081 Vagrant basebox is jayunit/centos7 with Oracle Virtualbox provider jenkins-cli.jar comes from wget localhost:8081/jnlpjars/jenkins-cli.jar and it is then renamed to have the version string, e.g. mv jenkins-cli.jar jenkins-cli-1.580.2.jar Command is java -jar jenkins-cli-1.580.2.jar -s http://localhost:8081/ list-plugins Have tried with both Jenkins 1.580.2 stable and 1.596 latest.

          Wilfred Hughes added a comment - - edited

          I've seen the same exception with a different header:

          Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 31352D30

          It turned out the cause was my Jenkins slave writing to the channel. My slave configuration was 'Launch slave via execution of the command on the master`, and my script set some JVM flags. The offending flags were:

          -XX:+PrintGCDetails -Xloggc:/foo/bar/jenkins_gc.log

          However, since the /foo/bar directory didn't exist, the JVM wrote GC logs to stdout. This resulted in the Stream seeing garbage. If you decode 31352D30 you get '15-0'. This is the start of a log line of the form:

          2015-01-23T14:01:04.789+0000: 26.069: [CMS-concurrent-sweep-start]

          My workaround was to change my slave to launch via SSH, but I fixed it by ensuring that my GC log directory exists.

          Wilfred Hughes added a comment - - edited I've seen the same exception with a different header: Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 31352D30 It turned out the cause was my Jenkins slave writing to the channel. My slave configuration was 'Launch slave via execution of the command on the master`, and my script set some JVM flags. The offending flags were: -XX:+PrintGCDetails -Xloggc:/foo/bar/jenkins_gc.log However, since the /foo/bar directory didn't exist, the JVM wrote GC logs to stdout. This resulted in the Stream seeing garbage. If you decode 31352D30 you get '15-0'. This is the start of a log line of the form: 2015-01-23T14:01:04.789+0000: 26.069: [CMS-concurrent-sweep-start] My workaround was to change my slave to launch via SSH, but I fixed it by ensuring that my GC log directory exists.

          cepnyci added a comment -

          We configured the slaves to connect via JNLP. No more experiencing this issue.

          cepnyci added a comment - We configured the slaves to connect via JNLP. No more experiencing this issue.

          Jan Hollevoet added a comment -

          Now I am at 1.565.3. Right after the upgrade, this error was still hitting us. Then I reconfigured all Windows slaves as JNLP nodes with the same revision of the client software (2.46). After this, the situation improved a lot. The number /usr/sbin/lsof | grep protocol | grep [jenkins-pid] | wc -l increased less fast than before. And the server is much more stable.

          I have the impression that the Windows clients are the root cause of the error. Maybe I should reconfigure them simply as SSH slaves.

          Jan Hollevoet added a comment - Now I am at 1.565.3. Right after the upgrade, this error was still hitting us. Then I reconfigured all Windows slaves as JNLP nodes with the same revision of the client software (2.46). After this, the situation improved a lot. The number /usr/sbin/lsof | grep protocol | grep [jenkins-pid] | wc -l increased less fast than before. And the server is much more stable. I have the impression that the Windows clients are the root cause of the error. Maybe I should reconfigure them simply as SSH slaves.

          Loic Dachary added a comment -

          This error shows up when the JNLP port (50000 typically) is blocked. It would be very helpful to display an error such as : unable to connect to port XXXX or something instead.

          SEVERE: I/O error in channel Chunked connection to http://127.0.0.1:11080/cli

          Loic Dachary added a comment - This error shows up when the JNLP port (50000 typically) is blocked. It would be very helpful to display an error such as : unable to connect to port XXXX or something instead. SEVERE: I/O error in channel Chunked connection to http://127.0.0.1:11080/cli

          Pere Orga added a comment -

          I got rid of this by enabling the TCP port for JNLP slave agents

          Thank you.

          Pere Orga added a comment - I got rid of this by enabling the TCP port for JNLP slave agents Thank you.

          Still happens randomly with 1.625.1 LTS when using jenkins-cli.jar to upload job config.xml file. Sometimes it fails after couple of requests sometimes after 20+.
          Server and client are running on CentOS 7.1
          I have -Dhudson.diyChunking=false in /etc/sysconfig/jenkins
          Enabling TCP port for JNLP slave agents seems to fix this or at least reduce the frequency.

          Slawomir Czarko added a comment - Still happens randomly with 1.625.1 LTS when using jenkins-cli.jar to upload job config.xml file. Sometimes it fails after couple of requests sometimes after 20+. Server and client are running on CentOS 7.1 I have -Dhudson.diyChunking=false in /etc/sysconfig/jenkins Enabling TCP port for JNLP slave agents seems to fix this or at least reduce the frequency.

          Jeff Feldstein added a comment - - edited

          I've been working with this all day today in both 1.6.25.2 and 1.638
          Centos 6.5
          jenkins-build-monitor plugin is installed
          If the default view is "All" in config.xml, jenkins-cli works.
          If the default view is changed to a view defined by jenkins-build-monitor, the following stack trace occurs:

          oot@zmon10 jenkins]# java -jar /usr/local/bin/jenkins-cli.jar -s http://localhost:8080 version
          Nov 13, 2015 1:50:10 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
          SEVERE: I/O error in channel Chunked connection to http://localhost:8080/cli
          java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A
          	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806)
          	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
          	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          
          hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A
          	at hudson.remoting.Request.abort(Request.java:297)
          	at hudson.remoting.Channel.terminate(Channel.java:847)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
          	at ......remote call to Chunked connection to http://localhost:8080/cli(Native Method)
          	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
          	at hudson.remoting.Request.call(Request.java:172)
          	at hudson.remoting.Channel.call(Channel.java:780)
          	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
          	at hudson.remoting.$Proxy1.waitForProperty(Unknown Source)
          	at hudson.remoting.Channel.waitForRemoteProperty(Channel.java:1258)
          	at hudson.cli.CLI.<init>(CLI.java:147)
          	at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:72)
          	at hudson.cli.CLI._main(CLI.java:479)
          	at hudson.cli.CLI.main(CLI.java:390)
          Caused by: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A
          	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806)
          	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
          	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
          

          This line in config.xml works:

           <primaryView>All</primaryView>
          

          This line in config.xml causes the stack trace above:

           <primaryView>SYNCl</primaryView>
          

          Jeff Feldstein added a comment - - edited I've been working with this all day today in both 1.6.25.2 and 1.638 Centos 6.5 jenkins-build-monitor plugin is installed If the default view is "All" in config.xml, jenkins-cli works. If the default view is changed to a view defined by jenkins-build-monitor, the following stack trace occurs: oot@zmon10 jenkins]# java -jar /usr/local/bin/jenkins-cli.jar -s http: //localhost:8080 version Nov 13, 2015 1:50:10 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel Chunked connection to http: //localhost:8080/cli java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A at hudson.remoting.Request.abort(Request.java:297) at hudson.remoting.Channel.terminate(Channel.java:847) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) at ......remote call to Chunked connection to http: //localhost:8080/cli(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) at hudson.remoting.Request.call(Request.java:172) at hudson.remoting.Channel.call(Channel.java:780) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250) at hudson.remoting.$Proxy1.waitForProperty(Unknown Source) at hudson.remoting.Channel.waitForRemoteProperty(Channel.java:1258) at hudson.cli.CLI.<init>(CLI.java:147) at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:72) at hudson.cli.CLI._main(CLI.java:479) at hudson.cli.CLI.main(CLI.java:390) Caused by: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) This line in config.xml works: <primaryView>All</primaryView> This line in config.xml causes the stack trace above: <primaryView>SYNCl</primaryView>

          Daniel Beck added a comment -

          Needs a release of executable-war with https://github.com/jenkinsci/extras-executable-war/commit/5a5f01362e7649da3b39a3a45940ba3ce1ab65e8

          oleg_nenashev Any idea whether you can release this?

          Daniel Beck added a comment - Needs a release of executable-war with https://github.com/jenkinsci/extras-executable-war/commit/5a5f01362e7649da3b39a3a45940ba3ce1ab65e8 oleg_nenashev Any idea whether you can release this?

          Oleg Nenashev added a comment -

          danielbeck Not sure, but I'll give a try. FTR JENKINS-34287

          Oleg Nenashev added a comment - danielbeck Not sure, but I'll give a try. FTR JENKINS-34287

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - PR is ready for review: https://github.com/jenkinsci/extras-executable-war/pull/7

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          src/main/java/Main.java
          http://jenkins-ci.org/commit/extras-executable-war/b12662aed40ee06a75c03a51a6fc87a75c642179
          Log:
          [JENKINS-34287,JENKINS-23232] Update the component towards 1.34 (#7)

          • Update to the latest Parent POM and Java 7
          • FindBugs: Propagate issues with file deletion
          • Pick parent POM 1.37 with Java 7 support
          • FindBugs: Fix resource leak when exception happens during the dependency list read
          • Downgrade the code to Java 6
          • Downgrade back to Java 1.3
          • Address comment from @rsandell. Just in case

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml src/main/java/Main.java http://jenkins-ci.org/commit/extras-executable-war/b12662aed40ee06a75c03a51a6fc87a75c642179 Log: [JENKINS-34287,JENKINS-23232] Update the component towards 1.34 (#7) Update to the latest Parent POM and Java 7 FindBugs: Propagate issues with file deletion Pick parent POM 1.37 with Java 7 support FindBugs: Fix resource leak when exception happens during the dependency list read Downgrade the code to Java 6 Downgrade back to Java 1.3 Address comment from @rsandell. Just in case

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/24443acb3c92612dc623849c9458000c98a0a265 Log: [JENKINS-34287,JENKINS-23232] Update executable-war to 1.34 Diff: https://github.com/jenkinsci/extras-executable-war/compare/executable-war-1.33...executable-war-1.34 https://issues.jenkins-ci.org/browse/JENKINS-23232 https://issues.jenkins-ci.org/browse/JENKINS-34287

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/d7598f118fd6d9989164cd2ff7126001a6ee2ff2 Log: [JENKINS-34287,JENKINS-23232] Update executable-war to 1.34 (#2565) Diff: https://github.com/jenkinsci/extras-executable-war/compare/executable-war-1.33...executable-war-1.34 https://issues.jenkins-ci.org/browse/JENKINS-23232 https://issues.jenkins-ci.org/browse/JENKINS-34287

          Oleg Nenashev added a comment -

          The fix is integrated towards 2.24

          Oleg Nenashev added a comment - The fix is integrated towards 2.24

            oleg_nenashev Oleg Nenashev
            ashokm Ashok Manji
            Votes:
            6 Vote for this issue
            Watchers:
            22 Start watching this issue

              Created:
              Updated:
              Resolved: