We use the jenkins-cli.jar to attach new slaves and this has been working successfully in v1.534, however when upgrading to v1.554.1, we are unable to do so and a "SEVERE: I/O error in channel Chunked connection" error occurs.
Tested on linux slaves using java version "1.6.0_35" and "1.7.0_45" and the same error occurs.
Attached console output to error.log
Workaround: -Dhudson.diyChunking=false on both slave/agent and master sides
- is duplicated by
-
JENKINS-34287 [2.0] CLI over HTTP doesn't work
-
- Resolved
-
-
JENKINS-35197 unable to use jenkins-cli: java.io.StreamCorruptedException
-
- Resolved
-
- is related to
-
JENKINS-20128 HTTP two-way remoting does not work (jenkins-cli.jar without JNLP)
-
- Resolved
-
-
JENKINS-23223 Jenkins CLI to start a build no longer working in 1.564
-
- Resolved
-
[JENKINS-23232] Jenkins CLI: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A
Ashok: What's the output of
System.getProperty('hudson.diyChunking')
in Manage Jenkins » Script Console?
Ashok: Check your startup scripts (e.g. /etc/sysconfig/jenkins, /etc/init.d/jenkins, ... when using the RPM installer) for any occurrence of
-Dhudson.diyChunking=true
and remove it. Restart Jenkins using
service restart jenkins
(rather than via e.g. /safeRestart) so it picks up the changes. See whether the issue disappears.
By setting
-Dhudson.diyChunking=false
in /etc/sysconfig/jenkins and restarting I am now able to run remote build calls. Not sure if this fixes the remote slave issue but it certainly solves my issue when running the jenkins-cli.jar problems with Chunking
@Daniel - There were no occurrences of "-Dhudson.diyChunking=true" in my startup scripts, yet this is set as true (even though the default is false)
I followed Peter's comment above to set this property to 'false' in /etc/sysconfig/jenkins and this has resolved the reported problem.
In my case (1.574, CentOS 6.5), I was getting the same error, but it seems it was a case of DNS resolution failure with my URL. I had to explicitly map the IP of the server to the server's hostname (as used in the jenkins-cli command) and this error went away. The diyChunking property was false when I checked.
Given the recent responses, reducing priority.
kohsuke: Is there still a bug left to be fixed? Or is this a doc issue? Or not a defect at all?
I've seen this error message when trying to use jenkins-cli pointing to a Jenkins master where the JNLP port is blocked by a firewall
Not sure if this helps or not, but I faced a similar issue with this network topology:
- LAN A (OpenStack cluster with a 192.168.2.0/24 internal network, NAT'd to the rest of the LAN via "floating" public IPs on 10.166.140.0/24)
- Jenkins master @ 192.168.2.10 or 10.166.140.10
- Jenkins slave 1 @ 192.168.2.11
- Jenkins slave 2 @ 192.168.2.12
- LAN B (3rd party data center, connected to LAN A via site-to-site VPN which NAT to 10.21.26.0/24 on LAN A)
- Jenkins slave 3 @ 10.21.26.50
- Jenkins slave 4 @ 10.21.26.51
The CLI worked as expected when using the master node's private ip address but not the master node's public ip address.
For example `java -jar jenkins-cli.jar -s http://192.168.2.10:8080/ help` worked, but `java -jar jenkins-cli.jar -s http://10.166.140.10:8080/ help` raised the error "SEVERE: I/O error in channel Chunked connection" on any slave.
The problem was ultimately the software firewall in the OpenStack cluster that only applied to the floating IPs (e.g., 10.166.140.0/24).
The resolution was to allowing UDP ingress on ports 1024-65535 and TCP ingress on ports 1024-65535 where the destination was the Jenkins master node and the source was my slaves public IPs (10.21.26.0/24 and 10.166.140.0/24).
After the firewall change, executing `java -jar jenkins-cli.jar -s http://10.166.140.10:8080/ help` worked as expected on all slaves in both data centers.
My $0.02 - there is some complex networking for CLI connections: there seems to be ICMP, UDP, and TCP in use at the same time, along with random (JNLP?) ports on the server. The CLI client could use connectivity checks and/or clearcut error messages to better explain when connectivity fails for a specific protocol/address/port.
Thanks Steve, it does help. However, we'd have a hard time asking our security team to open a firewall rule allowing so many ports (1024-65535). I don't understand the details of Jenkins CLI networking, is it possible to narrow the range of ports, or are they this much dynamic?
Doesn't it just use the port configured on the Configure Global Security page? (needs security enabled)
Hi Daniel & Daniel,
I'd love to use a more specific firewall rule. Re: the Configure Global Security port, I think that port is only for JNLP. I have the default setting to use a random port for JNLP. Does a static port for JNLP also effect the CLI? That probably makes sense for the CLI to use JNLP...
Cheers,
Steve
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/Functions.java#L1869
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/TcpSlaveAgentListener.java#L42
We experience the same issue with this network topology:
Jenkins master @ 10.10.100.204
Jenkins slave 1 @ 10.10.100.205
Jenkins slave 2 @ 10.10.100.207
Jenkins slave 3 @ 10.10.100.193
on the same LAN.
Jenkins ver. 1.584
Oct 16, 2014 12:18:27 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel 10.10.100.207 java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
09:18:06 FATAL: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41) 09:18:06 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34) 09:18:06 at hudson.remoting.Request.call(Request.java:174) 09:18:06 at hudson.remoting.Channel.call(Channel.java:742) 09:18:06 at hudson.FilePath.act(FilePath.java:907) 09:18:06 at hudson.FilePath.act(FilePath.java:896) 09:18:06 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:910) 09:18:06 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:845) 09:18:06 at hudson.model.AbstractProject.checkout(AbstractProject.java:1253) 09:18:06 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622) 09:18:06 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) 09:18:06 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528) 09:18:06 at hudson.model.Run.execute(Run.java:1754) 09:18:06 at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529) 09:18:06 at hudson.model.ResourceController.execute(ResourceController.java:89) 09:18:06 at hudson.model.Executor.run(Executor.java:240) 09:18:06 Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 at hudson.remoting.Request.abort(Request.java:299) 09:18:06 at hudson.remoting.Channel.terminate(Channel.java:805) 09:18:06 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) 09:18:06 Caused by: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61 09:18:06 at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:782) 09:18:06 at java.io.ObjectInputStream.<init>(ObjectInputStream.java:279) 09:18:06 at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40) 09:18:06 at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) 09:18:06 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
We are at v1.559 and intensively use the dist-fork plugin. This error hit us severely. Since a fixed port is used for JNLP slave agents (Global Security), the error did not occur anymore.
cepnyci: The stream header is in hex digits and equals ASCII 'nloa'. Does that tell you anything?
terrapin: What did you use before? Disabled or dynamic port?
Daniel, I was using the dynamic port setting.
Yesterday, I saw it once again on 480 launches of the cli:
Nov 7, 2014 7:06:12 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel Chunked connection to http://192.168.100.61:8080/cli
java.io.IOException: Unexpected termination of the channel
I am also using the ip address instead of the hostname for the Jenkins server. Not sure whether that helped.
And hudson.diyChunking = false
Using FQDN fails and using 127.0.0.1 works for me too.
Same error the others reported.
Daniel, I think you meant ASCII "nload": unknown host. We will update the configuration accordingly and monitor the slaves.Thanks for the help.
I got it straight from the error message you provided:
Caused by: java.io.StreamCorruptedException: invalid stream header: 6E6C6F61
That hex part translates to 'nloa'. Unfortunately it only ever provides four characters.
I get the same issue with the following steps:
- Client machine is CentOS 6.5 with Java 1.7.0-b71 (from CentOS package java-1.7.0-openjdk-devel)
- Jenkins VM is CentOS 7 with Java 1.7.0-b71 (from CentOS package java-1.7.0-openjdk-devel)
- port 8080 forwarded to host machine port 8081
- Vagrant basebox is jayunit/centos7 with Oracle Virtualbox provider
- jenkins-cli.jar comes from wget localhost:8081/jnlpjars/jenkins-cli.jar and it is then renamed to have the version string, e.g. mv jenkins-cli.jar jenkins-cli-1.580.2.jar
- Command is java -jar jenkins-cli-1.580.2.jar -s http://localhost:8081/ list-plugins
Have tried with both Jenkins 1.580.2 stable and 1.596 latest.
I've seen the same exception with a different header:
Caused by: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 31352D30
It turned out the cause was my Jenkins slave writing to the channel. My slave configuration was 'Launch slave via execution of the command on the master`, and my script set some JVM flags. The offending flags were:
-XX:+PrintGCDetails -Xloggc:/foo/bar/jenkins_gc.log
However, since the /foo/bar directory didn't exist, the JVM wrote GC logs to stdout. This resulted in the Stream seeing garbage. If you decode 31352D30 you get '15-0'. This is the start of a log line of the form:
2015-01-23T14:01:04.789+0000: 26.069: [CMS-concurrent-sweep-start]
My workaround was to change my slave to launch via SSH, but I fixed it by ensuring that my GC log directory exists.
We configured the slaves to connect via JNLP. No more experiencing this issue.
Now I am at 1.565.3. Right after the upgrade, this error was still hitting us. Then I reconfigured all Windows slaves as JNLP nodes with the same revision of the client software (2.46). After this, the situation improved a lot. The number /usr/sbin/lsof | grep protocol | grep [jenkins-pid] | wc -l increased less fast than before. And the server is much more stable.
I have the impression that the Windows clients are the root cause of the error. Maybe I should reconfigure them simply as SSH slaves.
This error shows up when the JNLP port (50000 typically) is blocked. It would be very helpful to display an error such as : unable to connect to port XXXX or something instead.
SEVERE: I/O error in channel Chunked connection to http://127.0.0.1:11080/cli
I got rid of this by enabling the TCP port for JNLP slave agents
Thank you.
Still happens randomly with 1.625.1 LTS when using jenkins-cli.jar to upload job config.xml file. Sometimes it fails after couple of requests sometimes after 20+.
Server and client are running on CentOS 7.1
I have -Dhudson.diyChunking=false in /etc/sysconfig/jenkins
Enabling TCP port for JNLP slave agents seems to fix this or at least reduce the frequency.
I've been working with this all day today in both 1.6.25.2 and 1.638
Centos 6.5
jenkins-build-monitor plugin is installed
If the default view is "All" in config.xml, jenkins-cli works.
If the default view is changed to a view defined by jenkins-build-monitor, the following stack trace occurs:
oot@zmon10 jenkins]# java -jar /usr/local/bin/jenkins-cli.jar -s http://localhost:8080 version Nov 13, 2015 1:50:10 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel Chunked connection to http://localhost:8080/cli java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A at hudson.remoting.Request.abort(Request.java:297) at hudson.remoting.Channel.terminate(Channel.java:847) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) at ......remote call to Chunked connection to http://localhost:8080/cli(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) at hudson.remoting.Request.call(Request.java:172) at hudson.remoting.Channel.call(Channel.java:780) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250) at hudson.remoting.$Proxy1.waitForProperty(Unknown Source) at hudson.remoting.Channel.waitForRemoteProperty(Channel.java:1258) at hudson.cli.CLI.<init>(CLI.java:147) at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:72) at hudson.cli.CLI._main(CLI.java:479) at hudson.cli.CLI.main(CLI.java:390) Caused by: java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
This line in config.xml works:
<primaryView>All</primaryView>
This line in config.xml causes the stack trace above:
<primaryView>SYNCl</primaryView>
Needs a release of executable-war with https://github.com/jenkinsci/extras-executable-war/commit/5a5f01362e7649da3b39a3a45940ba3ce1ab65e8
oleg_nenashev Any idea whether you can release this?
PR is ready for review: https://github.com/jenkinsci/extras-executable-war/pull/7
Code changed in jenkins
User: Oleg Nenashev
Path:
pom.xml
src/main/java/Main.java
http://jenkins-ci.org/commit/extras-executable-war/b12662aed40ee06a75c03a51a6fc87a75c642179
Log:
[JENKINS-34287,JENKINS-23232] Update the component towards 1.34 (#7)
- Update to the latest Parent POM and Java 7
- FindBugs: Propagate issues with file deletion
- Pick parent POM 1.37 with Java 7 support
- FindBugs: Fix resource leak when exception happens during the dependency list read
- Downgrade the code to Java 6
- Downgrade back to Java 1.3
- Address comment from @rsandell. Just in case
Code changed in jenkins
User: Oleg Nenashev
Path:
war/pom.xml
http://jenkins-ci.org/commit/jenkins/24443acb3c92612dc623849c9458000c98a0a265
Log:
[JENKINS-34287,JENKINS-23232] Update executable-war to 1.34
Diff: https://github.com/jenkinsci/extras-executable-war/compare/executable-war-1.33...executable-war-1.34
Code changed in jenkins
User: Oleg Nenashev
Path:
war/pom.xml
http://jenkins-ci.org/commit/jenkins/d7598f118fd6d9989164cd2ff7126001a6ee2ff2
Log:
[JENKINS-34287,JENKINS-23232] Update executable-war to 1.34 (#2565)
Diff: https://github.com/jenkinsci/extras-executable-war/compare/executable-war-1.33...executable-war-1.34
My guess is that this is because hudson.diyChunking is still set to true, and between Jetty and Jenkins we end up double-decoding chunked encoding. Needs to verify.