-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Powered by SuggestiMate
When trying to contact a Jenkins (version 1.447.2) behind an Apache reverse proxy via Jenkins CLI, there is no response until reverse proxy timeout is reached and then the connection gets closed.
Debugging into it, the CLI hangs on the line where it is reading the input stream. After the connection was closed, the server's answer "Starting HTTP duplex channel" can be read from that stream.
The CLI terminates with an EOFException:
Exception in thread "main" java.io.EOFException: unexpected stream termination at hudson.remoting.Channel.<init>(Channel.java:408) at hudson.remoting.Channel.<init>(Channel.java:366) at hudson.remoting.Channel.<init>(Channel.java:327) at hudson.remoting.Channel.<init>(Channel.java:323) at hudson.remoting.Channel.<init>(Channel.java:311) at hudson.cli.CLI.connectViaHttp(CLI.java:124) at hudson.cli.CLI.<init>(CLI.java:107) at hudson.cli.CLI.<init>(CLI.java:85) at hudson.cli.CLI._main(CLI.java:300) at hudson.cli.CLI.main(CLI.java:245)
The server does not log any errors and judging from the input stream's content is was contacted properly.
The reverse proxy error log when reaching timeout:
(70007)The timeout specified has expired: Error reading chunk (104)Connection reset by peer: Error reading chunk (-3)Unknown error 18446744073709551613: proxy: prefetch request body failed
- is related to
-
JENKINS-11560 CLI hangs through reverse proxy with JNLP disabled
-
- Open
-
-
JENKINS-17142 java cli does not work behind proxy
-
- Resolved
-
[JENKINS-16345] CLI does not work through reverse proxy
Hi,
same here (but not the exact same trace) :
Exception in thread "main" java.io.EOFException: unexpected stream termination at hudson.remoting.ClassicCommandTransport.create(ClassicCommandTransport.java:100) at hudson.remoting.Channel.<init>(Channel.java:392) at hudson.remoting.Channel.<init>(Channel.java:388) at hudson.remoting.Channel.<init>(Channel.java:349) at hudson.remoting.Channel.<init>(Channel.java:345) at hudson.remoting.Channel.<init>(Channel.java:333) at hudson.cli.CLI.connectViaHttp(CLI.java:157) at hudson.cli.CLI.<init>(CLI.java:140) at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:68) at hudson.cli.CLI._main(CLI.java:439) at hudson.cli.CLI.main(CLI.java:374)
I had just launch a - "java -jar jenkins-cli.jar -s http://my.jenkins.server version"
Any idea?
The error on the server side :
[Tue Mar 12 16:02:12 2013] [error] [client 85.170.86.148] (70007)The timeout specified has expired: Error reading chunk [Tue Mar 12 16:02:12 2013] [error] (-3)Unknown error 18446744073709551613: proxy: prefetch request body failed to 127.0.0.1:8080 (localhost) from 85.170.86.148 ()
No response to comment asking for additional information in months, so resolving as Incomplete.
I'm having the same issue with jenkins behind an nginx reverse proxy.
Martin: Be more specific. Provide the information requested here: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue
Same problem: cli works when accessing the server directly, but not when accessed via an nginx proxy.
Jenkins version 1.593. No configuration done before testing cli, so everything is default. jenkins-cli.jar newly downloaded from this jenkins. Cli tests done on the jenkins server.
Jenkins is running directly (without container), installed and started as:
$ wget http://mirrors.jenkins-ci.org/war/1.593/jenkins.war
$ /usr/bin/java -Xms256m -Xmx1024m -DJENKINS_HOME=/home/esusedl/.jenkins-sjk2 -jar /home/esusedl/.jenkins-sjk2/jenkins.war --httpPort=8080 --httpsPort=8081 --ajp13Port=-1 --prefix=/sjk2
When using cli towards the server directly it works fine:
$ /usr/bin/java -jar jenkins-cli.jar -s https://jenkins_server_fqdn:8081/jenkins -noCertificateCheck version
Skipping HTTPS certificate checks altogether. Note that this is not secure at all.
Enter passphrase for /home/myuser/.ssh/id_rsa:
[WARN] Failed to authenticate with your SSH keys. Proceeding as anonymous
1.593
$ /usr/bin/java -jar jenkins-cli.jar -s https://jenkins_proxy_fqdn/jenkins -noCertificateCheck version
Skipping HTTPS certificate checks altogether. Note that this is not secure at all.
Enter passphrase for /home/myuser/.ssh/id_rsa:
Exception in thread "main" java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at hudson.cli.CLI.connectViaCliPort(CLI.java:203)
at hudson.cli.CLI.<init>(CLI.java:126)
at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:72)
at hudson.cli.CLI._main(CLI.java:466)
at hudson.cli.CLI.main(CLI.java:382)
Suppressed: java.io.EOFException: unexpected stream termination
at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:331)
at hudson.remoting.Channel.<init>(Channel.java:421)
at hudson.remoting.Channel.<init>(Channel.java:400)
at hudson.remoting.Channel.<init>(Channel.java:396)
at hudson.remoting.Channel.<init>(Channel.java:385)
at hudson.remoting.Channel.<init>(Channel.java:377)
at hudson.remoting.Channel.<init>(Channel.java:353)
at hudson.cli.CLI.connectViaHttp(CLI.java:157)
at hudson.cli.CLI.<init>(CLI.java:130)
... 3 more
$ /usr/bin/java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
Jenkins server: RHEL 6.5 x86_64 with a self-signed certificate.
Jenkins proxy server: RHEL 6.5 x86_64 with a verisign certificate. Nginx config for the jenkins instance:
location /jenkins {
sendfile off;
proxy_pass https://jenkins_server_fqdn:8081/jenkins;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
susannelindgren: I still see no bug. If Jenkins is running on a different host than the reverse proxy that is the canonical URL for Jenkins, you need to set the system property hudson.TcpSlaveAgentListener.hostName as described on https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties. Alternatively, forward the TCP slave agent port from the proxy to the correct host in e.g. iptables.
How is hudson.TcpSlaveAgentListener.hostName used? I tried adding -Dhudson.TcpSlaveAgentListener.hostName=$HOSTNAME to the parameters when starting my jenkins, but cli access failed in the same way as before.
Is the host name from the environment variable known to the CLI client (including how it's (not) fully qualified)?
Yes, both the short hostname and the fully qualified hostname are found with nslookup on the host that I execute the CLI client from. I tested with both, with the same result as when I don't use the hudson.TcpSlaveAgentListener.hostName parameter at all.
Adding -Dhudson.TcpSlaveAgentListener.hostName="my-jenkins-hostname" in Jenkins startup parameters works for me. Thanks danielbeck
I've a similar issue. Running Jenkins and Apache proxy in a docker environment both within the same docker network. Https access to jenkins is configured via* reverse proxy* in Apache (ldap and local file). Base url is https://mydockerhost/cm-jenkins. Jenkins is configured to authenticate via reverse proxy (HTTP Header by reverse proxy). The apache container forwards to Jenkins on the container name (=cm-jenkins). "cm-jenkins" is the host name under which other containers (and thus the apache docker container as well) in the same network can reach my jenkins master container.
Part of my apache vhosts configuration for the jenkins master:
<Location "/cm-jenkins/"> ProxyPass http://cm-jenkins:8080/cm-jenkins/ nocanon ProxyPassReverse http://cm-jenkins:8080/cm-jenkins/ </Location>
This works great.
I've exposed port 50000 on the Jenkins master (cm-jenkins), configured the TCP port for JNLP slave agents to "
Fixed : 50000" and set both org.jenkinsci.main.modules.sshd.SSHD.hostName=cm-jenkins and hudson.TcpSlaveAgentListener.hostName=cm-jenkins in the java options at startup. Not sure if the first is required, but found that in another post. Also tried with quotes around the hostname (-Dhudson.TcpSlaveAgentListener.hostName="cm-jenkins").
I'm trying to run the jenkins-cli.jar from another container in the same docker network. Ssh keys are not the problem. The connection is established (I can see this via netstat -an on the jenkins master). But nothing happens. No timeout (at least, I did not wait longer than 5 minutes).
My command:
java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://cm-jenkins:50000/cm-jenkins/ -i /var/jenkins_home/.ssh/id_rsa version Enter passphrase for /var/jenkins_home/.ssh/id_rsa:
Nothing happens for more than 5 minutes.
When I open the https://mydockerhost/cm-jenkins/cli, I still do see a message in the black bar pointing to the base Jenkins url:
"java -jar jenkins-cli.jar -s https://mydockerhost/cm-jenkins/ help". So, maybe the hudson.TcpSlaveAgentListener.hostName="cm-jenkins" is incorrect.
Trying to connect on https://mydockerhost/cm-jenkins/ obviously will not work due to the reverse proxy and authentication. I think the CLI should connect not through the reverse proxy but directly use ssh and ssh keys.
Any help would be appreciated or tips how to debug this. I know this might not be the correct place to ask, but as this is still an open issue, I reckoned it is ok.
tallandtree https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump Do this for both the CLI client process, and the Jenkins master.
@Daniel, please see attached my dumpfiles. The client connects from ip: 172.19.0.6. Jenkins master ip is: 172.19.0.5. Hope you can figure out what is wrong. If you need more info, please let me know.
danielbeck, It seems I've got it working after all. What I did wrong (obviously , I found out after I tried the same without reverse proxy and using basic Jenkins authentication:
I assumed, that copying the public key to $JENKINS_HOME/.ssh/authorized_keys, would be the way tell jenkins which keys are allowed. But I needed to copy the public key to https://mydockerhost/cm-jenkins/me/configure. As jenkins is not a 'real' ssh-daemon, only acts as one. And the way to connect is to port 8080 and not 50000. Although I did try that yesterday as well. The big issue was not loading the public key at the correct way.
java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://cm-jenkins:8080/cm-jenkins/ -i /var/jenkins_home/.ssh/id_rsa version Enter passphrase for /var/jenkins_home/.ssh/id_rsa:
So, thanks for your help even though I was finally able to figured it out myself . Hope this will help someone else. Maybe just close the issue?
As jenkins is not a 'real' ssh-daemon, only acts as one. And the way to connect is to port 8080 and not 50000.
The CLI is exposed both as standalone service for use with jenkins-cli.jar (for which you specify the HTTP URL of Jenkins and then it either goes through HTTP, or connects to the TCP agent listener port), as well as SSH daemon for use with an SSH client.
I'm experiencing the same problem. Any information that could be provided to help diagnose?