• Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • cli
    • None

      When trying to contact a Jenkins (version 1.447.2) behind an Apache reverse proxy via Jenkins CLI, there is no response until reverse proxy timeout is reached and then the connection gets closed.

      Debugging into it, the CLI hangs on the line where it is reading the input stream. After the connection was closed, the server's answer "Starting HTTP duplex channel" can be read from that stream.

      The CLI terminates with an EOFException:

      Exception in thread "main" java.io.EOFException: unexpected stream termination
              at hudson.remoting.Channel.<init>(Channel.java:408)
              at hudson.remoting.Channel.<init>(Channel.java:366)
              at hudson.remoting.Channel.<init>(Channel.java:327)
              at hudson.remoting.Channel.<init>(Channel.java:323)
              at hudson.remoting.Channel.<init>(Channel.java:311)
              at hudson.cli.CLI.connectViaHttp(CLI.java:124)
              at hudson.cli.CLI.<init>(CLI.java:107)
              at hudson.cli.CLI.<init>(CLI.java:85)
              at hudson.cli.CLI._main(CLI.java:300)
              at hudson.cli.CLI.main(CLI.java:245)
      

      The server does not log any errors and judging from the input stream's content is was contacted properly.

      The reverse proxy error log when reaching timeout:

      (70007)The timeout specified has expired: Error reading chunk
      (104)Connection reset by peer: Error reading chunk
      (-3)Unknown error 18446744073709551613: proxy: prefetch request body failed
      

          [JENKINS-16345] CLI does not work through reverse proxy

          I'm experiencing the same problem. Any information that could be provided to help diagnose?

          Sidnei da Silva added a comment - I'm experiencing the same problem. Any information that could be provided to help diagnose?

          Eric Cambray added a comment - - edited

          Hi,
          same here (but not the exact same trace) :

          Exception in thread "main" java.io.EOFException: unexpected stream termination
          	at hudson.remoting.ClassicCommandTransport.create(ClassicCommandTransport.java:100)
          	at hudson.remoting.Channel.<init>(Channel.java:392)
          	at hudson.remoting.Channel.<init>(Channel.java:388)
          	at hudson.remoting.Channel.<init>(Channel.java:349)
          	at hudson.remoting.Channel.<init>(Channel.java:345)
          	at hudson.remoting.Channel.<init>(Channel.java:333)
          	at hudson.cli.CLI.connectViaHttp(CLI.java:157)
          	at hudson.cli.CLI.<init>(CLI.java:140)
          	at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:68)
          	at hudson.cli.CLI._main(CLI.java:439)
          	at hudson.cli.CLI.main(CLI.java:374)
          

          I had just launch a - "java -jar jenkins-cli.jar -s http://my.jenkins.server version"

          Any idea?

          The error on the server side :

          [Tue Mar 12 16:02:12 2013] [error] [client 85.170.86.148] (70007)The timeout specified has expired: Error reading chunk  
          [Tue Mar 12 16:02:12 2013] [error] (-3)Unknown error 18446744073709551613: proxy: prefetch request body failed to 127.0.0.1:8080 (localhost) from 85.170.86.148 ()
          

          Eric Cambray added a comment - - edited Hi, same here (but not the exact same trace) : Exception in thread "main" java.io.EOFException: unexpected stream termination at hudson.remoting.ClassicCommandTransport.create(ClassicCommandTransport.java:100) at hudson.remoting.Channel.<init>(Channel.java:392) at hudson.remoting.Channel.<init>(Channel.java:388) at hudson.remoting.Channel.<init>(Channel.java:349) at hudson.remoting.Channel.<init>(Channel.java:345) at hudson.remoting.Channel.<init>(Channel.java:333) at hudson.cli.CLI.connectViaHttp(CLI.java:157) at hudson.cli.CLI.<init>(CLI.java:140) at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:68) at hudson.cli.CLI._main(CLI.java:439) at hudson.cli.CLI.main(CLI.java:374) I had just launch a - "java -jar jenkins-cli.jar -s http://my.jenkins.server version" Any idea? The error on the server side : [Tue Mar 12 16:02:12 2013] [error] [client 85.170.86.148] (70007)The timeout specified has expired: Error reading chunk [Tue Mar 12 16:02:12 2013] [error] (-3)Unknown error 18446744073709551613: proxy: prefetch request body failed to 127.0.0.1:8080 (localhost) from 85.170.86.148 ()

          Same issue using Jenkins 1.531 and Jenkins behind Nginx.

          Michael Pailloncy added a comment - Same issue using Jenkins 1.531 and Jenkins behind Nginx.

          For the record, still confirmed with 1.559.

          Baptiste Mathus added a comment - For the record, still confirmed with 1.559.

          Daniel Beck added a comment -

          Is JNLP port set to disabled? When you enable it, does it work then?

          Daniel Beck added a comment - Is JNLP port set to disabled? When you enable it, does it work then?

          Daniel Beck added a comment -

          Ping!

          Daniel Beck added a comment - Ping!

          Daniel Beck added a comment -

          No response to comment asking for additional information in months, so resolving as Incomplete.

          Daniel Beck added a comment - No response to comment asking for additional information in months, so resolving as Incomplete.

          I'm having the same issue with jenkins behind an nginx reverse proxy.

          Martín Fernández added a comment - I'm having the same issue with jenkins behind an nginx reverse proxy.

          Daniel Beck added a comment -

          Martin: Be more specific. Provide the information requested here: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue

          Daniel Beck added a comment - Martin: Be more specific. Provide the information requested here: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue

          Same problem: cli works when accessing the server directly, but not when accessed via an nginx proxy.

          Jenkins version 1.593. No configuration done before testing cli, so everything is default. jenkins-cli.jar newly downloaded from this jenkins. Cli tests done on the jenkins server.

          Jenkins is running directly (without container), installed and started as:
          $ wget http://mirrors.jenkins-ci.org/war/1.593/jenkins.war
          $ /usr/bin/java -Xms256m -Xmx1024m -DJENKINS_HOME=/home/esusedl/.jenkins-sjk2 -jar /home/esusedl/.jenkins-sjk2/jenkins.war --httpPort=8080 --httpsPort=8081 --ajp13Port=-1 --prefix=/sjk2

          When using cli towards the server directly it works fine:

          $ /usr/bin/java -jar jenkins-cli.jar -s https://jenkins_server_fqdn:8081/jenkins -noCertificateCheck version
          Skipping HTTPS certificate checks altogether. Note that this is not secure at all.
          Enter passphrase for /home/myuser/.ssh/id_rsa:
          [WARN] Failed to authenticate with your SSH keys. Proceeding as anonymous
          1.593

          $ /usr/bin/java -jar jenkins-cli.jar -s https://jenkins_proxy_fqdn/jenkins -noCertificateCheck version
          Skipping HTTPS certificate checks altogether. Note that this is not secure at all.
          Enter passphrase for /home/myuser/.ssh/id_rsa:
          Exception in thread "main" java.net.ConnectException: Connection refused
          at java.net.PlainSocketImpl.socketConnect(Native Method)
          at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
          at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
          at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
          at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
          at java.net.Socket.connect(Socket.java:579)
          at hudson.cli.CLI.connectViaCliPort(CLI.java:203)
          at hudson.cli.CLI.<init>(CLI.java:126)
          at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:72)
          at hudson.cli.CLI._main(CLI.java:466)
          at hudson.cli.CLI.main(CLI.java:382)
          Suppressed: java.io.EOFException: unexpected stream termination
          at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:331)
          at hudson.remoting.Channel.<init>(Channel.java:421)
          at hudson.remoting.Channel.<init>(Channel.java:400)
          at hudson.remoting.Channel.<init>(Channel.java:396)
          at hudson.remoting.Channel.<init>(Channel.java:385)
          at hudson.remoting.Channel.<init>(Channel.java:377)
          at hudson.remoting.Channel.<init>(Channel.java:353)
          at hudson.cli.CLI.connectViaHttp(CLI.java:157)
          at hudson.cli.CLI.<init>(CLI.java:130)
          ... 3 more

          $ /usr/bin/java -version
          java version "1.7.0_45"
          Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
          Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

          Jenkins server: RHEL 6.5 x86_64 with a self-signed certificate.

          Jenkins proxy server: RHEL 6.5 x86_64 with a verisign certificate. Nginx config for the jenkins instance:

          location /jenkins {
          sendfile off;
          proxy_pass https://jenkins_server_fqdn:8081/jenkins;

          proxy_set_header Host $http_host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          }

          Susanne Lindgren added a comment - Same problem: cli works when accessing the server directly, but not when accessed via an nginx proxy. Jenkins version 1.593. No configuration done before testing cli, so everything is default. jenkins-cli.jar newly downloaded from this jenkins. Cli tests done on the jenkins server. Jenkins is running directly (without container), installed and started as: $ wget http://mirrors.jenkins-ci.org/war/1.593/jenkins.war $ /usr/bin/java -Xms256m -Xmx1024m -DJENKINS_HOME=/home/esusedl/.jenkins-sjk2 -jar /home/esusedl/.jenkins-sjk2/jenkins.war --httpPort=8080 --httpsPort=8081 --ajp13Port=-1 --prefix=/sjk2 When using cli towards the server directly it works fine: $ /usr/bin/java -jar jenkins-cli.jar -s https://jenkins_server_fqdn:8081/jenkins -noCertificateCheck version Skipping HTTPS certificate checks altogether. Note that this is not secure at all. Enter passphrase for /home/myuser/.ssh/id_rsa: [WARN] Failed to authenticate with your SSH keys. Proceeding as anonymous 1.593 $ /usr/bin/java -jar jenkins-cli.jar -s https://jenkins_proxy_fqdn/jenkins -noCertificateCheck version Skipping HTTPS certificate checks altogether. Note that this is not secure at all. Enter passphrase for /home/myuser/.ssh/id_rsa: Exception in thread "main" java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at hudson.cli.CLI.connectViaCliPort(CLI.java:203) at hudson.cli.CLI.<init>(CLI.java:126) at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:72) at hudson.cli.CLI._main(CLI.java:466) at hudson.cli.CLI.main(CLI.java:382) Suppressed: java.io.EOFException: unexpected stream termination at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:331) at hudson.remoting.Channel.<init>(Channel.java:421) at hudson.remoting.Channel.<init>(Channel.java:400) at hudson.remoting.Channel.<init>(Channel.java:396) at hudson.remoting.Channel.<init>(Channel.java:385) at hudson.remoting.Channel.<init>(Channel.java:377) at hudson.remoting.Channel.<init>(Channel.java:353) at hudson.cli.CLI.connectViaHttp(CLI.java:157) at hudson.cli.CLI.<init>(CLI.java:130) ... 3 more $ /usr/bin/java -version java version "1.7.0_45" Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) Jenkins server: RHEL 6.5 x86_64 with a self-signed certificate. Jenkins proxy server: RHEL 6.5 x86_64 with a verisign certificate. Nginx config for the jenkins instance: location /jenkins { sendfile off; proxy_pass https://jenkins_server_fqdn:8081/jenkins ; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }

          Daniel Beck added a comment -

          susannelindgren: I still see no bug. If Jenkins is running on a different host than the reverse proxy that is the canonical URL for Jenkins, you need to set the system property hudson.TcpSlaveAgentListener.hostName as described on https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties. Alternatively, forward the TCP slave agent port from the proxy to the correct host in e.g. iptables.

          Daniel Beck added a comment - susannelindgren : I still see no bug. If Jenkins is running on a different host than the reverse proxy that is the canonical URL for Jenkins, you need to set the system property hudson.TcpSlaveAgentListener.hostName as described on https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties . Alternatively, forward the TCP slave agent port from the proxy to the correct host in e.g. iptables.

          How is hudson.TcpSlaveAgentListener.hostName used? I tried adding -Dhudson.TcpSlaveAgentListener.hostName=$HOSTNAME to the parameters when starting my jenkins, but cli access failed in the same way as before.

          Susanne Lindgren added a comment - How is hudson.TcpSlaveAgentListener.hostName used? I tried adding -Dhudson.TcpSlaveAgentListener.hostName=$HOSTNAME to the parameters when starting my jenkins, but cli access failed in the same way as before.

          Daniel Beck added a comment -

          Is the host name from the environment variable known to the CLI client (including how it's (not) fully qualified)?

          Daniel Beck added a comment - Is the host name from the environment variable known to the CLI client (including how it's (not) fully qualified)?

          Yes, both the short hostname and the fully qualified hostname are found with nslookup on the host that I execute the CLI client from. I tested with both, with the same result as when I don't use the hudson.TcpSlaveAgentListener.hostName parameter at all.

          Susanne Lindgren added a comment - Yes, both the short hostname and the fully qualified hostname are found with nslookup on the host that I execute the CLI client from. I tested with both, with the same result as when I don't use the hudson.TcpSlaveAgentListener.hostName parameter at all.

          Adding -Dhudson.TcpSlaveAgentListener.hostName="my-jenkins-hostname" in Jenkins startup parameters works for me. Thanks danielbeck

          Michael Pailloncy added a comment - Adding -Dhudson.TcpSlaveAgentListener.hostName="my-jenkins-hostname" in Jenkins startup parameters works for me. Thanks danielbeck

          I've a similar issue. Running Jenkins and Apache proxy in a docker environment both within the same docker network. Https access to jenkins is configured via* reverse proxy* in Apache (ldap and local file). Base url is https://mydockerhost/cm-jenkins. Jenkins is configured to authenticate via reverse proxy (HTTP Header by reverse proxy). The apache container forwards to Jenkins on the container name (=cm-jenkins). "cm-jenkins" is the host name under which other containers (and thus the apache docker container as well) in the same network can reach my jenkins master container.
          Part of my apache vhosts configuration for the jenkins master:

          <Location "/cm-jenkins/">
                  ProxyPass http://cm-jenkins:8080/cm-jenkins/ nocanon
                  ProxyPassReverse http://cm-jenkins:8080/cm-jenkins/
          </Location>
          
          

          This works great.
          I've exposed port 50000 on the Jenkins master (cm-jenkins), configured the TCP port for JNLP slave agents to "
          Fixed : 50000" and set both org.jenkinsci.main.modules.sshd.SSHD.hostName=cm-jenkins and hudson.TcpSlaveAgentListener.hostName=cm-jenkins in the java options at startup. Not sure if the first is required, but found that in another post. Also tried with quotes around the hostname (-Dhudson.TcpSlaveAgentListener.hostName="cm-jenkins").

          I'm trying to run the jenkins-cli.jar from another container in the same docker network. Ssh keys are not the problem. The connection is established (I can see this via netstat -an on the jenkins master). But nothing happens. No timeout (at least, I did not wait longer than 5 minutes).

          My command:

          java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://cm-jenkins:50000/cm-jenkins/  -i /var/jenkins_home/.ssh/id_rsa version
          Enter passphrase for /var/jenkins_home/.ssh/id_rsa:
          

          Nothing happens for more than 5 minutes.
          When I open the https://mydockerhost/cm-jenkins/cli, I still do see a message in the black bar pointing to the base Jenkins url:
          "java -jar jenkins-cli.jar -s https://mydockerhost/cm-jenkins/ help". So, maybe the hudson.TcpSlaveAgentListener.hostName="cm-jenkins" is incorrect.
          Trying to connect on https://mydockerhost/cm-jenkins/ obviously will not work due to the reverse proxy and authentication. I think the CLI should connect not through the reverse proxy but directly use ssh and ssh keys.

          Any help would be appreciated or tips how to debug this. I know this might not be the correct place to ask, but as this is still an open issue, I reckoned it is ok.

          Mariska Tallandtree added a comment - I've a similar issue. Running Jenkins and Apache proxy in a docker environment both within the same docker network. Https access to jenkins is configured via* reverse proxy* in Apache (ldap and local file). Base url is https://mydockerhost/cm-jenkins . Jenkins is configured to authenticate via reverse proxy (HTTP Header by reverse proxy). The apache container forwards to Jenkins on the container name (= cm-jenkins ). "cm-jenkins" is the host name under which other containers (and thus the apache docker container as well) in the same network can reach my jenkins master container. Part of my apache vhosts configuration for the jenkins master: <Location "/cm-jenkins/"> ProxyPass http://cm-jenkins:8080/cm-jenkins/ nocanon ProxyPassReverse http://cm-jenkins:8080/cm-jenkins/ </Location> This works great. I've exposed port 50000 on the Jenkins master (cm-jenkins), configured the TCP port for JNLP slave agents to " Fixed : 50000" and set both org.jenkinsci.main.modules.sshd.SSHD.hostName=cm-jenkins and hudson.TcpSlaveAgentListener.hostName=cm-jenkins in the java options at startup. Not sure if the first is required, but found that in another post. Also tried with quotes around the hostname (-Dhudson.TcpSlaveAgentListener.hostName="cm-jenkins"). I'm trying to run the jenkins-cli.jar from another container in the same docker network. Ssh keys are not the problem. The connection is established (I can see this via netstat -an on the jenkins master). But nothing happens. No timeout (at least, I did not wait longer than 5 minutes). My command: java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://cm-jenkins:50000/cm-jenkins/ -i /var/jenkins_home/.ssh/id_rsa version Enter passphrase for /var/jenkins_home/.ssh/id_rsa: Nothing happens for more than 5 minutes. When I open the https://mydockerhost/cm-jenkins/cli , I still do see a message in the black bar pointing to the base Jenkins url: "java -jar jenkins-cli.jar -s https://mydockerhost/cm-jenkins/ help". So, maybe the hudson.TcpSlaveAgentListener.hostName="cm-jenkins" is incorrect. Trying to connect on https://mydockerhost/cm-jenkins/ obviously will not work due to the reverse proxy and authentication. I think the CLI should connect not through the reverse proxy but directly use ssh and ssh keys. Any help would be appreciated or tips how to debug this. I know this might not be the correct place to ask, but as this is still an open issue, I reckoned it is ok.

          Daniel Beck added a comment -

          tallandtree https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump Do this for both the CLI client process, and the Jenkins master.

          Daniel Beck added a comment - tallandtree https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump Do this for both the CLI client process, and the Jenkins master.

          @Daniel, please see attached my dumpfiles. The client connects from ip: 172.19.0.6. Jenkins master ip is: 172.19.0.5. Hope you can figure out what is wrong. If you need more info, please let me know.

          Mariska Tallandtree added a comment - @Daniel, please see attached my dumpfiles. The client connects from ip: 172.19.0.6. Jenkins master ip is: 172.19.0.5. Hope you can figure out what is wrong. If you need more info, please let me know.

          danielbeck, It seems I've got it working after all. What I did wrong (obviously , I found out after I tried the same without reverse proxy and using basic Jenkins authentication:
          I assumed, that copying the public key to $JENKINS_HOME/.ssh/authorized_keys, would be the way tell jenkins which keys are allowed. But I needed to copy the public key to https://mydockerhost/cm-jenkins/me/configure. As jenkins is not a 'real' ssh-daemon, only acts as one. And the way to connect is to port 8080 and not 50000. Although I did try that yesterday as well. The big issue was not loading the public key at the correct way.

          java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://cm-jenkins:8080/cm-jenkins/  -i /var/jenkins_home/.ssh/id_rsa version
          Enter passphrase for /var/jenkins_home/.ssh/id_rsa:
          

          So, thanks for your help even though I was finally able to figured it out myself . Hope this will help someone else. Maybe just close the issue?

          Mariska Tallandtree added a comment - danielbeck , It seems I've got it working after all. What I did wrong (obviously , I found out after I tried the same without reverse proxy and using basic Jenkins authentication: I assumed, that copying the public key to $JENKINS_HOME/.ssh/authorized_keys, would be the way tell jenkins which keys are allowed. But I needed to copy the public key to https://mydockerhost/cm-jenkins/me/configure . As jenkins is not a 'real' ssh-daemon, only acts as one. And the way to connect is to port 8080 and not 50000. Although I did try that yesterday as well. The big issue was not loading the public key at the correct way. java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://cm-jenkins:8080/cm-jenkins/ -i /var/jenkins_home/.ssh/id_rsa version Enter passphrase for /var/jenkins_home/.ssh/id_rsa: So, thanks for your help even though I was finally able to figured it out myself . Hope this will help someone else. Maybe just close the issue?

          Daniel Beck added a comment -

          As jenkins is not a 'real' ssh-daemon, only acts as one. And the way to connect is to port 8080 and not 50000.

          The CLI is exposed both as standalone service for use with jenkins-cli.jar (for which you specify the HTTP URL of Jenkins and then it either goes through HTTP, or connects to the TCP agent listener port), as well as SSH daemon for use with an SSH client.

          Daniel Beck added a comment - As jenkins is not a 'real' ssh-daemon, only acts as one. And the way to connect is to port 8080 and not 50000. The CLI is exposed both as standalone service for use with jenkins-cli.jar (for which you specify the HTTP URL of Jenkins and then it either goes through HTTP, or connects to the TCP agent listener port), as well as SSH daemon for use with an SSH client.

            Unassigned Unassigned
            laccu Lars Klonowski
            Votes:
            10 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: