• Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • cli
    • None

      When trying to contact a Jenkins (version 1.447.2) behind an Apache reverse proxy via Jenkins CLI, there is no response until reverse proxy timeout is reached and then the connection gets closed.

      Debugging into it, the CLI hangs on the line where it is reading the input stream. After the connection was closed, the server's answer "Starting HTTP duplex channel" can be read from that stream.

      The CLI terminates with an EOFException:

      Exception in thread "main" java.io.EOFException: unexpected stream termination
              at hudson.remoting.Channel.<init>(Channel.java:408)
              at hudson.remoting.Channel.<init>(Channel.java:366)
              at hudson.remoting.Channel.<init>(Channel.java:327)
              at hudson.remoting.Channel.<init>(Channel.java:323)
              at hudson.remoting.Channel.<init>(Channel.java:311)
              at hudson.cli.CLI.connectViaHttp(CLI.java:124)
              at hudson.cli.CLI.<init>(CLI.java:107)
              at hudson.cli.CLI.<init>(CLI.java:85)
              at hudson.cli.CLI._main(CLI.java:300)
              at hudson.cli.CLI.main(CLI.java:245)
      

      The server does not log any errors and judging from the input stream's content is was contacted properly.

      The reverse proxy error log when reaching timeout:

      (70007)The timeout specified has expired: Error reading chunk
      (104)Connection reset by peer: Error reading chunk
      (-3)Unknown error 18446744073709551613: proxy: prefetch request body failed
      

          [JENKINS-16345] CLI does not work through reverse proxy

          Daniel Beck added a comment -

          susannelindgren: I still see no bug. If Jenkins is running on a different host than the reverse proxy that is the canonical URL for Jenkins, you need to set the system property hudson.TcpSlaveAgentListener.hostName as described on https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties. Alternatively, forward the TCP slave agent port from the proxy to the correct host in e.g. iptables.

          Daniel Beck added a comment - susannelindgren : I still see no bug. If Jenkins is running on a different host than the reverse proxy that is the canonical URL for Jenkins, you need to set the system property hudson.TcpSlaveAgentListener.hostName as described on https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties . Alternatively, forward the TCP slave agent port from the proxy to the correct host in e.g. iptables.

          How is hudson.TcpSlaveAgentListener.hostName used? I tried adding -Dhudson.TcpSlaveAgentListener.hostName=$HOSTNAME to the parameters when starting my jenkins, but cli access failed in the same way as before.

          Susanne Lindgren added a comment - How is hudson.TcpSlaveAgentListener.hostName used? I tried adding -Dhudson.TcpSlaveAgentListener.hostName=$HOSTNAME to the parameters when starting my jenkins, but cli access failed in the same way as before.

          Daniel Beck added a comment -

          Is the host name from the environment variable known to the CLI client (including how it's (not) fully qualified)?

          Daniel Beck added a comment - Is the host name from the environment variable known to the CLI client (including how it's (not) fully qualified)?

          Yes, both the short hostname and the fully qualified hostname are found with nslookup on the host that I execute the CLI client from. I tested with both, with the same result as when I don't use the hudson.TcpSlaveAgentListener.hostName parameter at all.

          Susanne Lindgren added a comment - Yes, both the short hostname and the fully qualified hostname are found with nslookup on the host that I execute the CLI client from. I tested with both, with the same result as when I don't use the hudson.TcpSlaveAgentListener.hostName parameter at all.

          Adding -Dhudson.TcpSlaveAgentListener.hostName="my-jenkins-hostname" in Jenkins startup parameters works for me. Thanks danielbeck

          Michael Pailloncy added a comment - Adding -Dhudson.TcpSlaveAgentListener.hostName="my-jenkins-hostname" in Jenkins startup parameters works for me. Thanks danielbeck

          I've a similar issue. Running Jenkins and Apache proxy in a docker environment both within the same docker network. Https access to jenkins is configured via* reverse proxy* in Apache (ldap and local file). Base url is https://mydockerhost/cm-jenkins. Jenkins is configured to authenticate via reverse proxy (HTTP Header by reverse proxy). The apache container forwards to Jenkins on the container name (=cm-jenkins). "cm-jenkins" is the host name under which other containers (and thus the apache docker container as well) in the same network can reach my jenkins master container.
          Part of my apache vhosts configuration for the jenkins master:

          <Location "/cm-jenkins/">
                  ProxyPass http://cm-jenkins:8080/cm-jenkins/ nocanon
                  ProxyPassReverse http://cm-jenkins:8080/cm-jenkins/
          </Location>
          
          

          This works great.
          I've exposed port 50000 on the Jenkins master (cm-jenkins), configured the TCP port for JNLP slave agents to "
          Fixed : 50000" and set both org.jenkinsci.main.modules.sshd.SSHD.hostName=cm-jenkins and hudson.TcpSlaveAgentListener.hostName=cm-jenkins in the java options at startup. Not sure if the first is required, but found that in another post. Also tried with quotes around the hostname (-Dhudson.TcpSlaveAgentListener.hostName="cm-jenkins").

          I'm trying to run the jenkins-cli.jar from another container in the same docker network. Ssh keys are not the problem. The connection is established (I can see this via netstat -an on the jenkins master). But nothing happens. No timeout (at least, I did not wait longer than 5 minutes).

          My command:

          java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://cm-jenkins:50000/cm-jenkins/  -i /var/jenkins_home/.ssh/id_rsa version
          Enter passphrase for /var/jenkins_home/.ssh/id_rsa:
          

          Nothing happens for more than 5 minutes.
          When I open the https://mydockerhost/cm-jenkins/cli, I still do see a message in the black bar pointing to the base Jenkins url:
          "java -jar jenkins-cli.jar -s https://mydockerhost/cm-jenkins/ help". So, maybe the hudson.TcpSlaveAgentListener.hostName="cm-jenkins" is incorrect.
          Trying to connect on https://mydockerhost/cm-jenkins/ obviously will not work due to the reverse proxy and authentication. I think the CLI should connect not through the reverse proxy but directly use ssh and ssh keys.

          Any help would be appreciated or tips how to debug this. I know this might not be the correct place to ask, but as this is still an open issue, I reckoned it is ok.

          Mariska Tallandtree added a comment - I've a similar issue. Running Jenkins and Apache proxy in a docker environment both within the same docker network. Https access to jenkins is configured via* reverse proxy* in Apache (ldap and local file). Base url is https://mydockerhost/cm-jenkins . Jenkins is configured to authenticate via reverse proxy (HTTP Header by reverse proxy). The apache container forwards to Jenkins on the container name (= cm-jenkins ). "cm-jenkins" is the host name under which other containers (and thus the apache docker container as well) in the same network can reach my jenkins master container. Part of my apache vhosts configuration for the jenkins master: <Location "/cm-jenkins/"> ProxyPass http://cm-jenkins:8080/cm-jenkins/ nocanon ProxyPassReverse http://cm-jenkins:8080/cm-jenkins/ </Location> This works great. I've exposed port 50000 on the Jenkins master (cm-jenkins), configured the TCP port for JNLP slave agents to " Fixed : 50000" and set both org.jenkinsci.main.modules.sshd.SSHD.hostName=cm-jenkins and hudson.TcpSlaveAgentListener.hostName=cm-jenkins in the java options at startup. Not sure if the first is required, but found that in another post. Also tried with quotes around the hostname (-Dhudson.TcpSlaveAgentListener.hostName="cm-jenkins"). I'm trying to run the jenkins-cli.jar from another container in the same docker network. Ssh keys are not the problem. The connection is established (I can see this via netstat -an on the jenkins master). But nothing happens. No timeout (at least, I did not wait longer than 5 minutes). My command: java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://cm-jenkins:50000/cm-jenkins/ -i /var/jenkins_home/.ssh/id_rsa version Enter passphrase for /var/jenkins_home/.ssh/id_rsa: Nothing happens for more than 5 minutes. When I open the https://mydockerhost/cm-jenkins/cli , I still do see a message in the black bar pointing to the base Jenkins url: "java -jar jenkins-cli.jar -s https://mydockerhost/cm-jenkins/ help". So, maybe the hudson.TcpSlaveAgentListener.hostName="cm-jenkins" is incorrect. Trying to connect on https://mydockerhost/cm-jenkins/ obviously will not work due to the reverse proxy and authentication. I think the CLI should connect not through the reverse proxy but directly use ssh and ssh keys. Any help would be appreciated or tips how to debug this. I know this might not be the correct place to ask, but as this is still an open issue, I reckoned it is ok.

          Daniel Beck added a comment -

          tallandtree https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump Do this for both the CLI client process, and the Jenkins master.

          Daniel Beck added a comment - tallandtree https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump Do this for both the CLI client process, and the Jenkins master.

          @Daniel, please see attached my dumpfiles. The client connects from ip: 172.19.0.6. Jenkins master ip is: 172.19.0.5. Hope you can figure out what is wrong. If you need more info, please let me know.

          Mariska Tallandtree added a comment - @Daniel, please see attached my dumpfiles. The client connects from ip: 172.19.0.6. Jenkins master ip is: 172.19.0.5. Hope you can figure out what is wrong. If you need more info, please let me know.

          danielbeck, It seems I've got it working after all. What I did wrong (obviously , I found out after I tried the same without reverse proxy and using basic Jenkins authentication:
          I assumed, that copying the public key to $JENKINS_HOME/.ssh/authorized_keys, would be the way tell jenkins which keys are allowed. But I needed to copy the public key to https://mydockerhost/cm-jenkins/me/configure. As jenkins is not a 'real' ssh-daemon, only acts as one. And the way to connect is to port 8080 and not 50000. Although I did try that yesterday as well. The big issue was not loading the public key at the correct way.

          java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://cm-jenkins:8080/cm-jenkins/  -i /var/jenkins_home/.ssh/id_rsa version
          Enter passphrase for /var/jenkins_home/.ssh/id_rsa:
          

          So, thanks for your help even though I was finally able to figured it out myself . Hope this will help someone else. Maybe just close the issue?

          Mariska Tallandtree added a comment - danielbeck , It seems I've got it working after all. What I did wrong (obviously , I found out after I tried the same without reverse proxy and using basic Jenkins authentication: I assumed, that copying the public key to $JENKINS_HOME/.ssh/authorized_keys, would be the way tell jenkins which keys are allowed. But I needed to copy the public key to https://mydockerhost/cm-jenkins/me/configure . As jenkins is not a 'real' ssh-daemon, only acts as one. And the way to connect is to port 8080 and not 50000. Although I did try that yesterday as well. The big issue was not loading the public key at the correct way. java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://cm-jenkins:8080/cm-jenkins/ -i /var/jenkins_home/.ssh/id_rsa version Enter passphrase for /var/jenkins_home/.ssh/id_rsa: So, thanks for your help even though I was finally able to figured it out myself . Hope this will help someone else. Maybe just close the issue?

          Daniel Beck added a comment -

          As jenkins is not a 'real' ssh-daemon, only acts as one. And the way to connect is to port 8080 and not 50000.

          The CLI is exposed both as standalone service for use with jenkins-cli.jar (for which you specify the HTTP URL of Jenkins and then it either goes through HTTP, or connects to the TCP agent listener port), as well as SSH daemon for use with an SSH client.

          Daniel Beck added a comment - As jenkins is not a 'real' ssh-daemon, only acts as one. And the way to connect is to port 8080 and not 50000. The CLI is exposed both as standalone service for use with jenkins-cli.jar (for which you specify the HTTP URL of Jenkins and then it either goes through HTTP, or connects to the TCP agent listener port), as well as SSH daemon for use with an SSH client.

            Unassigned Unassigned
            laccu Lars Klonowski
            Votes:
            10 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: