Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47279

Full-duplex HTTP(S) transport with plain CLI protocol does not work with Apache reverse proxy in Jenkins >= 2.54

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Minor Minor
    • cli, core
    • Jenkins >= 2.54 (Tested on 2.78 and 2.82)
      Apache 2.2.22 and 2.4.22
      Ubuntu 12.04

      I've been testing various reverse proxies with Jenkins, and am unable to use the CLI over an Apache reverse proxy using mod_proxy with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

      I used Wireshark to see what was happening, and the issue appears to be that Apache tries to buffer the body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The relevant setting in Apache has a minimum value of 512, and neither SetEnv proxy-sendchunks 1 nor SetEnv proxy-sendchunked 1 have a noticeable effect.

      I am testing using https://github.com/jglick/jenkins-demo-reverse-proxy, and attempting to run the following CLI command:

      java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
      

      This is the error that is thrown by Jenkins 2.82:

      WARNING: null
      java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
      	at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
      	at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
      

      Assuming that Jenkins' full-duplex HTTP connection is spec compliant, this is arguably a bug in Apache.

          [JENKINS-47279] Full-duplex HTTP(S) transport with plain CLI protocol does not work with Apache reverse proxy in Jenkins >= 2.54

          Devin Nusbaum created issue -
          Devin Nusbaum made changes -
          Description Original: I've been testing various reverse proxies with Jenkins, and can not find a way to use the CLI with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54 to access Jenkins through an Apache reverse proxy. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

          The issue appears to be that Apache tries to buffer the full body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The [relevant setting in Apache|https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxyreceivebuffersize] has a minimum value of 512, and I have been unable to find a way to otherwise get rid of the buffering.

          I am testing using [https://github.com/jglick/jenkins-demo-reverse-proxy], and attempting to run the CLI command {{java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help}}.
           This is the error that is thrown by Jenkins 2.82:
          {noformat}
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
          {noformat}
          New: I've been testing various reverse proxies with Jenkins, and can not find a way to use the CLI with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54 to access Jenkins through an Apache reverse proxy. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

          The issue appears to be that Apache tries to buffer the full body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The [relevant setting in Apache|https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxyreceivebuffersize] has a minimum value of 512, and I have been unable to find a way to otherwise get rid of the buffering.

          I am testing using [https://github.com/jglick/jenkins-demo-reverse-proxy], and attempting to run the following CLI command:
          {noformat}
          java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
          {noformat}
          This is the error that is thrown by Jenkins 2.82:
          {noformat}
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
          {noformat}
          Devin Nusbaum made changes -
          Description Original: I've been testing various reverse proxies with Jenkins, and can not find a way to use the CLI with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54 to access Jenkins through an Apache reverse proxy. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

          The issue appears to be that Apache tries to buffer the full body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The [relevant setting in Apache|https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxyreceivebuffersize] has a minimum value of 512, and I have been unable to find a way to otherwise get rid of the buffering.

          I am testing using [https://github.com/jglick/jenkins-demo-reverse-proxy], and attempting to run the following CLI command:
          {noformat}
          java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
          {noformat}
          This is the error that is thrown by Jenkins 2.82:
          {noformat}
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
          {noformat}
          New: I've been testing various reverse proxies with Jenkins, and can not find a way to use the CLI with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54 to access Jenkins through an Apache reverse proxy. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

          I used Wireshark to see what was happening, and the issue appears to be that Apache tries to buffer the body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The [relevant setting in Apache|https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxyreceivebuffersize] has a minimum value of 512, and I have been unable to find a way to otherwise get rid of the buffering.

          I am testing using [https://github.com/jglick/jenkins-demo-reverse-proxy], and attempting to run the following CLI command:
          {noformat}
          java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
          {noformat}
          This is the error that is thrown by Jenkins 2.82:
          {noformat}
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
          {noformat}
          Devin Nusbaum made changes -
          Environment Original: Jenkins >= 2.54 (I have tested on 2.78 and 2.82, but it should affect all versions after 2.54)
          Apache 2.2.22 (Also tested with 2.4.22)
          Ubuntu 12.04
          New: Jenkins >= 2.54 (Tested on 2.78 and 2.82)
          Apache 2.2.22 and 2.4.22
          Ubuntu 12.04
          Devin Nusbaum made changes -
          Remote Link New: This issue links to "Page (Jenkins Wiki)" [ 17820 ]
          Devin Nusbaum made changes -
          Description Original: I've been testing various reverse proxies with Jenkins, and can not find a way to use the CLI with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54 to access Jenkins through an Apache reverse proxy. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

          I used Wireshark to see what was happening, and the issue appears to be that Apache tries to buffer the body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The [relevant setting in Apache|https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxyreceivebuffersize] has a minimum value of 512, and I have been unable to find a way to otherwise get rid of the buffering.

          I am testing using [https://github.com/jglick/jenkins-demo-reverse-proxy], and attempting to run the following CLI command:
          {noformat}
          java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
          {noformat}
          This is the error that is thrown by Jenkins 2.82:
          {noformat}
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
          {noformat}
          New: I've been testing various reverse proxies with Jenkins, and am unable to use the CLI over an Apache reverse proxy with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

          I used Wireshark to see what was happening, and the issue appears to be that Apache tries to buffer the body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The [relevant setting in Apache|https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxyreceivebuffersize] has a minimum value of 512, and I have been unable to find a way to otherwise get rid of the buffering.

          I am testing using [https://github.com/jglick/jenkins-demo-reverse-proxy], and attempting to run the following CLI command:
          {noformat}
          java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
          {noformat}
          This is the error that is thrown by Jenkins 2.82:
          {noformat}
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
          {noformat}

          Devin Nusbaum added a comment - - edited

          As a workaround, you can use the CLI over SSH, or use Nginx or HAProxy for your reverse proxy server.

          Devin Nusbaum added a comment - - edited As a workaround, you can use the CLI over SSH , or use Nginx or HAProxy for your reverse proxy server.
          Devin Nusbaum made changes -
          Description Original: I've been testing various reverse proxies with Jenkins, and am unable to use the CLI over an Apache reverse proxy with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

          I used Wireshark to see what was happening, and the issue appears to be that Apache tries to buffer the body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The [relevant setting in Apache|https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxyreceivebuffersize] has a minimum value of 512, and I have been unable to find a way to otherwise get rid of the buffering.

          I am testing using [https://github.com/jglick/jenkins-demo-reverse-proxy], and attempting to run the following CLI command:
          {noformat}
          java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
          {noformat}
          This is the error that is thrown by Jenkins 2.82:
          {noformat}
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
          {noformat}
          New: I've been testing various reverse proxies with Jenkins, and am unable to use the CLI over an Apache reverse proxy with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

          I used Wireshark to see what was happening, and the issue appears to be that Apache tries to buffer the body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The [relevant setting in Apache|https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxyreceivebuffersize] has a minimum value of 512, and I have been unable to find a way to otherwise get rid of the buffering.

          I am testing using [https://github.com/jglick/jenkins-demo-reverse-proxy], and attempting to run the following CLI command:
          {noformat}
          java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
          {noformat}
          This is the error that is thrown by Jenkins 2.82:
          {noformat}
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
          {noformat}

          Assuming that Jenkins' full-duplex HTTP connection is spec compliant, this is arguably a bug in Apache.
          Devin Nusbaum made changes -
          Description Original: I've been testing various reverse proxies with Jenkins, and am unable to use the CLI over an Apache reverse proxy with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

          I used Wireshark to see what was happening, and the issue appears to be that Apache tries to buffer the body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The [relevant setting in Apache|https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxyreceivebuffersize] has a minimum value of 512, and I have been unable to find a way to otherwise get rid of the buffering.

          I am testing using [https://github.com/jglick/jenkins-demo-reverse-proxy], and attempting to run the following CLI command:
          {noformat}
          java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
          {noformat}
          This is the error that is thrown by Jenkins 2.82:
          {noformat}
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
          {noformat}

          Assuming that Jenkins' full-duplex HTTP connection is spec compliant, this is arguably a bug in Apache.
          New: I've been testing various reverse proxies with Jenkins, and am unable to use the CLI over an Apache reverse proxy with the new full-duplex HTTP transport and the plain CLI protocol that was introduced in Jenkins 2.54. Both Nginx and HAProxy work using the full duplex HTTP transport correctly. Accessing Jenkins through my browser over the Apache proxy works correctly.

          I used Wireshark to see what was happening, and the issue appears to be that Apache tries to buffer the body of the upload connection before proxying it to Jenkins, so Jenkins never sees the upload connection. As far as I can tell, the current code for FullDuplexHttpService and FullDuplexHttpStream require that both connections are unbuffered. The [relevant setting in Apache|https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxyreceivebuffersize] has a minimum value of 512, and neither {{SetEnv proxy-sendchunks 1}} nor {{SetEnv proxy-sendchunked 1}} have a noticeable effect.

          I am testing using [https://github.com/jglick/jenkins-demo-reverse-proxy], and attempting to run the following CLI command:
          {noformat}
          java -jar jenkins-cli.jar -s "http://localhost:80/jenkins|http://localhost/jenkins" -logger FINE help
          {noformat}
          This is the error that is thrown by Jenkins 2.82:
          {noformat}
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 829c689a-05cb-4e9a-a559-eeed9c099460
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:108)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:175)
          {noformat}
          Assuming that Jenkins' full-duplex HTTP connection is spec compliant, this is arguably a bug in Apache.

          Jesse Glick added a comment -

          Search JIRA for other issues mentioning HTTP full-duplex channel timeout and link them if appropriate.

          Jesse Glick added a comment - Search JIRA for other issues mentioning HTTP full-duplex channel timeout and link them if appropriate.

            Unassigned Unassigned
            dnusbaum Devin Nusbaum
            Votes:
            7 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: