Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61823

"--httpKeepAliveTimeout" no longer has any effect after upgrading to 2.222.1

    • Jenkins 2.243

      After upgrading from Jenkins 2.204.2 to 2.222.1, our builds started failing non-deterministically when trying to download artifacts.
      It seems that Jenkins no longer honors the "--httpKeepAliveTimeout=60000" switch.
      I confirmed that Jenkins is running with this switch.
      Yet a download with a slow consumer will fail with a partial download. (for testing: "curl -f -u $jenkins_auth $artifact_url | (sleep 6; wc -c)" 

      On the server side, the log indicates that the command-line switch was ineffective and the default 5s timeout is still in use:

      java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms
       at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
       at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
       at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      Caused: java.io.IOException
       at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
       at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)
       at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:825)
       at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:612)
       at org.kohsuke.stapler.ResponseImpl.serveFile(ResponseImpl.java:216)
       at hudson.model.DirectoryBrowserSupport.serveFile(DirectoryBrowserSupport.java:370)
       at hudson.model.DirectoryBrowserSupport.generateResponse(DirectoryBrowserSupport.java:155)
       at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
       at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
       at org.kohsuke.stapler.Function.renderResponse(Function.java:164)
       at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
       at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
      

      The default 5s timeout is short enough that a simple "curl | tar xf" sometimes trips it when unpacking many small files on an overloaded VM. Please consider changing the default back to jetty's 30s.

          [JENKINS-61823] "--httpKeepAliveTimeout" no longer has any effect after upgrading to 2.222.1

          Daniel Grunwald created issue -
          Daniel Grunwald made changes -
          Link New: This issue is related to JENKINS-42513 [ JENKINS-42513 ]
          Daniel Grunwald made changes -
          Link New: This issue is related to JENKINS-39909 [ JENKINS-39909 ]
          Daniel Grunwald made changes -
          Description Original: After upgrading from Jenkins 2.204.2 to 2.222.1, our builds started failing non-deterministically when trying to download artifacts.
          It seems that Jenkins no longer honors the "\-\-httpKeepAliveTimeout=60000" switch.
          I confirmed that Jenkins is running with switch.
          Yet a download with a slow consumer will fail with a partial download. (for testing: "curl -f -u $jenkins_auth $artifact_url | (sleep 6; wc -c)" 

          On the server side, the log indicates that the command-line switch was ineffective and the default 5s timeout is still in use:

          {code}
          java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms
           at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
           at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
           at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
           at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
           at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
           at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
           at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          Caused: java.io.IOException
           at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
           at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)
           at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:825)
           at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:612)
           at org.kohsuke.stapler.ResponseImpl.serveFile(ResponseImpl.java:216)
           at hudson.model.DirectoryBrowserSupport.serveFile(DirectoryBrowserSupport.java:370)
           at hudson.model.DirectoryBrowserSupport.generateResponse(DirectoryBrowserSupport.java:155)
           at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
           at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
           at org.kohsuke.stapler.Function.renderResponse(Function.java:164)
           at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
           at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
          {code}

          The default 5s timeout is short enough that a simple "curl | tar xf" sometimes trips it when unpacking many small files on an overloaded VM. Please consider changing the default back to jetty's 30s.
          New: After upgrading from Jenkins 2.204.2 to 2.222.1, our builds started failing non-deterministically when trying to download artifacts.
          It seems that Jenkins no longer honors the "\-\-httpKeepAliveTimeout=60000" switch.
          I confirmed that Jenkins is running with this switch.
          Yet a download with a slow consumer will fail with a partial download. (for testing: "curl -f -u $jenkins_auth $artifact_url | (sleep 6; wc -c)" 

          On the server side, the log indicates that the command-line switch was ineffective and the default 5s timeout is still in use:

          {code}
          java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms
           at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
           at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
           at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
           at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
           at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
           at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
           at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          Caused: java.io.IOException
           at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
           at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)
           at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:825)
           at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:612)
           at org.kohsuke.stapler.ResponseImpl.serveFile(ResponseImpl.java:216)
           at hudson.model.DirectoryBrowserSupport.serveFile(DirectoryBrowserSupport.java:370)
           at hudson.model.DirectoryBrowserSupport.generateResponse(DirectoryBrowserSupport.java:155)
           at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
           at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
           at org.kohsuke.stapler.Function.renderResponse(Function.java:164)
           at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
           at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
          {code}

          The default 5s timeout is short enough that a simple "curl | tar xf" sometimes trips it when unpacking many small files on an overloaded VM. Please consider changing the default back to jetty's 30s.
          Daniel Grunwald made changes -
          Environment New: Debian 10 with jenkins from http://pkg.jenkins-ci.org/debian-stable

          I downgraded to 2.204.6, and now my downloads are complete again.

          Daniel Grunwald added a comment - I downgraded to 2.204.6, and now my downloads are complete again.
          Daniel Grunwald made changes -
          Priority Original: Minor [ 4 ] New: Critical [ 2 ]

          Daniel Beck added a comment -

          olamy Another victim of our Jetty upgrades?

          Daniel Beck added a comment - olamy Another victim of our Jetty upgrades?
          Olivier Lamy made changes -
          Assignee New: Olivier Lamy [ olamy ]

          Olivier Lamy added a comment -

          Olivier Lamy added a comment - nope nothing related to Jetty but to this  https://github.com/jenkinsci/winstone/commit/5e1a8982f1412cacbe3bc4d0feb29436918bf8be#r38330848

            olamy Olivier Lamy
            grunwald Daniel Grunwald
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: