Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61823

"--httpKeepAliveTimeout" no longer has any effect after upgrading to 2.222.1

    • Jenkins 2.243

      After upgrading from Jenkins 2.204.2 to 2.222.1, our builds started failing non-deterministically when trying to download artifacts.
      It seems that Jenkins no longer honors the "--httpKeepAliveTimeout=60000" switch.
      I confirmed that Jenkins is running with this switch.
      Yet a download with a slow consumer will fail with a partial download. (for testing: "curl -f -u $jenkins_auth $artifact_url | (sleep 6; wc -c)" 

      On the server side, the log indicates that the command-line switch was ineffective and the default 5s timeout is still in use:

      java.util.concurrent.TimeoutException: Idle timeout expired: 5000/5000 ms
       at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
       at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
       at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      Caused: java.io.IOException
       at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
       at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)
       at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:825)
       at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:612)
       at org.kohsuke.stapler.ResponseImpl.serveFile(ResponseImpl.java:216)
       at hudson.model.DirectoryBrowserSupport.serveFile(DirectoryBrowserSupport.java:370)
       at hudson.model.DirectoryBrowserSupport.generateResponse(DirectoryBrowserSupport.java:155)
       at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
       at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
       at org.kohsuke.stapler.Function.renderResponse(Function.java:164)
       at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
       at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
      

      The default 5s timeout is short enough that a simple "curl | tar xf" sometimes trips it when unpacking many small files on an overloaded VM. Please consider changing the default back to jetty's 30s.

          [JENKINS-61823] "--httpKeepAliveTimeout" no longer has any effect after upgrading to 2.222.1

          I downgraded to 2.204.6, and now my downloads are complete again.

          Daniel Grunwald added a comment - I downgraded to 2.204.6, and now my downloads are complete again.

          Daniel Beck added a comment -

          olamy Another victim of our Jetty upgrades?

          Daniel Beck added a comment - olamy Another victim of our Jetty upgrades?

          Olivier Lamy added a comment -

          Olivier Lamy added a comment - nope nothing related to Jetty but to this  https://github.com/jenkinsci/winstone/commit/5e1a8982f1412cacbe3bc4d0feb29436918bf8be#r38330848

          Olivier Lamy added a comment -

          temporary workaround until is fixed with new version is to use  --KeepAliveTimeout

          but it will be fixed with next version of winstone and this workaround will not work anymore  

          Olivier Lamy added a comment - temporary workaround until is fixed with new version is to use  --KeepAliveTimeout but it will be fixed with next version of winstone and this workaround will not work anymore  

          Olivier Lamy added a comment -

          Olivier Lamy added a comment - pr  https://github.com/jenkinsci/winstone/pull/100  

          Oleg Nenashev added a comment - - edited

          BACKPORTING NOTES:

          • Winstone master branch includes a patch for Winstone redirects by slide_o_mixhttps://github.com/jenkinsci/winstone/pull/98 
          • The fix is quite trivial, but we may not want to backport it to 2.222.x due to our experience in the past months
          • I can prepare a release of Winstone 5.9.1 with a backport -> DONE

          https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9.1

          Oleg Nenashev added a comment - - edited BACKPORTING NOTES: Winstone master branch includes a patch for Winstone redirects by slide_o_mix :  https://github.com/jenkinsci/winstone/pull/98   The fix is quite trivial, but we may not want to backport it to 2.222.x due to our experience in the past months I can prepare a release of Winstone 5.9.1 with a backport -> DONE https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9.1

          Oleg Nenashev added a comment -

          I submitted https://github.com/jenkinsci/jenkins/pull/4811 with an update for the Jenkins weekly.

           

           

          Oleg Nenashev added a comment - I submitted  https://github.com/jenkinsci/jenkins/pull/4811  with an update for the Jenkins weekly.    

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - https://github.com/jenkinsci/jenkins/pull/4812  for the LTS backport

            olamy Olivier Lamy
            grunwald Daniel Grunwald
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: