Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38720

500 Server Error when notifying master of new mercurial commit: "Response header too large"

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • mercurial-plugin
    • Jenkins 2.24, Linux vmkvmx01 2.6.32-431.29.2.el6.x86_64 #1 SMP Sun Jul 27 15:55:46 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

      Notifying Jenkins of a new mercurial commit via /mercurial/notifyCommit responds with a '500 Server Error'. It looks like the notification is accepted in the Jenkins logs but an exception is also raised. The logs also show all the job polling that is triggered, perhaps related to the issue?

      curl -v  http://jenkins.bld.cbr.internal.example.com:8080/mercurial/notifyCommit?url=ssh://hg@scm01.dev.cbr.internal.example.com/example-source
      *   Trying 192.168.12.1...
      * Connected to jenkins.bld.cbr.internal.example.com (192.168.12.1) port 8080 (#0)
      > GET /mercurial/notifyCommit?url=ssh://hg@scm01.dev.cbr.internal.example.com/example-source HTTP/1.1
      > Host: jenkins.bld.cbr.internal.example.com:8080
      > User-Agent: curl/7.43.0
      > Accept: */*
      > 
      < HTTP/1.1 500 Server Error
      < Connection: close
      < Server: Jetty(9.2.z-SNAPSHOT)
      < 
      * Closing connection 0
      

      Log snippet:

      Oct 05, 2016 4:26:47 PM hudson.plugins.mercurial.MercurialStatus handleNotifyCommit
      INFO: Scheduling example-source for refresh
      Oct 05, 2016 4:26:47 PM org.eclipse.jetty.util.log.JavaUtilLog warn
      WARNING: Commit failed
      java.io.IOException: Response header too large
      	at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:402)
      	at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:655)
      	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
      	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
      	at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:471)
      	at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:763)
      	at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)
      	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)
      	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:470)
      	at org.eclipse.jetty.server.Iso88591HttpWriter.write(Iso88591HttpWriter.java:47)
      	at org.eclipse.jetty.server.HttpWriter.write(HttpWriter.java:71)
      	at java.io.Writer.write(Writer.java:157)
      	at java.io.PrintWriter.newLine(PrintWriter.java:480)
      	at java.io.PrintWriter.println(PrintWriter.java:629)
      	at java.io.PrintWriter.println(PrintWriter.java:740)
      	at hudson.plugins.mercurial.MercurialStatus$1.generateResponse(MercurialStatus.java:175)
      	at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
      	at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
      	at org.kohsuke.stapler.Function.renderResponse(Function.java:119)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:102)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:499)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.nio.BufferOverflowException
      	at java.nio.Buffer.nextPutIndex(Buffer.java:521)
      	at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:169)
      	at org.eclipse.jetty.http.HttpGenerator.putSanitisedName(HttpGenerator.java:1052)
      	at org.eclipse.jetty.http.HttpGenerator.putTo(HttpGenerator.java:1086)
      	at org.eclipse.jetty.http.HttpGenerator.generateHeaders(HttpGenerator.java:705)
      	at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:387)
      	... 64 more
      
      
      

          [JENKINS-38720] 500 Server Error when notifying master of new mercurial commit: "Response header too large"

          Jesse Glick added a comment -
          INFO: Triggering polling of <example job name>
          

          looks pretty fishy.

          Jesse Glick added a comment - INFO: Triggering polling of <example job name> looks pretty fishy.

          jglick hah, I know. It was my attempt to de-identify the logs. They are all unique job names however.

          Nick Sonneveld added a comment - jglick hah, I know. It was my attempt to de-identify the logs. They are all unique job names however.

          Jesse Glick added a comment -

          Well, then you just have a huge number of jobs all using the same repository, I guess? So MercurialStatus needs to be patched to cap the number of job names it will enumerate in its header.

          Jesse Glick added a comment - Well, then you just have a huge number of jobs all using the same repository, I guess? So MercurialStatus needs to be patched to cap the number of job names it will enumerate in its header.

          jglick Hrm, I wonder if you can generate the "Response header too large" exception while adding properties to the header. It looks like the size isn't checked until the header is flushed just before the body starts being written.

          I'm certainly not checking the results in the header but others might. Should it just be a cap with no warning? Or should there be a further header with number of projects triggered? (with a possible "truncated" property as well?)

          Nick Sonneveld added a comment - jglick Hrm, I wonder if you can generate the "Response header too large" exception while adding properties to the header. It looks like the size isn't checked until the header is flushed just before the body starts being written. I'm certainly not checking the results in the header but others might. Should it just be a cap with no warning? Or should there be a further header with number of projects triggered? (with a possible "truncated" property as well?)

          Jesse Glick added a comment -

          Should it just be a cap with no warning?

          Maybe some kind of notation indicating that the list was truncated, so scripts relying on the header would fail predictably.

          Jesse Glick added a comment - Should it just be a cap with no warning? Maybe some kind of notation indicating that the list was truncated, so scripts relying on the header would fail predictably.

          Jesse Glick added a comment -

          Similar code exists in git-plugin but strangely I can find no analogous issue.

          Jesse Glick added a comment - Similar code exists in git-plugin but strangely I can find no analogous issue.

          John Default added a comment -

          To remove the strangeness, I just ran into the same problem (500 Server Error,Commit failed,Response header too large) using only git repos. (Jenkins 2.46.3, Git plugin 3.3.0, Git-client plugin 2.4.6)

           

          John Default added a comment - To remove the strangeness, I just ran into the same problem (500 Server Error,Commit failed,Response header too large) using only git repos. (Jenkins 2.46.3, Git plugin 3.3.0, Git-client plugin 2.4.6)  

          Jesse Glick added a comment -

          default then file an issue in git-plugin linked to this one.

          Jesse Glick added a comment - default then file an issue in git-plugin linked to this one.

            Unassigned Unassigned
            sonneveldsmartward Nick Sonneveld
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: