Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38720

500 Server Error when notifying master of new mercurial commit: "Response header too large"

    XMLWordPrintable

Details

    • Bug
    • Status: In Review (View Workflow)
    • Major
    • Resolution: Unresolved
    • mercurial-plugin
    • Jenkins 2.24, Linux vmkvmx01 2.6.32-431.29.2.el6.x86_64 #1 SMP Sun Jul 27 15:55:46 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

    Description

      Notifying Jenkins of a new mercurial commit via /mercurial/notifyCommit responds with a '500 Server Error'. It looks like the notification is accepted in the Jenkins logs but an exception is also raised. The logs also show all the job polling that is triggered, perhaps related to the issue?

      curl -v  http://jenkins.bld.cbr.internal.example.com:8080/mercurial/notifyCommit?url=ssh://hg@scm01.dev.cbr.internal.example.com/example-source
      *   Trying 192.168.12.1...
      * Connected to jenkins.bld.cbr.internal.example.com (192.168.12.1) port 8080 (#0)
      > GET /mercurial/notifyCommit?url=ssh://hg@scm01.dev.cbr.internal.example.com/example-source HTTP/1.1
      > Host: jenkins.bld.cbr.internal.example.com:8080
      > User-Agent: curl/7.43.0
      > Accept: */*
      > 
      < HTTP/1.1 500 Server Error
      < Connection: close
      < Server: Jetty(9.2.z-SNAPSHOT)
      < 
      * Closing connection 0
      

      Log snippet:

      Oct 05, 2016 4:26:47 PM hudson.plugins.mercurial.MercurialStatus handleNotifyCommit
      INFO: Scheduling example-source for refresh
      Oct 05, 2016 4:26:47 PM org.eclipse.jetty.util.log.JavaUtilLog warn
      WARNING: Commit failed
      java.io.IOException: Response header too large
      	at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:402)
      	at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:655)
      	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
      	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
      	at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:471)
      	at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:763)
      	at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)
      	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)
      	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:470)
      	at org.eclipse.jetty.server.Iso88591HttpWriter.write(Iso88591HttpWriter.java:47)
      	at org.eclipse.jetty.server.HttpWriter.write(HttpWriter.java:71)
      	at java.io.Writer.write(Writer.java:157)
      	at java.io.PrintWriter.newLine(PrintWriter.java:480)
      	at java.io.PrintWriter.println(PrintWriter.java:629)
      	at java.io.PrintWriter.println(PrintWriter.java:740)
      	at hudson.plugins.mercurial.MercurialStatus$1.generateResponse(MercurialStatus.java:175)
      	at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
      	at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
      	at org.kohsuke.stapler.Function.renderResponse(Function.java:119)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:102)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:499)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.nio.BufferOverflowException
      	at java.nio.Buffer.nextPutIndex(Buffer.java:521)
      	at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:169)
      	at org.eclipse.jetty.http.HttpGenerator.putSanitisedName(HttpGenerator.java:1052)
      	at org.eclipse.jetty.http.HttpGenerator.putTo(HttpGenerator.java:1086)
      	at org.eclipse.jetty.http.HttpGenerator.generateHeaders(HttpGenerator.java:705)
      	at org.eclipse.jetty.http.HttpGenerator.generateResponse(HttpGenerator.java:387)
      	... 64 more
      
      
      

      Attachments

        Issue Links

          Activity

            jglick Jesse Glick added a comment -
            INFO: Triggering polling of <example job name>
            

            looks pretty fishy.

            jglick Jesse Glick added a comment - INFO: Triggering polling of <example job name> looks pretty fishy.

            jglick hah, I know. It was my attempt to de-identify the logs. They are all unique job names however.

            sonneveldsmartward Nick Sonneveld added a comment - jglick hah, I know. It was my attempt to de-identify the logs. They are all unique job names however.
            jglick Jesse Glick added a comment -

            Well, then you just have a huge number of jobs all using the same repository, I guess? So MercurialStatus needs to be patched to cap the number of job names it will enumerate in its header.

            jglick Jesse Glick added a comment - Well, then you just have a huge number of jobs all using the same repository, I guess? So MercurialStatus needs to be patched to cap the number of job names it will enumerate in its header.

            jglick Hrm, I wonder if you can generate the "Response header too large" exception while adding properties to the header. It looks like the size isn't checked until the header is flushed just before the body starts being written.

            I'm certainly not checking the results in the header but others might. Should it just be a cap with no warning? Or should there be a further header with number of projects triggered? (with a possible "truncated" property as well?)

            sonneveldsmartward Nick Sonneveld added a comment - jglick Hrm, I wonder if you can generate the "Response header too large" exception while adding properties to the header. It looks like the size isn't checked until the header is flushed just before the body starts being written. I'm certainly not checking the results in the header but others might. Should it just be a cap with no warning? Or should there be a further header with number of projects triggered? (with a possible "truncated" property as well?)
            jglick Jesse Glick added a comment -

            Should it just be a cap with no warning?

            Maybe some kind of notation indicating that the list was truncated, so scripts relying on the header would fail predictably.

            jglick Jesse Glick added a comment - Should it just be a cap with no warning? Maybe some kind of notation indicating that the list was truncated, so scripts relying on the header would fail predictably.
            jglick Jesse Glick added a comment -

            Similar code exists in git-plugin but strangely I can find no analogous issue.

            jglick Jesse Glick added a comment - Similar code exists in git-plugin but strangely I can find no analogous issue.
            default John Default added a comment -

            To remove the strangeness, I just ran into the same problem (500 Server Error,Commit failed,Response header too large) using only git repos. (Jenkins 2.46.3, Git plugin 3.3.0, Git-client plugin 2.4.6)

             

            default John Default added a comment - To remove the strangeness, I just ran into the same problem (500 Server Error,Commit failed,Response header too large) using only git repos. (Jenkins 2.46.3, Git plugin 3.3.0, Git-client plugin 2.4.6)  
            jglick Jesse Glick added a comment -

            default then file an issue in git-plugin linked to this one.

            jglick Jesse Glick added a comment - default then file an issue in git-plugin linked to this one.

            People

              Unassigned Unassigned
              sonneveldsmartward Nick Sonneveld
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: