Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20640

Jenkins become very slow and had 'proxy error'

    XMLWordPrintable

Details

    Description

      We have to restart out hanging jenkins instance because of this issue. I'm not changing it to Critical priority.

      Jenkins version : 1.539

      Proxy Error
      
      The proxy server received an invalid response from an upstream server.
      The proxy server could not handle the request GET /monitoring.
      
      Reason: Error reading from remote server
      
      
      WARNING: /monitoring
      org.apache.commons.jelly.JellyTagException: jar:file:/tmp/jetty-127.0.0.1-8080-jenkins.war--any-/webapp/WEB-INF/lib/jenkins-core-1.539.jar!/lib/layout/layout.jelly:75:72: <st:include> org.eclipse.jetty.io.EofException
              at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:726)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:279)
              at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
              at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
              at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
              at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
              at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
              at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
              at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
              at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
              at hudson.init.impl.InstallUncaughtExceptionHandler$1.reportException(InstallUncaughtExceptionHandler.java:25)
              at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:63)
              at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
              at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
              at org.eclipse.jetty.server.Server.handle(Server.java:370)
              at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
              at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
              at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
              at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
              at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
              at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
              at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:679)
      Caused by: org.eclipse.jetty.io.EofException
              at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
              at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:443)
              at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:100)
              at org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1101)
              at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:161)
              at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
              at com.jcraft.jzlib.DeflaterOutputStream.deflate(DeflaterOutputStream.java:144)
              at com.jcraft.jzlib.DeflaterOutputStream.write(DeflaterOutputStream.java:102)
              at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31)
              at org.kohsuke.stapler.jelly.DefaultScriptInvoker$1.write(DefaultScriptInvoker.java:99)
              at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220)
              at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:290)
              at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:294)
              at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:140)
              at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
              at java.io.BufferedWriter.flush(BufferedWriter.java:253)
              at org.dom4j.io.XMLWriter.flush(XMLWriter.java:276)
              at org.apache.commons.jelly.XMLOutput.flush(XMLOutput.java:127)
              at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271)
              ... 46 more
      Caused by: java.nio.channels.ClosedChannelException
              at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:146)
              at sun.nio.ch.SocketChannelImpl.write0(SocketChannelImpl.java:373)
              at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:406)
              at java.nio.channels.SocketChannel.write(SocketChannel.java:384)
              at org.eclipse.jetty.io.nio.ChannelEndPoint.gatheringFlush(ChannelEndPoint.java:376)
              at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:335)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:376)
              at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:838)
      

      Attachments

        Issue Links

          Activity

            evernat evernat added a comment -

            This issue is perhaps related to the HTTP session issue.
            To be sure that you don't have the HTTP session issue, I suggest to upgrade to the monitoring plugin v1.48 if you use Jenkins v1.535 or later.

            evernat evernat added a comment - This issue is perhaps related to the HTTP session issue. To be sure that you don't have the HTTP session issue, I suggest to upgrade to the monitoring plugin v1.48 if you use Jenkins v1.535 or later.
            samxiao Sam Xiao added a comment -

            I don't think these are monitor plugin specific bugs, I've seen the proxy errors almost every endpoints. I think there's an upstream bug somewhere caused this, and only when Jenkins is under heavy load.

            samxiao Sam Xiao added a comment - I don't think these are monitor plugin specific bugs, I've seen the proxy errors almost every endpoints. I think there's an upstream bug somewhere caused this, and only when Jenkins is under heavy load.

            @evernat Do you have a link to the bug about HTTP sessions? I'm seeing problems (with all endpoints) with both our 1.532.1 Jenkins servers.

            docwhat Christian Höltje added a comment - @evernat Do you have a link to the bug about HTTP sessions? I'm seeing problems (with all endpoints) with both our 1.532.1 Jenkins servers.
            evernat evernat added a comment -

            @Christian Höltje
            The bug with HTTP sessions is JENKINS-20532
            It happened since Jenkins v1.535 with the change to Jetty.
            The bug is fixed since the Monitoring plugin v1.48

            evernat evernat added a comment - @Christian Höltje The bug with HTTP sessions is JENKINS-20532 It happened since Jenkins v1.535 with the change to Jetty. The bug is fixed since the Monitoring plugin v1.48
            oleg_nenashev Oleg Nenashev added a comment -

            Duplicates JENKINS-20074 (Fixed)

            oleg_nenashev Oleg Nenashev added a comment - Duplicates JENKINS-20074 (Fixed)

            People

              Unassigned Unassigned
              samxiao Sam Xiao
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: