Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20640

Jenkins become very slow and had 'proxy error'

XMLWordPrintable

      We have to restart out hanging jenkins instance because of this issue. I'm not changing it to Critical priority.

      Jenkins version : 1.539

      Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /monitoring.

Reason: Error reading from remote server


      WARNING: /monitoring
org.apache.commons.jelly.JellyTagException: jar:file:/tmp/jetty-127.0.0.1-8080-jenkins.war--any-/webapp/WEB-INF/lib/jenkins-core-1.539.jar!/lib/layout/layout.jelly:75:72: <st:include> org.eclipse.jetty.io.EofException
        at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:726)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:279)
        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
        at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
        at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
        at hudson.init.impl.InstallUncaughtExceptionHandler$1.reportException(InstallUncaughtExceptionHandler.java:25)
        at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:63)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
        at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:679)
Caused by: org.eclipse.jetty.io.EofException
        at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
        at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:443)
        at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:100)
        at org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1101)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:161)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
        at com.jcraft.jzlib.DeflaterOutputStream.deflate(DeflaterOutputStream.java:144)
        at com.jcraft.jzlib.DeflaterOutputStream.write(DeflaterOutputStream.java:102)
        at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31)
        at org.kohsuke.stapler.jelly.DefaultScriptInvoker$1.write(DefaultScriptInvoker.java:99)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:290)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:294)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:140)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
        at java.io.BufferedWriter.flush(BufferedWriter.java:253)
        at org.dom4j.io.XMLWriter.flush(XMLWriter.java:276)
        at org.apache.commons.jelly.XMLOutput.flush(XMLOutput.java:127)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271)
        ... 46 more
Caused by: java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:146)
        at sun.nio.ch.SocketChannelImpl.write0(SocketChannelImpl.java:373)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:406)
        at java.nio.channels.SocketChannel.write(SocketChannel.java:384)
        at org.eclipse.jetty.io.nio.ChannelEndPoint.gatheringFlush(ChannelEndPoint.java:376)
        at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:335)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:376)
        at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:838)

        1. http_session.png
          http_session.png
          479 kB

          [JENKINS-20640] Jenkins become very slow and had 'proxy error'

          evernat added a comment -

          In the monitoring plugin, how many sessions are counted?
          And can you attach the image given by the following?
          http://localhost:8080/monitoring?part=graph&graph=httpSessions

          (where localhost and 8080 are the host and port of your server)

          evernat added a comment - In the monitoring plugin, how many sessions are counted? And can you attach the image given by the following? http://localhost:8080/monitoring?part=graph&graph=httpSessions (where localhost and 8080 are the host and port of your server)

          Sam Xiao added a comment -

          evernat
          See attached screenshot, you can clearly see the spike on the http session right after our upgrade to 1.539

          Sam Xiao added a comment - evernat See attached screenshot, you can clearly see the spike on the http session right after our upgrade to 1.539

          evernat added a comment - - edited

          The related issue about HTTP sessions, JENKINS-20352, is now fixed in the monitoring plugin v1.48.
          The monitoring plugin v1.48 should be available in the update center of your Jenkins, or from http://mirrors.jenkins-ci.org/plugins/monitoring/, in a few hours from now (24h max?).

          I suggest to try that version of the plugin.
          But I can't say if that version will fix your issue, or if you have another bigger issue, related to Jetty 8 or anything else.

          evernat added a comment - - edited The related issue about HTTP sessions, JENKINS-20352 , is now fixed in the monitoring plugin v1.48. The monitoring plugin v1.48 should be available in the update center of your Jenkins, or from http://mirrors.jenkins-ci.org/plugins/monitoring/ , in a few hours from now (24h max?). I suggest to try that version of the plugin. But I can't say if that version will fix your issue, or if you have another bigger issue, related to Jetty 8 or anything else.

          Sam Xiao added a comment -

          evernat

          Thanks, but I thought the HTTP session problem is different than the proxy error. Or are they actually the same or related?

          Sam Xiao added a comment - evernat Thanks, but I thought the HTTP session problem is different than the proxy error. Or are they actually the same or related?

          Chris Z added a comment -

          Hi, I've got similar issue. In monitoring you can see peeks of active threads. When I get screenshoot (when jenkins unfreez) i will attach it.

          Chris Z added a comment - Hi, I've got similar issue. In monitoring you can see peeks of active threads. When I get screenshoot (when jenkins unfreez) i will attach it.

          evernat added a comment -

          This issue is perhaps related to the HTTP session issue.
          To be sure that you don't have the HTTP session issue, I suggest to upgrade to the monitoring plugin v1.48 if you use Jenkins v1.535 or later.

          evernat added a comment - This issue is perhaps related to the HTTP session issue. To be sure that you don't have the HTTP session issue, I suggest to upgrade to the monitoring plugin v1.48 if you use Jenkins v1.535 or later.

          Sam Xiao added a comment -

          I don't think these are monitor plugin specific bugs, I've seen the proxy errors almost every endpoints. I think there's an upstream bug somewhere caused this, and only when Jenkins is under heavy load.

          Sam Xiao added a comment - I don't think these are monitor plugin specific bugs, I've seen the proxy errors almost every endpoints. I think there's an upstream bug somewhere caused this, and only when Jenkins is under heavy load.

          Christian Höltje added a comment -

          @evernat Do you have a link to the bug about HTTP sessions? I'm seeing problems (with all endpoints) with both our 1.532.1 Jenkins servers.

          Christian Höltje added a comment - @evernat Do you have a link to the bug about HTTP sessions? I'm seeing problems (with all endpoints) with both our 1.532.1 Jenkins servers.

          evernat added a comment -

          @Christian Höltje
          The bug with HTTP sessions is JENKINS-20532
          It happened since Jenkins v1.535 with the change to Jetty.
          The bug is fixed since the Monitoring plugin v1.48

          evernat added a comment - @Christian Höltje The bug with HTTP sessions is JENKINS-20532 It happened since Jenkins v1.535 with the change to Jetty. The bug is fixed since the Monitoring plugin v1.48

          Oleg Nenashev added a comment -

          Duplicates JENKINS-20074 (Fixed)

          Oleg Nenashev added a comment - Duplicates JENKINS-20074 (Fixed)

            Unassigned Unassigned
            samxiao Sam Xiao
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: