Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37625

exception in logs when hitting /monitoring?action=gc

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have a script that runs GC periodically by hitting "/monitoring?action=gc".

      We upgraded to the latest version of the monitoring plugin and Jenkins and we are now getting the following exception in the logs:

      /monitoring?action=gc
      java.lang.IllegalStateException: Response is committed
      	at org.eclipse.jetty.server.Request.getSession(Request.java:1400)
      	at org.eclipse.jetty.security.authentication.FormAuthenticator.validateRequest(FormAuthenticator.java:260)
      	at org.eclipse.jetty.security.authentication.DeferredAuthentication.authenticate(DeferredAuthentication.java:68)
      	at org.eclipse.jetty.server.Request.getUserPrincipal(Request.java:1479)
      	at org.eclipse.jetty.server.Request.getRemoteUser(Request.java:1087)
      	at winstone.accesslog.SimpleAccessLogger.log(SimpleAccessLogger.java:89)
      	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:113)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:499)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

      It looks benign, but it's something that we haven't seen before.

      I can't tell if the "/monitoring?action=gc" is actually working, but it looks like it is.
      If it is not, I'll upgrade the severity of this ticket.

        Attachments

          Activity

          Hide
          evernat evernat added a comment -

          I have not reproduced the issue using Jenkins 2.21 and monitoring plugin 1.61.0.

          Show
          evernat evernat added a comment - I have not reproduced the issue using Jenkins 2.21 and monitoring plugin 1.61.0.
          Hide
          batmat Baptiste Mathus added a comment - - edited

          I think this bug report is missing a critical piece of information. I think this issue is triggered when the access logging is enabled. We encountered this on some plugin, and I found this issue searching for similar stack trace.

          I just managed to reproduce it with the monitoring plugin.

          evernat add --accessLoggerClassName=winstone.accesslog.SimpleAccessLogger --simpleAccessLogger.format=combined --simpleAccessLogger.file=$(pwd)/access_log to the jenkins startup options and I guess this should show you the stack trace above.

          Alex Gray can you confirm access logging is enabled on your instance?

          Thank you
          HTH

          Show
          batmat Baptiste Mathus added a comment - - edited I think this bug report is missing a critical piece of information. I think this issue is triggered when the access logging is enabled. We encountered this on some plugin, and I found this issue searching for similar stack trace. I just managed to reproduce it with the monitoring plugin. evernat add --accessLoggerClassName=winstone.accesslog.SimpleAccessLogger --simpleAccessLogger.format=combined --simpleAccessLogger.file=$(pwd)/access_log to the jenkins startup options and I guess this should show you the stack trace above. Alex Gray can you confirm access logging is enabled on your instance? Thank you HTH
          Hide
          grayaii Alex Gray added a comment -

          Wow, it is indeed enabled!

          JENKINS_ENABLE_ACCESS_LOG="yes"
          JENKINS_ACCESSLOG="--accessLoggerClassName=winstone.accesslog.SimpleAccessLogger --simpleAccessLogger.format=combined --simpleAccessLogger.file=/var/log/jenkins/access.log"
          
          Show
          grayaii Alex Gray added a comment - Wow, it is indeed enabled! JENKINS_ENABLE_ACCESS_LOG="yes" JENKINS_ACCESSLOG="--accessLoggerClassName=winstone.accesslog.SimpleAccessLogger --simpleAccessLogger.format=combined --simpleAccessLogger.file=/var/log/jenkins/access.log"
          Hide
          evernat evernat added a comment - - edited

          For related info:
          http://stackoverflow.com/questions/41191519/fail-to-get-views-config-xml-via-python-requests-to-secure-jenkins-self-signed

          Alex Gray: Do you have some "user:password" in the url called from the script?

          Anyway, I don't think that this is an issue of the monitoring plugin. I think it is an issue of the winstone SimpleAccessLogger: that logger calls Request.getRemoteUser() at some point which calls authenticate and Request.getSession(true). And so the monitoring plugin probably can't do anything about it.

          Alex Gray: If you can, I suggest as a workaround to execute a system groovy script inside a Jenkins job instead of calling the /monitoring?action=gc URL. The script for GC would be:

          import net.bull.javamelody.*;
          
          before = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
          System.gc();
          after = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
          println I18N.getFormattedString("ramasse_miette_execute", Math.round((before - after) / 1024));
          

          For that you will need the groovy plugin.
          And you can see the documentation with several monitoring scripts

          Show
          evernat evernat added a comment - - edited For related info: http://stackoverflow.com/questions/41191519/fail-to-get-views-config-xml-via-python-requests-to-secure-jenkins-self-signed Alex Gray : Do you have some "user:password" in the url called from the script? Anyway, I don't think that this is an issue of the monitoring plugin. I think it is an issue of the winstone SimpleAccessLogger: that logger calls Request.getRemoteUser() at some point which calls authenticate and Request.getSession(true). And so the monitoring plugin probably can't do anything about it. Alex Gray : If you can, I suggest as a workaround to execute a system groovy script inside a Jenkins job instead of calling the /monitoring?action=gc URL. The script for GC would be: import net.bull.javamelody.*; before = Runtime .getRuntime().totalMemory() - Runtime .getRuntime().freeMemory(); System .gc(); after = Runtime .getRuntime().totalMemory() - Runtime .getRuntime().freeMemory(); println I18N.getFormattedString( "ramasse_miette_execute" , Math .round((before - after) / 1024)); For that you will need the groovy plugin . And you can see the documentation with several monitoring scripts
          Hide
          cleclerc Cyrille Le Clerc added a comment -

          There seem to be a bug in winstone.accesslog.SimpleAccessLogger regarding the remote user.

          Could Jenkins rely on the standard Jetty Access Log implementation with RequestLogHandler and NCSARequestLog ? Are there reasons why Jenkins couldn't use it?

          As we can see on AbstractNCSARequestLog.java#L130, we should test for anonymous before trying to get the remote user.

          https://github.com/eclipse/jetty.project/blob/jetty-9.4.0.v20161208/jetty-server/src/main/java/org/eclipse/jetty/server/AbstractNCSARequestLog.java#L130

          Authentication authentication = request.getAuthentication();
          append(buf,(authentication instanceof Authentication.User)?((Authentication.User)authentication).getUserIdentity().getUserPrincipal().getName():null);
          
          Show
          cleclerc Cyrille Le Clerc added a comment - There seem to be a bug in winstone.accesslog.SimpleAccessLogger regarding the remote user. Could Jenkins rely on the standard Jetty Access Log implementation with RequestLogHandler and NCSARequestLog ? Are there reasons why Jenkins couldn't use it? As we can see on AbstractNCSARequestLog.java#L130 , we should test for anonymous before trying to get the remote user. https://github.com/eclipse/jetty.project/blob/jetty-9.4.0.v20161208/jetty-server/src/main/java/org/eclipse/jetty/server/AbstractNCSARequestLog.java#L130 Authentication authentication = request.getAuthentication(); append(buf,(authentication instanceof Authentication.User)?((Authentication.User)authentication).getUserIdentity().getUserPrincipal().getName(): null );
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Cyrille Le Clerc
          Path:
          src/java/winstone/accesslog/SimpleAccessLogger.java
          http://jenkins-ci.org/commit/winstone/3a1b4fd46de4ab1629b92ff709f8a8d4d1593013
          Log:
          JENKINS-37625 Don’t get the request.remoteUser if it’s a deferred authentication (`org.eclipse.jetty.server.Authentication.Deferred`) to prevent "IllegalStateException: Response is committed"

          ```
          java.lang.IllegalStateException: Response is committed
          at org.eclipse.jetty.server.Request.getSession(Request.java:1400)
          at org.eclipse.jetty.security.authentication.FormAuthenticator.validateRequest(FormAuthenticator.java:260)
          at org.eclipse.jetty.security.authentication.DeferredAuthentication.authenticate(DeferredAuthentication.java:68)
          at org.eclipse.jetty.server.Request.getUserPrincipal(Request.java:1479)
          at org.eclipse.jetty.server.Request.getRemoteUser(Request.java:1087)
          at winstone.accesslog.SimpleAccessLogger.log(SimpleAccessLogger.java:89)
          at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:113)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.server.Server.handle(Server.java:499)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
          at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
          ```

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: src/java/winstone/accesslog/SimpleAccessLogger.java http://jenkins-ci.org/commit/winstone/3a1b4fd46de4ab1629b92ff709f8a8d4d1593013 Log: JENKINS-37625 Don’t get the request.remoteUser if it’s a deferred authentication (`org.eclipse.jetty.server.Authentication.Deferred`) to prevent "IllegalStateException: Response is committed" ``` java.lang.IllegalStateException: Response is committed at org.eclipse.jetty.server.Request.getSession(Request.java:1400) at org.eclipse.jetty.security.authentication.FormAuthenticator.validateRequest(FormAuthenticator.java:260) at org.eclipse.jetty.security.authentication.DeferredAuthentication.authenticate(DeferredAuthentication.java:68) at org.eclipse.jetty.server.Request.getUserPrincipal(Request.java:1479) at org.eclipse.jetty.server.Request.getRemoteUser(Request.java:1087) at winstone.accesslog.SimpleAccessLogger.log(SimpleAccessLogger.java:89) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:113) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ```
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/java/winstone/accesslog/SimpleAccessLogger.java
          http://jenkins-ci.org/commit/winstone/784537203190576eb8523deab79cf035fe6da40a
          Log:
          Merge pull request #31 from cyrille-leclerc/master

          JENKINS-37625 Don’t get the request.remoteUser if it’s a deferred authentication

          Compare: https://github.com/jenkinsci/winstone/compare/e5be8235c72e...784537203190

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/java/winstone/accesslog/SimpleAccessLogger.java http://jenkins-ci.org/commit/winstone/784537203190576eb8523deab79cf035fe6da40a Log: Merge pull request #31 from cyrille-leclerc/master JENKINS-37625 Don’t get the request.remoteUser if it’s a deferred authentication Compare: https://github.com/jenkinsci/winstone/compare/e5be8235c72e...784537203190
          Hide
          jglick Jesse Glick added a comment -

          I believe this is at root a Jetty bug, so I filed it as such.

          Show
          jglick Jesse Glick added a comment - I believe this is at root a Jetty bug, so I filed it as such.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/1e5e53a5fbf1e40ba637f1b21214e0fb8a0bee8b
          Log:
          [FIXED JENKINS-37625] Update Winstone to fix an IllegalStateException.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/1e5e53a5fbf1e40ba637f1b21214e0fb8a0bee8b Log: [FIXED JENKINS-37625] Update Winstone to fix an IllegalStateException.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Jesse Glick
          Path:
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/c732d8a44a0c23ef1399a7480b3b455ce6aaf185
          Log:
          [FIXED JENKINS-37625] Update Winstone to fix an IllegalStateException.

          (cherry picked from commit 1e5e53a5fbf1e40ba637f1b21214e0fb8a0bee8b)

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/c732d8a44a0c23ef1399a7480b3b455ce6aaf185 Log: [FIXED JENKINS-37625] Update Winstone to fix an IllegalStateException. (cherry picked from commit 1e5e53a5fbf1e40ba637f1b21214e0fb8a0bee8b)

            People

            Assignee:
            cleclerc Cyrille Le Clerc
            Reporter:
            grayaii Alex Gray
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: