• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • dashboard-view-plugin
    • None
    • Jenkins 2.89.4

      The hudson.security.csrf.CrumbFilter generates so many log entries it causes parts of Jenkins to stall until the rate of log messages slows down.

      2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 418axxxx20cb74b577eaae393aa8ac0e. Will check remaining parameters for a valid one...
      2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.

       
      The amount of these logs was causing my Jenkins to stop working: The executors were not being released by jobs (even after they were done running) until the log entry could be written.

      I checked through the logs and all the entries I have are for these URLs (there could be more, due to the logs rolling so quick):

      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getDisconnectedSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOfflineSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getRunningJobs
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getTasksInQueue

      I would suggest that either these logs shouldn't be generated at all, or only as FINE or FINER entries.

          [JENKINS-49745] Too many CrumbFilter log entries per second

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - CC wfollonier

          Oleg Nenashev added a comment -

          There were some fixes in the recent core for not showing these warning for requests coming from old open browser pages after restart in 2.82: JENKINS-40344 . Maybe the fix was not complete. Seems docwhat has a discussion there as well. Also CC danielbeck

          Oleg Nenashev added a comment - There were some fixes in the recent core for not showing these warning for requests coming from old open browser pages after restart in 2.82: JENKINS-40344 . Maybe the fix was not complete. Seems docwhat has a discussion there as well. Also CC danielbeck

          Daniel Beck added a comment -

          I consider this to be a bug in Dashboard View Plugin that seems to provide these URLs and doesn't seem to handle 403 well (https://github.com/jenkinsci/dashboard-view-plugin/blob/392aaa7c4631a459ce7af801a66ee0fdd09cab08/src/main/java/hudson/plugins/view/dashboard/stats/StatSlaves.java).

          Daniel Beck added a comment - I consider this to be a bug in Dashboard View Plugin that seems to provide these URLs and doesn't seem to handle 403 well ( https://github.com/jenkinsci/dashboard-view-plugin/blob/392aaa7c4631a459ce7af801a66ee0fdd09cab08/src/main/java/hudson/plugins/view/dashboard/stats/StatSlaves.java ).

          Daniel Beck added a comment - https://github.com/jenkinsci/dashboard-view-plugin/blob/392aaa7c4631a459ce7af801a66ee0fdd09cab08/src/main/resources/hudson/plugins/view/dashboard/stats/StatSlaves/statslaves.jelly#L32...L47 seems to be overly simplistic.

          Christian Höltje added a comment - - edited

          I'm glad my extra research helped identify the culprit (dashboard-view).

          I'm not familiar enough with Jelly to know exactly what's going on but it looks like some of the problems are:

          • It doesn't check the return value (or try/catch if done that way) in case there is an error (e.g. a 403). This means there is nothing to bounce the page to the login page (if it requires auth) or refetch a CSRF token.
          • It fetches everything, even if only some items are needed.
          • This stuff, if everything is needed, could be returned as a single struct.
          • Did I miss anything? I'm not familiar with how the CSRF stuff is implemented in Jenkins.

          It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403.

          Christian Höltje added a comment - - edited I'm glad my extra research helped identify the culprit (dashboard-view). I'm not familiar enough with Jelly to know exactly what's going on but it looks like some of the problems are: It doesn't check the return value (or try / catch if done that way) in case there is an error (e.g. a 403). This means there is nothing to bounce the page to the login page (if it requires auth) or refetch a CSRF token. It fetches everything, even if only some items are needed. This stuff, if everything is needed, could be returned as a single struct. Did I miss anything? I'm not familiar with how the CSRF stuff is implemented in Jenkins. It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403.

          Daniel Beck added a comment -

          It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403.

          GET doesn't have CSRF protection, this sends POST. Possibly because it's the default.

          Daniel Beck added a comment - It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403. GET doesn't have CSRF protection, this sends POST. Possibly because it's the default.

          This is now a bit saner:

          • Only one request for all fields
          • After 10 unsuccessful requests, further updates are suspended

          Test build will probably be available later: https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fdashboard-view-plugin/branches

           

          Tobias Gruetzmacher added a comment - This is now a bit saner: Only one request for all fields After 10 unsuccessful requests, further updates are suspended Test build will probably be available later: https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fdashboard-view-plugin/branches  

          Daniel Beck added a comment -
          • After 10 unsuccessful requests, further updates are suspended

          Might be interesting for you to look at what the "Jenkins is (re)starting" screen in Jenkins core does – it distinguishes between getting no response or getting a 500 response from anything different. This way, the UI survives a Jenkins restart, but would show legitimate errors if they appear.

          https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/core/src/main/resources/hudson/util/HudsonIsRestarting/index.jelly#L60

          https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/war/src/main/webapp/scripts/loading.js

          Daniel Beck added a comment - After 10 unsuccessful requests, further updates are suspended Might be interesting for you to look at what the "Jenkins is (re)starting" screen in Jenkins core does – it distinguishes between getting no response or getting a 500 response from anything different. This way, the UI survives a Jenkins restart, but would show legitimate errors if they appear. https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/core/src/main/resources/hudson/util/HudsonIsRestarting/index.jelly#L60 https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/war/src/main/webapp/scripts/loading.js

          Fixed in dashboard-view plugin 2.10.

          Tobias Gruetzmacher added a comment - Fixed in dashboard-view plugin 2.10.

            tgr Tobias Gruetzmacher
            docwhat Christian Höltje
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: