Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49745

Too many CrumbFilter log entries per second

    XMLWordPrintable

Details

    Description

      The hudson.security.csrf.CrumbFilter generates so many log entries it causes parts of Jenkins to stall until the rate of log messages slows down.

      2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 418axxxx20cb74b577eaae393aa8ac0e. Will check remaining parameters for a valid one...
2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.

       
      The amount of these logs was causing my Jenkins to stop working: The executors were not being released by jobs (even after they were done running) until the log entry could be written.

      I checked through the logs and all the entries I have are for these URLs (there could be more, due to the logs rolling so quick):

      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getDisconnectedSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOfflineSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getRunningJobs
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getTasksInQueue

      I would suggest that either these logs shouldn't be generated at all, or only as FINE or FINER entries.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-40344 Leaving a page open past session expiry fills the logs on the master with "Found invalid crumb" warnings

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            View 4 older comments
            docwhat Christian Höltje added a comment - - edited

            I'm glad my extra research helped identify the culprit (dashboard-view).

            I'm not familiar enough with Jelly to know exactly what's going on but it looks like some of the problems are:

            • It doesn't check the return value (or try/catch if done that way) in case there is an error (e.g. a 403). This means there is nothing to bounce the page to the login page (if it requires auth) or refetch a CSRF token.
            • It fetches everything, even if only some items are needed.
            • This stuff, if everything is needed, could be returned as a single struct.
            • Did I miss anything? I'm not familiar with how the CSRF stuff is implemented in Jenkins.

            It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403.

            docwhat Christian Höltje added a comment - - edited I'm glad my extra research helped identify the culprit (dashboard-view). I'm not familiar enough with Jelly to know exactly what's going on but it looks like some of the problems are: It doesn't check the return value (or try / catch if done that way) in case there is an error (e.g. a 403). This means there is nothing to bounce the page to the login page (if it requires auth) or refetch a CSRF token. It fetches everything, even if only some items are needed. This stuff, if everything is needed, could be returned as a single struct. Did I miss anything? I'm not familiar with how the CSRF stuff is implemented in Jenkins. It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403.
            danielbeck Daniel Beck added a comment -

            It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403.

            GET doesn't have CSRF protection, this sends POST. Possibly because it's the default.

            danielbeck Daniel Beck added a comment - It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403. GET doesn't have CSRF protection, this sends POST. Possibly because it's the default.
            tgr Tobias Gruetzmacher added a comment -

            This is now a bit saner:

            • Only one request for all fields
            • After 10 unsuccessful requests, further updates are suspended

            Test build will probably be available later: https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fdashboard-view-plugin/branches

             

            tgr Tobias Gruetzmacher added a comment - This is now a bit saner: Only one request for all fields After 10 unsuccessful requests, further updates are suspended Test build will probably be available later: https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fdashboard-view-plugin/branches  
            danielbeck Daniel Beck added a comment -
            • After 10 unsuccessful requests, further updates are suspended

            Might be interesting for you to look at what the "Jenkins is (re)starting" screen in Jenkins core does – it distinguishes between getting no response or getting a 500 response from anything different. This way, the UI survives a Jenkins restart, but would show legitimate errors if they appear.

            https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/core/src/main/resources/hudson/util/HudsonIsRestarting/index.jelly#L60

            https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/war/src/main/webapp/scripts/loading.js

            danielbeck Daniel Beck added a comment - After 10 unsuccessful requests, further updates are suspended Might be interesting for you to look at what the "Jenkins is (re)starting" screen in Jenkins core does – it distinguishes between getting no response or getting a 500 response from anything different. This way, the UI survives a Jenkins restart, but would show legitimate errors if they appear. https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/core/src/main/resources/hudson/util/HudsonIsRestarting/index.jelly#L60 https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/war/src/main/webapp/scripts/loading.js
            tgr Tobias Gruetzmacher added a comment -

            Fixed in dashboard-view plugin 2.10.

            tgr Tobias Gruetzmacher added a comment - Fixed in dashboard-view plugin 2.10.

            People

              tgr Tobias Gruetzmacher
              docwhat Christian Höltje
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: