Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40344

Leaving a page open past session expiry fills the logs on the master with "Found invalid crumb" warnings

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • None

      I noticed that I had thousands of WARNING messages in my master logs this morning because some users are leaving Jenkins home pages open past the user's session expiry.

      I understand that part of the problem here is the busy-wait looping on /ajaxBuildQueue, but finding an entire log file filled with this garbage seems like a bug

      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      

          [JENKINS-40344] Leaving a page open past session expiry fills the logs on the master with "Found invalid crumb" warnings

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/security/csrf/CrumbFilter.java
          http://jenkins-ci.org/commit/jenkins/5c98cf41afdfe15e4e82d13c9a019cb74c65461c
          Log:
          JENKINS-40344 Don't log second warning for anon either

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java http://jenkins-ci.org/commit/jenkins/5c98cf41afdfe15e4e82d13c9a019cb74c65461c Log: JENKINS-40344 Don't log second warning for anon either

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/security/csrf/CrumbFilter.java
          http://jenkins-ci.org/commit/jenkins/a798750f4a8b461045ffc6079e0db6d233bfd2d9
          Log:
          JENKINS-40344 Log the user whose crumb was invalid

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java http://jenkins-ci.org/commit/jenkins/a798750f4a8b461045ffc6079e0db6d233bfd2d9 Log: JENKINS-40344 Log the user whose crumb was invalid

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/security/csrf/CrumbFilter.java
          http://jenkins-ci.org/commit/jenkins/6d7f5a0e94ef20f2a7b3f58f4b04aeec799f33fc
          Log:
          JENKINS-40344 Fix check for anonymous authentication

          Same check as User.get(Authentication) uses, so this should work

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java http://jenkins-ci.org/commit/jenkins/6d7f5a0e94ef20f2a7b3f58f4b04aeec799f33fc Log: JENKINS-40344 Fix check for anonymous authentication Same check as User.get(Authentication) uses, so this should work

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/security/csrf/CrumbFilter.java
          http://jenkins-ci.org/commit/jenkins/576f5b5c0d0d4932dff874ce5ec766e14c28f0c6
          Log:
          Merge pull request #3049 from daniel-beck/JENKINS-40344

          JENKINS-40344 Don't log warning for invalid crumb from anon

          Compare: https://github.com/jenkinsci/jenkins/compare/be7ac438e013...576f5b5c0d0d

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java http://jenkins-ci.org/commit/jenkins/576f5b5c0d0d4932dff874ce5ec766e14c28f0c6 Log: Merge pull request #3049 from daniel-beck/ JENKINS-40344 JENKINS-40344 Don't log warning for invalid crumb from anon Compare: https://github.com/jenkinsci/jenkins/compare/be7ac438e013...576f5b5c0d0d

          Daniel Beck added a comment -

          Fixed towards 2.82.

          Daniel Beck added a comment - Fixed towards 2.82.

          \O/ Thanks danielbeck

          Arnaud Héritier added a comment - \O/ Thanks danielbeck

          Christian Höltje added a comment - - edited

          This isn't really fixed.  I have had to resort to changing the log levels (the URL /log/levels) to prevent it from logging.

          I'm seeing things like this (from the support logs, because it was more informative):

          2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 418axxxx20cb74b577eaae393aa8ac0e. Will check remaining parameters for a valid one...
          2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.

           

          The amount of these logs was causing my Jenkins to stop working: The executors were not being released by jobs (even after they were done running) until the log entry could be written.

          I checked through the logs and all the entries I have are for these URLs (there could be more, due to the logs rolling so quick):

          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getDisconnectedSlaves
          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOfflineSlaves
          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves
          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getRunningJobs
          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getSlaves
          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getTasksInQueue

           

          Christian Höltje added a comment - - edited This isn't really fixed.  I have had to resort to changing the log levels (the URL /log/levels) to prevent it from logging. I'm seeing things like this (from the support logs, because it was more informative): 2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 418axxxx20cb74b577eaae393aa8ac0e. Will check remaining parameters for a valid one... 2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.   The amount of these logs was causing my Jenkins to stop working: The executors were not being released by jobs (even after they were done running) until the log entry could be written. I checked through the logs and all the entries I have are for these URLs (there could be more, due to the logs rolling so quick): /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getDisconnectedSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOfflineSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getRunningJobs /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getTasksInQueue  

          Daniel Beck added a comment -

          docwhat What version of Jenkins?

          Daniel Beck added a comment - docwhat What version of Jenkins?

          Daniel Beck added a comment -

          Even on current versions of Jenkins, this should still happen for docwhat. The error message explains why:

           2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.

          This seems to be about a different user (joecool) having logged in since, or a crumb issuer that takes session information into account. IOW, it's not just an expired session, there's another valid session.

          The problem and fix here was about a logged out (session expired) user spamming the log; you're asking for no log messages when a logged in user sends a crumb that's invalid for them. That is a different issue.

          Daniel Beck added a comment - Even on current versions of Jenkins, this should still happen for docwhat . The error message explains why: 2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403. This seems to be about a different user (joecool) having logged in since, or a crumb issuer that takes session information into account. IOW, it's not just an expired session, there's another valid session. The problem and fix here was about a logged out (session expired) user spamming the log; you're asking for no log messages when a logged in user sends a crumb that's invalid for them. That is a different issue.

          The Jenkins version is 2.89.4.

          I'll open a new ticket for my case.  Thanks!

          Christian Höltje added a comment - The Jenkins version is 2.89.4. I'll open a new ticket for my case.  Thanks!

            danielbeck Daniel Beck
            rtyler R. Tyler Croy
            Votes:
            4 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: