Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40344

Leaving a page open past session expiry fills the logs on the master with "Found invalid crumb" warnings

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • None

      I noticed that I had thousands of WARNING messages in my master logs this morning because some users are leaving Jenkins home pages open past the user's session expiry.

      I understand that part of the problem here is the busy-wait looping on /ajaxBuildQueue, but finding an entire log file filled with this garbage seems like a bug

      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      

          [JENKINS-40344] Leaving a page open past session expiry fills the logs on the master with "Found invalid crumb" warnings

          Oleg Nenashev added a comment -

          I agree. Ideally the widget should show the session expiration warning (or the entire page)

          Oleg Nenashev added a comment - I agree. Ideally the widget should show the session expiration warning (or the entire page)

          and the widget should stop to do that requests ....

          Note that each invalid request is generating 2 warning lines :

          WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
          Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
          WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
          Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
          

          For me we should have a protection like for exceptions to avoid to fill logs with the same error repeated again and again

          This bug makes these warning useless and for now the only workaround is to configure the logger "hudson.security.csrf.CrumbFilter" to the level "severe" and thus to discard that messages

          cc oleg_nenashev dbell

          Arnaud Héritier added a comment - and the widget should stop to do that requests .... Note that each invalid request is generating 2 warning lines : WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753. Will check remaining parameters for a valid one... Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403. Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter For me we should have a protection like for exceptions to avoid to fill logs with the same error repeated again and again This bug makes these warning useless and for now the only workaround is to configure the logger "hudson.security.csrf.CrumbFilter" to the level "severe" and thus to discard that messages cc oleg_nenashev dbell

          Monique Maker added a comment -

          I hit this issue after an upgrade from Jenkins 2.80 to 2.81. The downgrade to 2.80 solved the problem.

          Monique Maker added a comment - I hit this issue after an upgrade from Jenkins 2.80 to 2.81. The downgrade to 2.80 solved the problem.

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/security/csrf/CrumbFilter.java
          http://jenkins-ci.org/commit/jenkins/50c26b945e712edfbfc82553f8e0014a0329cd3f
          Log:
          JENKINS-40344 Don't log warning for invalid crumb from anon

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java http://jenkins-ci.org/commit/jenkins/50c26b945e712edfbfc82553f8e0014a0329cd3f Log: JENKINS-40344 Don't log warning for invalid crumb from anon

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/security/csrf/CrumbFilter.java
          http://jenkins-ci.org/commit/jenkins/5c98cf41afdfe15e4e82d13c9a019cb74c65461c
          Log:
          JENKINS-40344 Don't log second warning for anon either

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java http://jenkins-ci.org/commit/jenkins/5c98cf41afdfe15e4e82d13c9a019cb74c65461c Log: JENKINS-40344 Don't log second warning for anon either

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/security/csrf/CrumbFilter.java
          http://jenkins-ci.org/commit/jenkins/a798750f4a8b461045ffc6079e0db6d233bfd2d9
          Log:
          JENKINS-40344 Log the user whose crumb was invalid

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java http://jenkins-ci.org/commit/jenkins/a798750f4a8b461045ffc6079e0db6d233bfd2d9 Log: JENKINS-40344 Log the user whose crumb was invalid

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/security/csrf/CrumbFilter.java
          http://jenkins-ci.org/commit/jenkins/6d7f5a0e94ef20f2a7b3f58f4b04aeec799f33fc
          Log:
          JENKINS-40344 Fix check for anonymous authentication

          Same check as User.get(Authentication) uses, so this should work

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java http://jenkins-ci.org/commit/jenkins/6d7f5a0e94ef20f2a7b3f58f4b04aeec799f33fc Log: JENKINS-40344 Fix check for anonymous authentication Same check as User.get(Authentication) uses, so this should work

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/security/csrf/CrumbFilter.java
          http://jenkins-ci.org/commit/jenkins/576f5b5c0d0d4932dff874ce5ec766e14c28f0c6
          Log:
          Merge pull request #3049 from daniel-beck/JENKINS-40344

          JENKINS-40344 Don't log warning for invalid crumb from anon

          Compare: https://github.com/jenkinsci/jenkins/compare/be7ac438e013...576f5b5c0d0d

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/security/csrf/CrumbFilter.java http://jenkins-ci.org/commit/jenkins/576f5b5c0d0d4932dff874ce5ec766e14c28f0c6 Log: Merge pull request #3049 from daniel-beck/ JENKINS-40344 JENKINS-40344 Don't log warning for invalid crumb from anon Compare: https://github.com/jenkinsci/jenkins/compare/be7ac438e013...576f5b5c0d0d

          Daniel Beck added a comment -

          Fixed towards 2.82.

          Daniel Beck added a comment - Fixed towards 2.82.

          \O/ Thanks danielbeck

          Arnaud Héritier added a comment - \O/ Thanks danielbeck

          Christian Höltje added a comment - - edited

          This isn't really fixed.  I have had to resort to changing the log levels (the URL /log/levels) to prevent it from logging.

          I'm seeing things like this (from the support logs, because it was more informative):

          2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 418axxxx20cb74b577eaae393aa8ac0e. Will check remaining parameters for a valid one...
          2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.

           

          The amount of these logs was causing my Jenkins to stop working: The executors were not being released by jobs (even after they were done running) until the log entry could be written.

          I checked through the logs and all the entries I have are for these URLs (there could be more, due to the logs rolling so quick):

          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getDisconnectedSlaves
          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOfflineSlaves
          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves
          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getRunningJobs
          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getSlaves
          • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getTasksInQueue

           

          Christian Höltje added a comment - - edited This isn't really fixed.  I have had to resort to changing the log levels (the URL /log/levels) to prevent it from logging. I'm seeing things like this (from the support logs, because it was more informative): 2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 418axxxx20cb74b577eaae393aa8ac0e. Will check remaining parameters for a valid one... 2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.   The amount of these logs was causing my Jenkins to stop working: The executors were not being released by jobs (even after they were done running) until the log entry could be written. I checked through the logs and all the entries I have are for these URLs (there could be more, due to the logs rolling so quick): /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getDisconnectedSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOfflineSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getRunningJobs /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getTasksInQueue  

          Daniel Beck added a comment -

          docwhat What version of Jenkins?

          Daniel Beck added a comment - docwhat What version of Jenkins?

          Daniel Beck added a comment -

          Even on current versions of Jenkins, this should still happen for docwhat. The error message explains why:

           2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.

          This seems to be about a different user (joecool) having logged in since, or a crumb issuer that takes session information into account. IOW, it's not just an expired session, there's another valid session.

          The problem and fix here was about a logged out (session expired) user spamming the log; you're asking for no log messages when a logged in user sends a crumb that's invalid for them. That is a different issue.

          Daniel Beck added a comment - Even on current versions of Jenkins, this should still happen for docwhat . The error message explains why: 2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403. This seems to be about a different user (joecool) having logged in since, or a crumb issuer that takes session information into account. IOW, it's not just an expired session, there's another valid session. The problem and fix here was about a logged out (session expired) user spamming the log; you're asking for no log messages when a logged in user sends a crumb that's invalid for them. That is a different issue.

          The Jenkins version is 2.89.4.

          I'll open a new ticket for my case.  Thanks!

          Christian Höltje added a comment - The Jenkins version is 2.89.4. I'll open a new ticket for my case.  Thanks!

            danielbeck Daniel Beck
            rtyler R. Tyler Croy
            Votes:
            4 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: