Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56243

Jenkins GUI is slow -removing cookie fixes it (temporarily)

    • Jenkins 2.184

      The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

      Situation:

      • In a clean browser (no cache, cookies) Jenkins is very fast
      • After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
        Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
        Jenkins just isn't workable for users at that time.
      • Logging out + in again does not fix it for that user.
      • Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again.

       So, what happens with the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE? 
      Why does it cause the slowness after hours of use?

       [update]

       SECURITY-901 / CVE-2019-1003004 in Jenkins 2.150.2 introduced a security fix, but with a side effect that after some time (hours) the Jenkins GUI for that user starts to slow down to a crawl.

          [JENKINS-56243] Jenkins GUI is slow -removing cookie fixes it (temporarily)

          Henjo van Rees created issue -
          Henjo van Rees made changes -
          Description Original: The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

          Situation:
           * In a clean browser (no cache, cookies) Jenkins is very fast
           * After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
          Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
          Jenkins just isn't workable for users at that time.
           * Logging out + in again does not fix it for that user.
           * _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

           

          So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
          Why does it cause the slowness after hours of use?
          New: The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

          Situation:
           * In a clean browser (no cache, cookies) Jenkins is very fast
           * After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
           Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
           Jenkins just isn't workable for users at that time.
           * Logging out + in again does not fix it for that user.
           * _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

           So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
           Why does it cause the slowness after hours of use?

          Ian Driver added a comment -

          We are also experiencing this problem on the latest LTS release (2.150.3)

          Ian Driver added a comment - We are also experiencing this problem on the latest LTS release (2.150.3)

          Linus Geson added a comment -

          Its also present in 2.150.2. We are experiencing it after upgrade from 2.73.3 to 2.150.2.

          Linus Geson added a comment - Its also present in 2.150.2. We are experiencing it after upgrade from 2.73.3 to 2.150.2.

          Shevek . added a comment -

          This appears to be affecting us after the update to 2.150.3 and has made jenkins unusable

          Shevek . added a comment - This appears to be affecting us after the update to 2.150.3 and has made jenkins unusable

          Shevek . added a comment -

          Downgrade to 2.150.1 appears to solve the issue.

          Shevek . added a comment - Downgrade to 2.150.1 appears to solve the issue.

          Shevek . added a comment -

          Tempted to say this is a major or blocker as it kills our usage of Jenkins on the first non-login request. We do NOT get an hour of usability, we get NO usability on 2.150.3

          Shevek . added a comment - Tempted to say this is a major or blocker as it kills our usage of Jenkins on the first non-login request. We do NOT get an hour of usability, we get NO usability on 2.150.3

          Henjo van Rees added a comment - - edited

          So, 2.150.1 doesn't seem to have the problem.
          2.150.2 and higher have the problem. 

          When I look at the 2.150.2 changelog I immediately see this fix:

          "Deleting a user in an external security realm did not invalidate their session or 'Remember me' cookie
          SECURITY-901 / CVE-2019-1003004
          When using an external security realm such as LDAP or Active Directory, deleting a user from the security realm does not result in the user losing access to Jenkins.

          While deleting the user record from Jenkins did invalidate the 'Remember me' cookie, there was no way to invalidate active sessions besides restarting Jenkins or terminating sessions through other means, such as Monitoring Plugin.

          Jenkins now encodes a per-user seed value in sessions, 'Remember me' cookies, and cached authentications of the remoting-based CLI, that can manually be reset by a user themselves, or an administrator, on the user’s configuration page. Doing so will invalidate all current sessions, 'Remember me' cookies, and cached CLI authentications, requiring credentials to be entered again to authenticate. Deleting a user record in Jenkins will now also invalidate existing sessions, as the current seed value is deleted as well."

           

          So, concluding: This security fix introduces are very nasty slowdown when using Remember Me and LDAP/AD.

          How can we escalate this issue further? 

          Henjo van Rees added a comment - - edited So, 2.150.1 doesn't seem to have the problem. 2.150.2 and higher have the problem.  When I look at the 2.150.2 changelog I immediately see this fix: "Deleting a user in an external security realm did not invalidate their session or 'Remember me' cookie SECURITY-901 / CVE-2019-1003004 When using an external security realm such as LDAP or Active Directory, deleting a user from the security realm does not result in the user losing access to Jenkins. While deleting the user record from Jenkins did invalidate the 'Remember me' cookie, there was no way to invalidate active sessions besides restarting Jenkins or terminating sessions through other means, such as Monitoring Plugin. Jenkins now encodes a per-user seed value in sessions, 'Remember me' cookies, and cached authentications of the remoting-based CLI, that can manually be reset by a user themselves, or an administrator, on the user’s configuration page. Doing so will invalidate all current sessions, 'Remember me' cookies, and cached CLI authentications, requiring credentials to be entered again to authenticate. Deleting a user record in Jenkins will now also invalidate existing sessions, as the current seed value is deleted as well."   So, concluding: This security fix introduces are very nasty slowdown when using Remember Me and LDAP/AD. How can we escalate this issue further? 
          Henjo van Rees made changes -
          Priority Original: Minor [ 4 ] New: Major [ 3 ]
          Henjo van Rees made changes -
          Description Original: The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

          Situation:
           * In a clean browser (no cache, cookies) Jenkins is very fast
           * After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
           Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
           Jenkins just isn't workable for users at that time.
           * Logging out + in again does not fix it for that user.
           * _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

           So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
           Why does it cause the slowness after hours of use?
          New: The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

          Situation:
           * In a clean browser (no cache, cookies) Jenkins is very fast
           * After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
           Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
           Jenkins just isn't workable for users at that time.
           * Logging out + in again does not fix it for that user.
           * _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

           So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
           Why does it cause the slowness after hours of use?

           

          [update]

           

            jvz Matt Sicker
            henjovr Henjo van Rees
            Votes:
            26 Vote for this issue
            Watchers:
            43 Start watching this issue

              Created:
              Updated:
              Resolved: