Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56243

Jenkins GUI is slow -removing cookie fixes it (temporarily)

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      Jenkins 2.184

      Description

      The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

      Situation:

      • In a clean browser (no cache, cookies) Jenkins is very fast
      • After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
        Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
        Jenkins just isn't workable for users at that time.
      • Logging out + in again does not fix it for that user.
      • Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again.

       So, what happens with the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE? 
      Why does it cause the slowness after hours of use?

       [update]

       SECURITY-901 / CVE-2019-1003004 in Jenkins 2.150.2 introduced a security fix, but with a side effect that after some time (hours) the Jenkins GUI for that user starts to slow down to a crawl.

        Attachments

          Issue Links

            Activity

            henjovr Henjo van Rees created issue -
            henjovr Henjo van Rees made changes -
            Field Original Value New Value
            Description The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

            Situation:
             * In a clean browser (no cache, cookies) Jenkins is very fast
             * After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
            Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
            Jenkins just isn't workable for users at that time.
             * Logging out + in again does not fix it for that user.
             * _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

             

            So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
            Why does it cause the slowness after hours of use?
            The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

            Situation:
             * In a clean browser (no cache, cookies) Jenkins is very fast
             * After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
             Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
             Jenkins just isn't workable for users at that time.
             * Logging out + in again does not fix it for that user.
             * _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

             So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
             Why does it cause the slowness after hours of use?
            Hide
            idriver Ian Driver added a comment -

            We are also experiencing this problem on the latest LTS release (2.150.3)

            Show
            idriver Ian Driver added a comment - We are also experiencing this problem on the latest LTS release (2.150.3)
            Hide
            hms_lig Linus Geson added a comment -

            Its also present in 2.150.2. We are experiencing it after upgrade from 2.73.3 to 2.150.2.

            Show
            hms_lig Linus Geson added a comment - Its also present in 2.150.2. We are experiencing it after upgrade from 2.73.3 to 2.150.2.
            Hide
            shevek Shevek . added a comment -

            This appears to be affecting us after the update to 2.150.3 and has made jenkins unusable

            Show
            shevek Shevek . added a comment - This appears to be affecting us after the update to 2.150.3 and has made jenkins unusable
            Hide
            shevek Shevek . added a comment -

            Downgrade to 2.150.1 appears to solve the issue.

            Show
            shevek Shevek . added a comment - Downgrade to 2.150.1 appears to solve the issue.
            Hide
            shevek Shevek . added a comment -

            Tempted to say this is a major or blocker as it kills our usage of Jenkins on the first non-login request. We do NOT get an hour of usability, we get NO usability on 2.150.3

            Show
            shevek Shevek . added a comment - Tempted to say this is a major or blocker as it kills our usage of Jenkins on the first non-login request. We do NOT get an hour of usability, we get NO usability on 2.150.3
            Hide
            henjovr Henjo van Rees added a comment - - edited

            So, 2.150.1 doesn't seem to have the problem.
            2.150.2 and higher have the problem. 

            When I look at the 2.150.2 changelog I immediately see this fix:

            "Deleting a user in an external security realm did not invalidate their session or 'Remember me' cookie
            SECURITY-901 / CVE-2019-1003004
            When using an external security realm such as LDAP or Active Directory, deleting a user from the security realm does not result in the user losing access to Jenkins.

            While deleting the user record from Jenkins did invalidate the 'Remember me' cookie, there was no way to invalidate active sessions besides restarting Jenkins or terminating sessions through other means, such as Monitoring Plugin.

            Jenkins now encodes a per-user seed value in sessions, 'Remember me' cookies, and cached authentications of the remoting-based CLI, that can manually be reset by a user themselves, or an administrator, on the user’s configuration page. Doing so will invalidate all current sessions, 'Remember me' cookies, and cached CLI authentications, requiring credentials to be entered again to authenticate. Deleting a user record in Jenkins will now also invalidate existing sessions, as the current seed value is deleted as well."

             

            So, concluding: This security fix introduces are very nasty slowdown when using Remember Me and LDAP/AD.

            How can we escalate this issue further? 

            Show
            henjovr Henjo van Rees added a comment - - edited So, 2.150.1 doesn't seem to have the problem. 2.150.2 and higher have the problem.  When I look at the 2.150.2 changelog I immediately see this fix: "Deleting a user in an external security realm did not invalidate their session or 'Remember me' cookie SECURITY-901 / CVE-2019-1003004 When using an external security realm such as LDAP or Active Directory, deleting a user from the security realm does not result in the user losing access to Jenkins. While deleting the user record from Jenkins did invalidate the 'Remember me' cookie, there was no way to invalidate active sessions besides restarting Jenkins or terminating sessions through other means, such as Monitoring Plugin. Jenkins now encodes a per-user seed value in sessions, 'Remember me' cookies, and cached authentications of the remoting-based CLI, that can manually be reset by a user themselves, or an administrator, on the user’s configuration page. Doing so will invalidate all current sessions, 'Remember me' cookies, and cached CLI authentications, requiring credentials to be entered again to authenticate. Deleting a user record in Jenkins will now also invalidate existing sessions, as the current seed value is deleted as well."   So, concluding: This security fix introduces are very nasty slowdown when using Remember Me and LDAP/AD. How can we escalate this issue further? 
            henjovr Henjo van Rees made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            henjovr Henjo van Rees made changes -
            Description The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

            Situation:
             * In a clean browser (no cache, cookies) Jenkins is very fast
             * After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
             Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
             Jenkins just isn't workable for users at that time.
             * Logging out + in again does not fix it for that user.
             * _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

             So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
             Why does it cause the slowness after hours of use?
            The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

            Situation:
             * In a clean browser (no cache, cookies) Jenkins is very fast
             * After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
             Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
             Jenkins just isn't workable for users at that time.
             * Logging out + in again does not fix it for that user.
             * _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

             So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
             Why does it cause the slowness after hours of use?

             

            [update]

             
            henjovr Henjo van Rees made changes -
            Description The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

            Situation:
             * In a clean browser (no cache, cookies) Jenkins is very fast
             * After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
             Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
             Jenkins just isn't workable for users at that time.
             * Logging out + in again does not fix it for that user.
             * _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

             So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
             Why does it cause the slowness after hours of use?

             

            [update]

             
            The last few week/months all our Jenkins users experience very a very slow web GUI after some time. 

            Situation:
             * In a clean browser (no cache, cookies) Jenkins is very fast
             * After some time (workday - 8 hours of active Jenkins use), Jenkins GUI starts to slow down:
             Loading jobs takes 10+ seconds, loading of static resources are very long pending etc.
             Jenkins just isn't workable for users at that time.
             * Logging out + in again does not fix it for that user.
             * _Removing the ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE cookie fixes everything for that user and makes Jenkins blazing fast again._

             So, what happens with the _ACEGI_SECURITY_HASHED_REMEMBER_ME_COOKIE?_ 
             Why does it cause the slowness after hours of use?

             _*[update]*_

             _*SECURITY-901 / CVE-2019-1003004 in Jenkins 2.150.2 introduced a security fix, but with a side effect that after some time (hours) the Jenkins GUI for that user starts to slow down to a crawl.*_
            Hide
            danielbeck Daniel Beck added a comment -

            Could we get thread dumps for this while Jenkins is busy? https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump

            Note that this might simply be a side effect of a bad performing security realm (AD server) that's getting hit more frequently.

            Show
            danielbeck Daniel Beck added a comment - Could we get thread dumps for this while Jenkins is busy? https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump Note that this might simply be a side effect of a bad performing security realm (AD server) that's getting hit more frequently.
            Hide
            danielbeck Daniel Beck added a comment -

            CC Wadeck Follonier just in case.

            Show
            danielbeck Daniel Beck added a comment - CC Wadeck Follonier just in case.
            Hide
            wfollonier Wadeck Follonier added a comment -

            Henjo van Rees, Shevek ., Linus Geson, Ian Driver, could you provide additional information about your configuration? Especially the security realm that is used, in which version (if from plugin), with specific configuration as well (like AD/LDAP cache configuration).

            In addition, when you are seeing such performance problem, could you check the cookies of one of your request? (especially the number of cookie that is sent)

            From my PoV there is no huge performance impact on the REMEMBER_ME cookie as the only addition there is a User.getById call, that is doing nothing with external security realm.

            Show
            wfollonier Wadeck Follonier added a comment - Henjo van Rees , Shevek . , Linus Geson , Ian Driver , could you provide additional information about your configuration? Especially the security realm that is used, in which version (if from plugin), with specific configuration as well (like AD/LDAP cache configuration). In addition, when you are seeing such performance problem, could you check the cookies of one of your request? (especially the number of cookie that is sent) From my PoV there is no huge performance impact on the REMEMBER_ME cookie as the only addition there is a User.getById call, that is doing nothing with external security realm.
            Hide
            hms_lig Linus Geson added a comment -

            We are using Active Directory security realm using Active Directory Plugin version 2.12. The only additional configuration enabled is "Enable StartTLS".

            I'll try to get back to you with the other information as well.

            Show
            hms_lig Linus Geson added a comment - We are using Active Directory security realm using Active Directory Plugin version 2.12. The only additional configuration enabled is "Enable StartTLS". I'll try to get back to you with the other information as well.
            Hide
            stuartrowe Stuart Rowe added a comment -

            We experienced this same issue after upgrading to LTS 2.150.2 also using Active Directory. The problem was resolved after enabling cache under the security realm configuration. We thought this setting had been configured already - perhaps it was disabled by the upgrade to LTS 2.150.2 or just misconfiguration on our end.

            Show
            stuartrowe Stuart Rowe added a comment - We experienced this same issue after upgrading to LTS 2.150.2 also using Active Directory. The problem was resolved after enabling cache under the security realm configuration. We thought this setting had been configured already - perhaps it was disabled by the upgrade to LTS 2.150.2 or just misconfiguration on our end.
            Hide
            wfollonier Wadeck Follonier added a comment -

            perhaps it was disabled by the upgrade to LTS 2.150.2

            Nothing done during the security around that, sorry

            Linus Geson Try enabling the cache feature of the plugin, that will help you a lot I imagine.

            Show
            wfollonier Wadeck Follonier added a comment - perhaps it was disabled by the upgrade to LTS 2.150.2 Nothing done during the security around that, sorry Linus Geson Try enabling the cache feature of the plugin, that will help you a lot I imagine.
            Hide
            shevek Shevek . added a comment -

            Wadeck Follonier We are using built in user authentication with the built in access matrix. Nothing special whatsoever. Stock install via apt.

             

            Show
            shevek Shevek . added a comment - Wadeck Follonier We are using built in user authentication with the built in access matrix. Nothing special whatsoever. Stock install via apt.  
            Hide
            danielbeck Daniel Beck added a comment -

            Could we get thread dumps for this while Jenkins is busy? https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump

            Show
            danielbeck Daniel Beck added a comment - Could we get thread dumps for this while Jenkins is busy? https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump
            hms_lig Linus Geson made changes -
            Attachment Thread dump [Jenkins].html [ 46514 ]
            Hide
            hms_lig Linus Geson added a comment -

            Thread dump [Jenkins].html

            Here comes a thread dump while the GUI is slow. I logged in yesterday when leaving work (checked keep me signed in) and shut down my computer. Arriving now this morning and booting up my laptop and opening the Jenkins web it is slow, not unusable but extremely slow.

            Show
            hms_lig Linus Geson added a comment - Thread dump [Jenkins].html Here comes a thread dump while the GUI is slow. I logged in yesterday when leaving work (checked keep me signed in) and shut down my computer. Arriving now this morning and booting up my laptop and opening the Jenkins web it is slow, not unusable but extremely slow.
            Hide
            danielbeck Daniel Beck added a comment -

            At the time the thread dump was recorded, that was the only request being handled (and it doesn't show problems). I suppose it's best in this case to take the thread dump another way (e.g. signal), while a slow request is being handled by Jenkins.

            Show
            danielbeck Daniel Beck added a comment - At the time the thread dump was recorded, that was the only request being handled (and it doesn't show problems). I suppose it's best in this case to take the thread dump another way (e.g. signal), while a slow request is being handled by Jenkins.
            hms_lig Linus Geson made changes -
            Attachment Thread dump2 [Jenkins].html [ 46516 ]
            Hide
            hms_lig Linus Geson added a comment -

            This thread dump is taken in one chrome tab while another tab is busy loading another Jenkins page. The threadDump tab finished while the other tab was still busy. I could try the signalling method as well if this dump is not good enough, I just hurried to get the dump before reading your comment entirely and realizing you asked for a different method

             

            Thread dump2 [Jenkins].html

            Show
            hms_lig Linus Geson added a comment - This thread dump is taken in one chrome tab while another tab is busy loading another Jenkins page. The threadDump tab finished while the other tab was still busy. I could try the signalling method as well if this dump is not good enough, I just hurried to get the dump before reading your comment entirely and realizing you asked for a different method   Thread dump2 [Jenkins].html
            Hide
            danielbeck Daniel Beck added a comment -

            Interesting.

            https://github.com/jenkinsci/jenkins/blob/1c9eb43283e7321ee4d3a0e1e9995453493ff04a/core/src/main/java/hudson/security/TokenBasedRememberMeServices2.java#L240 is new in the security fix.

            With a slow security realm, this will affect not just existing user lookups e.g. from changelogs ( GitChangeSet.findOrCreateUser shows up 4 times in the thread dump), but also any other request. Four separate requests go through TokenBasedRememberMeServices2.retrieveAuthFromCookie – caching means only one request to the security realm, but if that's slow, all of them are handled slowly.

            The thread dump also indicates AD goes through referrals, which slows everything down further. I wonder whether the security realm config is just terrible. Make sure you're using caching, if available, and that you're contacting the global catalog (which is all I remember from working in an AD environment).

            Show
            danielbeck Daniel Beck added a comment - Interesting. https://github.com/jenkinsci/jenkins/blob/1c9eb43283e7321ee4d3a0e1e9995453493ff04a/core/src/main/java/hudson/security/TokenBasedRememberMeServices2.java#L240 is new in the security fix. With a slow security realm, this will affect not just existing user lookups e.g. from changelogs (  GitChangeSet.findOrCreateUser shows up 4 times in the thread dump), but also any other request. Four separate requests go through TokenBasedRememberMeServices2.retrieveAuthFromCookie – caching means only one request to the security realm, but if that's slow, all of them are handled slowly. The thread dump also indicates AD goes through referrals, which slows everything down further. I wonder whether the security realm config is just terrible. Make sure you're using caching, if available, and that you're contacting the global catalog (which is all I remember from working in an AD environment).
            Hide
            hms_lig Linus Geson added a comment - - edited

            I don't know anything at all about AD but we do have some performance issues related to it that has affected our initial login to Jenkins (the regular log in takes 20 seconds) even before the upgrade that brought the current problems in this issue. Our IT department have some kind of plan for handling those issue down the road, but yes we do have a slow security realm.

            Thanks for the tips I'll look into it on my end....

            ...I enable the cache and the GUI went from slow to normal fast without having to delete any cookie.

            Show
            hms_lig Linus Geson added a comment - - edited I don't know anything at all about AD but we do have some performance issues related to it that has affected our initial login to Jenkins (the regular log in takes 20 seconds) even before the upgrade that brought the current problems in this issue. Our IT department have some kind of plan for handling those issue down the road, but yes we do have a slow security realm. Thanks for the tips I'll look into it on my end.... ...I enable the cache and the GUI went from slow to normal fast without having to delete any cookie.
            Hide
            wfollonier Wadeck Follonier added a comment -

            is new in the security fix.

            Not really, it was just "moved" from super class to that class, in order to have the time-independant equal method, nothing else changed in the method.

            Show
            wfollonier Wadeck Follonier added a comment - is new in the security fix. Not really, it was just "moved" from super class to that class, in order to have the time-independant equal method, nothing else changed in the method.
            Hide
            fseneque Frédéric Sénèque added a comment -

            Hi all,

            We are currently running Jenkins 2.164.1 LTS with the AD Plugin for authentication and we are facing the same issues. Enabling cache has improved a lot the response time of all HTTP requests (static resources/xhr ..) to jenkins. Is it considered as a work around or a mandatory configuration while using this AD plugin?

            Show
            fseneque Frédéric Sénèque added a comment - Hi all, We are currently running Jenkins 2.164.1 LTS with the AD Plugin for authentication and we are facing the same issues. Enabling cache has improved a lot the response time of all HTTP requests (static resources/xhr ..) to jenkins. Is it considered as a work around or a mandatory configuration while using this AD plugin?
            Hide
            djviking Sverre Moe added a comment - - edited

            I cannot find a 2.150.2, mentioned above but 2.150-1.2
            https://pkg.jenkins.io/opensuse/jenkins-2.150-1.2.noarch.rpm

            I Just tried to downgrade to 2.149 and it did not go so well, as several plugins do not work now.
            Our users are having serious trouble with slow Jenkins user experience. I hope this can be fixed soon.

            Show
            djviking Sverre Moe added a comment - - edited I cannot find a 2.150.2, mentioned above but 2.150-1.2 https://pkg.jenkins.io/opensuse/jenkins-2.150-1.2.noarch.rpm I Just tried to downgrade to 2.149 and it did not go so well, as several plugins do not work now. Our users are having serious trouble with slow Jenkins user experience. I hope this can be fixed soon.
            Show
            danielbeck Daniel Beck added a comment - https://pkg.jenkins.io/opensuse-stable/
            Hide
            djviking Sverre Moe added a comment - - edited

            I was unaware of the opensuse-stable builds. So this is the Jenkins LTS releases.
            I tried to downgrade to the 2.150.1 LTS and it worked with our installed plugins.

            The LTS upgrade guide seems to have a workaround for this bug
            https://jenkins.io/doc/upgrade-guide/2.150/

            Show
            djviking Sverre Moe added a comment - - edited I was unaware of the opensuse-stable builds. So this is the Jenkins LTS releases. I tried to downgrade to the 2.150.1 LTS and it worked with our installed plugins. The LTS upgrade guide seems to have a workaround for this bug https://jenkins.io/doc/upgrade-guide/2.150/
            Hide
            dpeschman Dan Peschman added a comment -

            Enabling AD caching worked for us too.

            Show
            dpeschman Dan Peschman added a comment - Enabling AD caching worked for us too.
            Hide
            danielbeck Daniel Beck added a comment -

            The LTS upgrade guide seems to have a workaround for this bug

            Different issue. You're not even logged in with that one.

             

            Show
            danielbeck Daniel Beck added a comment - The LTS upgrade guide seems to have a workaround for this bug Different issue. You're not even logged in with that one.  
            Hide
            djviking Sverre Moe added a comment - - edited

            So the workaround -Djenkins.security.seed.UserSeedProperty.disableUserSeed=true will not fix this issue?
            It says its a workaround for SECURITY-901 which mentioned in the description here caused the regression.

            Show
            djviking Sverre Moe added a comment - - edited So the workaround -Djenkins.security.seed.UserSeedProperty.disableUserSeed=true will not fix this issue? It says its a workaround for SECURITY-901 which mentioned in the description here caused the regression.
            Hide
            danielbeck Daniel Beck added a comment -

            Unsure, may or may not help.

            Show
            danielbeck Daniel Beck added a comment - Unsure, may or may not help.
            Hide
            djviking Sverre Moe added a comment - - edited

            Well if it doesn't, the downgrading to Jenkins LTS 2.150.1 worked on our Test Jenkins, so we have that to fall back to if it doesn't work.

            Show
            djviking Sverre Moe added a comment - - edited Well if it doesn't, the downgrading to Jenkins LTS 2.150.1 worked on our Test Jenkins, so we have that to fall back to if it doesn't work.
            Hide
            shevek Shevek . added a comment -

            The active-directory thing is a total red herring. We are affected by this, and we do not and never have used AD. The downgrade to 2.150.1 does work.

            Show
            shevek Shevek . added a comment - The active-directory thing is a total red herring. We are affected by this, and we do not and never have used AD. The downgrade to 2.150.1 does work.
            Hide
            gmauri Gianvittorio Mauri added a comment -

            We had the same issue, the workaround https://jenkins.io/doc/upgrade-guide/2.150/ worked for us.

            Show
            gmauri Gianvittorio Mauri added a comment - We had the same issue, the workaround https://jenkins.io/doc/upgrade-guide/2.150/ worked for us.
            Hide
            euphxenos Andrew Lawrence added a comment -

            I upgraded our jenkins masters from 2.150.1 to 2.164.2 a few days ago.  The next day, we were intermittently seeing long page loads.  Today, I have consistently seen pages take 15-20 seconds to load.  This is from an instance of chrome 71 on Linux; this browser has been open for a very long time (weeks, at least).  I did a service jenkins restart on one of the masters, and page loads became fast again from that same client browser, even after logging back into jenkins.  If I open a new browser (I opened an instance of Firefox), it sees fast page loads on the server that was just restarted, and on the servers where the jenkins process was not restarted, even after logging into jenkins.  If I turn on the AD cache on the jenkins masters, page loads become fast on all machines, even from the long-running client browser.  I am unconvinced that AD server performance was the root cause, since page loads are fast from a newly-opened web browser to a jenkins server with the AD cache still turned off.

            Show
            euphxenos Andrew Lawrence added a comment - I upgraded our jenkins masters from 2.150.1 to 2.164.2 a few days ago.  The next day, we were intermittently seeing long page loads.  Today, I have consistently seen pages take 15-20 seconds to load.  This is from an instance of chrome 71 on Linux; this browser has been open for a very long time (weeks, at least).  I did a service jenkins restart on one of the masters, and page loads became fast again from that same client browser, even after logging back into jenkins.  If I open a new browser (I opened an instance of Firefox), it sees fast page loads on the server that was just restarted, and on the servers where the jenkins process was not restarted, even after logging into jenkins.  If I turn on the AD cache on the jenkins masters, page loads become fast on all machines, even from the long-running client browser.  I am unconvinced that AD server performance was the root cause, since page loads are fast from a newly-opened web browser to a jenkins server with the AD cache still turned off.
            Hide
            elliot_nelson Elliot Nelson added a comment -

            So far, the only thing that has worked for us on Jenkins 2.150+ is disabling the Remember Me option in Global Security.

            Show
            elliot_nelson Elliot Nelson added a comment - So far, the only thing that has worked for us on Jenkins 2.150+ is disabling the Remember Me option in Global Security.
            Hide
            henjovr Henjo van Rees added a comment - - edited

            So, summary from the comments:

            • With AD post 2.150.1 become slow over time with Remember Me on
            • Without AD post 2.150.1 become slow over time with Remember Me on
            • Disabling Remember Me in Global Security fixes the problem, no slowdown after several hours
            • SECURITY-901 / CVE-2019-1003004 is de cause of the issue

             

            @Jenkins developers:

            Please look into the problem! Since people reported the same problem WITHOUT Active Directoy, please do not blame AD or directory services for the issue.

            Disabling Remember Me is a good workaround, but nothing more than a workaround!

             

            Show
            henjovr Henjo van Rees added a comment - - edited So, summary from the comments: With AD post 2.150.1 become slow over time with Remember Me on Without AD post 2.150.1 become slow over time with Remember Me on Disabling  Remember Me in Global Security fixes the problem, no slowdown after several hours SECURITY-901 / CVE-2019-1003004  is de cause of the issue   @Jenkins developers: Please look into the problem!  Since people reported the same problem WITHOUT Active Directoy, please do not blame AD or directory services for the issue. Disabling Remember Me is a good workaround, but nothing more than a workaround!  
            Hide
            jladan James Ladan added a comment - - edited

            Henjo van Rees - did you try enabling the cache in Jenkins' LDAP plugin config? That helped for us. See Stuart Rowe's comment, above: comment-363002

            Show
            jladan James Ladan added a comment - - edited Henjo van Rees - did you try enabling the cache in Jenkins' LDAP plugin config? That helped for us. See Stuart Rowe's comment, above: comment-363002
            Hide
            henjovr Henjo van Rees added a comment -

            The LDAP was already enabled before the upgrade, still enabled after the upgrade from 2.150.1.

            So no, that did not fix it unfortunately.

            Show
            henjovr Henjo van Rees added a comment - The LDAP was already enabled before the upgrade, still enabled after the upgrade from 2.150.1. So no, that did not fix it unfortunately.
            Hide
            djviking Sverre Moe added a comment -

            We have tried several workarounds. None of them works.

            • Disable security
            • LDAP (Logged-in users can do anything)
            • LDAP (Anyone can do anything)
            • Disable remember me
            • Downgrade to Jenkins LTS 2.150.1

            Our users still experience that Jenkins UI is slow and loading takes time, sometimes hang.

            Show
            djviking Sverre Moe added a comment - We have tried several workarounds. None of them works. Disable security LDAP (Logged-in users can do anything) LDAP (Anyone can do anything) Disable remember me Downgrade to Jenkins LTS 2.150.1 Our users still experience that Jenkins UI is slow and loading takes time, sometimes hang.
            Hide
            bshahadat Basma Shahadat added a comment -

            Is there any update on this? We are using 2.174, and it is painfully slow. Clearing cookies can help for short span of time.

            Show
            bshahadat Basma Shahadat added a comment - Is there any update on this? We are using 2.174, and it is painfully slow. Clearing cookies can help for short span of time.
            Hide
            ppgwittel Greg Wittel added a comment - - edited

            Looking at the code added by SECURITY-901, the code for UserSeedProperty concerns me:

             https://github.com/jenkinsci/jenkins/commit/8c490d14c4ffe6162f6e97d25a66612330fe2ace#diff-3ae1a1f58f660098ff61a9afeb8d3a10

            The RNG call RANDOM.generateSeed is effectively single threaded based on how SecureRandom works.  I'm not a Jenkins expert, but it looks like the HttpSessionContextIntegrationFilter2 or AuthenticationProcessingFilter2 can cause creation of a User object with properties (via AllUsers). The UserSeedProperty instance then gets generated via the User properties constructor (causing a new RNG call). If that gets called a lot, that can be a severe bottleneck.

            Similarly HudsonPrivateSecurityRealm also triggers the seed re-generation.

            A test may be to disable the user seed property (note this exposes the issue that SECURITY-901 tries to fix) per this link

            Set jenkins.security.seed.UserSeedProperty.disableUserSeed to true

            Show
            ppgwittel Greg Wittel added a comment - - edited Looking at the code added by SECURITY-901, the code for UserSeedProperty concerns me:   https://github.com/jenkinsci/jenkins/commit/8c490d14c4ffe6162f6e97d25a66612330fe2ace#diff-3ae1a1f58f660098ff61a9afeb8d3a10 The RNG call RANDOM.generateSeed is effectively single threaded based on how SecureRandom works.  I'm not a Jenkins expert, but it looks like the HttpSessionContextIntegrationFilter2 or AuthenticationProcessingFilter2 can cause creation of a User object with properties (via AllUsers). The UserSeedProperty instance then gets generated via the User properties constructor (causing a new RNG call). If that gets called a lot, that can be a severe bottleneck. Similarly HudsonPrivateSecurityRealm also triggers the seed re-generation. A test may be to disable the user seed property (note this exposes the issue that SECURITY-901 tries to fix) per this link Set jenkins.security.seed.UserSeedProperty.disableUserSeed to true
            Hide
            meghnapandey Meghna Pandey added a comment -

            Hi,

            I am using 2.176.1. And it is painfully slow. I changed the security settings to 'Jenkin's Own User Database'. Speed was good. Due to abrupt reboot of Jenkins yesterday, its not allowing any kind of authentication now. But can someone tell me the exact version which do not have GUI slowness issues please. I am trying to install Jenkins from fresh on a new box so I would rather install a version which has no known issues. 

            Show
            meghnapandey Meghna Pandey added a comment - Hi, I am using 2.176.1. And it is painfully slow. I changed the security settings to 'Jenkin's Own User Database'. Speed was good. Due to abrupt reboot of Jenkins yesterday, its not allowing any kind of authentication now. But can someone tell me the exact version which do not have GUI slowness issues please. I am trying to install Jenkins from fresh on a new box so I would rather install a version which has no known issues. 
            meghnapandey Meghna Pandey made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            Hide
            meghnapandey Meghna Pandey added a comment - - edited

            This issue is resolved after upgrading the Jenkins from 2.176 to 2.179. Jenkins is working perfectly fine now. 

            Show
            meghnapandey Meghna Pandey added a comment - - edited This issue is resolved after upgrading the Jenkins from 2.176 to 2.179. Jenkins is working perfectly fine now.  
            Hide
            djviking Sverre Moe added a comment - - edited

            I have been trying Jenkins 2.181 on our Jenkins Test server. Really fast loading of UI, not seeing any slowness.
            Though the Jenkins Test server does not have as much build history, the real test would be to upgrade our Jenkins Production server.

            Edit: Until I went to Manage Jenkins, then loading the icons just hanging page loading.

            Edit: Opening a multibranch project branch, the page seems to have loaded completely, but loading is still ongoing for some more ~30 seconds. Looked at what was loading and seems to be prototype.js. There was an fix for this in JENKINS-49319, but it was reverted.
            Firefox is having much more problems than browsers with the Chromium engine. So it seems to be a javascript problem that last loading.

            Edit:
            All good things must come to an end: Now I am getting the same problem in Chrome. Several resources are not loading.

            Show
            djviking Sverre Moe added a comment - - edited I have been trying Jenkins 2.181 on our Jenkins Test server. Really fast loading of UI, not seeing any slowness. Though the Jenkins Test server does not have as much build history, the real test would be to upgrade our Jenkins Production server. Edit: Until I went to Manage Jenkins, then loading the icons just hanging page loading. Edit: Opening a multibranch project branch, the page seems to have loaded completely, but loading is still ongoing for some more ~30 seconds. Looked at what was loading and seems to be prototype.js. There was an fix for this in JENKINS-49319 , but it was reverted. Firefox is having much more problems than browsers with the Chromium engine. So it seems to be a javascript problem that last loading. Edit: All good things must come to an end: Now I am getting the same problem in Chrome. Several resources are not loading.
            Hide
            djviking Sverre Moe added a comment -

            What I find very odd. If I log in to Jenkins (using LDAP) all loading problems goes away. Why should logging in solve the loading slowness?

            Show
            djviking Sverre Moe added a comment - What I find very odd. If I log in to Jenkins (using LDAP) all loading problems goes away. Why should logging in solve the loading slowness?
            Hide
            djviking Sverre Moe added a comment - - edited

            The last remaining loading on Multibranch Pipeline branch project. Developer Tools in my browser showed me what was remaining trying to load.

            ajax	200	xhr	prototype.js:1585	310 B	15 ms
            runs?since=%232&fullStages=true&_=1561103938220	200	xhr	jquery2.js:998	48.8 KB	18 ms
            

            These two repeated several times for about 30+ seconds until the page was actually finished loading.

            Show
            djviking Sverre Moe added a comment - - edited The last remaining loading on Multibranch Pipeline branch project. Developer Tools in my browser showed me what was remaining trying to load. ajax 200 xhr prototype.js:1585 310 B 15 ms runs?since=%232&fullStages= true &_=1561103938220 200 xhr jquery2.js:998 48.8 KB 18 ms These two repeated several times for about 30+ seconds until the page was actually finished loading.
            djviking Sverre Moe made changes -
            Labels user-experience
            Hide
            jvz Matt Sicker added a comment -

            If there's contention in the SecureRandom instance, that could be causing issues. Let me see if I can reproduce any slowdowns with a benchmark test.

            Thanks for verifying that this isn't AD-specific at least. I might be able to help figure this out.

            Show
            jvz Matt Sicker added a comment - If there's contention in the SecureRandom instance, that could be causing issues. Let me see if I can reproduce any slowdowns with a benchmark test. Thanks for verifying that this isn't AD-specific at least. I might be able to help figure this out.
            Hide
            jvz Matt Sicker added a comment -

            So far from my testing, I'm not finding any slow code in seed renewal. Some basic JMH tests in this branch: https://github.com/jenkinsci/jenkins/compare/master...jvz:user-seed-perf-JENKINS-56243?expand=1

            Right now, my hypothesis is that if a SecurityRealm is having any performance issues, multiple requests to load the same user's details could be piling up due to the remember me cookie validation check. The same happens in the session cookie itself. Basically, the reason why it was performing better before was because it wasn't validating authentication properly in the first place.

            I'm working on some basic load tests to compare 2.150.1 and 2.150.2 to see if I can reproduce this idea. Based on the comments so far, it sounds like this should even be potentially reproducible using just the built-in user database. The JMH tests above only use an in-memory user database, so introducing lag in the calls to loadUserDetails() could be an interesting way to potentially test this as well.

            Show
            jvz Matt Sicker added a comment - So far from my testing, I'm not finding any slow code in seed renewal. Some basic JMH tests in this branch: https://github.com/jenkinsci/jenkins/compare/master...jvz:user-seed-perf-JENKINS-56243?expand=1 Right now, my hypothesis is that if a SecurityRealm is having any performance issues, multiple requests to load the same user's details could be piling up due to the remember me cookie validation check. The same happens in the session cookie itself. Basically, the reason why it was performing better before was because it wasn't validating authentication properly in the first place. I'm working on some basic load tests to compare 2.150.1 and 2.150.2 to see if I can reproduce this idea. Based on the comments so far, it sounds like this should even be potentially reproducible using just the built-in user database. The JMH tests above only use an in-memory user database, so introducing lag in the calls to loadUserDetails() could be an interesting way to potentially test this as well.
            Hide
            jvz Matt Sicker added a comment -

            I discovered that the remember me service bypasses the user details cache entirely. I've made a draft PR with this fixed: https://github.com/jenkinsci/jenkins/pull/4093

            Show
            jvz Matt Sicker added a comment - I discovered that the remember me service bypasses the user details cache entirely. I've made a draft PR with this fixed: https://github.com/jenkinsci/jenkins/pull/4093
            Hide
            djviking Sverre Moe added a comment -

            We have disabled the Remember Me option and still experience the slowness. The slowness only happens when users are not logged in.

            Show
            djviking Sverre Moe added a comment - We have disabled the Remember Me option and still experience the slowness. The slowness only happens when users are not logged in.
            Hide
            jvz Matt Sicker added a comment -

            I chatted with Wadeck Follonier earlier today, and we've found that the most likely culprit is that TokenBasedRememberMeServices2 does not cache the user seed property in their session. I'll submit a PR later to address this.

            Show
            jvz Matt Sicker added a comment - I chatted with Wadeck Follonier earlier today, and we've found that the most likely culprit is that TokenBasedRememberMeServices2 does not cache the user seed property in their session. I'll submit a PR later to address this.
            jvz Matt Sicker made changes -
            Assignee Matt Sicker [ jvz ]
            jvz Matt Sicker made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jvz Matt Sicker made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            jvz Matt Sicker added a comment -

            Added link to PR.

            Show
            jvz Matt Sicker added a comment - Added link to PR.
            jvz Matt Sicker made changes -
            Remote Link This issue links to "PR-4096 (Web Link)" [ 23141 ]
            jvz Matt Sicker made changes -
            Status In Review [ 10005 ] In Progress [ 3 ]
            Hide
            jvz Matt Sicker added a comment -

            Need to add another test, but this looks to be about fixed.

            Show
            jvz Matt Sicker added a comment - Need to add another test, but this looks to be about fixed.
            jvz Matt Sicker made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            djviking Sverre Moe added a comment -

            Looking forward to testing it out. Our developers are getting frustrated.

            Show
            djviking Sverre Moe added a comment - Looking forward to testing it out. Our developers are getting frustrated.
            Hide
            jvz Matt Sicker added a comment -

            Incremental release available: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/cli/2.184-rc28433.92d6063c40c3/

            Still waiting for reviews before someone can merge it for the next weekly.

            Show
            jvz Matt Sicker added a comment - Incremental release available: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/cli/2.184-rc28433.92d6063c40c3/ Still waiting for reviews before someone can merge it for the next weekly.
            Hide
            djviking Sverre Moe added a comment -

            I can test the incremental release on our Test Jenkins instance. I dare not install it in production.

            Show
            djviking Sverre Moe added a comment - I can test the incremental release on our Test Jenkins instance. I dare not install it in production.
            Hide
            wfollonier Wadeck Follonier added a comment -

            Sverre Moe as you said before, if the "Disable remember me" workaround was not working for you, do not expect this change to work either. It's "just" the correction of the root cause of this issue. From my PoV, wiht all the information you gave, you have another (unknown?) problem that is different from this one.

            Show
            wfollonier Wadeck Follonier added a comment - Sverre Moe as you said before, if the "Disable remember me" workaround was not working for you, do not expect this change to work either. It's "just" the correction of the root cause of this issue. From my PoV, wiht all the information you gave, you have another (unknown?) problem that is different from this one.
            Hide
            djviking Sverre Moe added a comment -

            There is another issue I have been tracking I think can be related to our problem of slowness. JENKINS-49319

            Show
            djviking Sverre Moe added a comment - There is another issue I have been tracking I think can be related to our problem of slowness. JENKINS-49319
            danielbeck Daniel Beck made changes -
            Labels user-experience lts-candidate user-experience
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The fix was released in Jenkins 2.184

            Show
            oleg_nenashev Oleg Nenashev added a comment - The fix was released in Jenkins 2.184
            oleg_nenashev Oleg Nenashev made changes -
            Released As Jenkins 2.184
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            Hide
            amidar Amit Dar added a comment -

            will this be included in the next LTS release?

            Show
            amidar Amit Dar added a comment - will this be included in the next LTS release?
            Hide
            danielbeck Daniel Beck added a comment -

            Next baseline for sure. 2.176.2 certainly not. 2.176.3 possibly.

            Show
            danielbeck Daniel Beck added a comment - Next baseline for sure. 2.176.2 certainly not. 2.176.3 possibly.
            olivergondza Oliver Gondža made changes -
            Labels lts-candidate user-experience 2.176.3-fixed user-experience

              People

              Assignee:
              jvz Matt Sicker
              Reporter:
              henjovr Henjo van Rees
              Votes:
              26 Vote for this issue
              Watchers:
              43 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: