-
Bug
-
Resolution: Unresolved
-
Major
-
Ubuntu 14.04 x86_64
Jenkins 2.19.2 LTS
Open JDK 1.7
Active Choices Plug-in 1.5
Folders Plugin 5.13
-
Powered by SuggestiMate
On the same Jenkins server,
I see significant performance slowness in release 1.5.
I have about 2.5K jobs in total within my Jenkins master,
and I have about 5~6 active-reactive input parameters per job.
With release 1.4,
I could fully open the "Build with parameter" page within about 4~6 seconds.
With release 1.5,
the loading time becomes about 20 seconds (approx 4 times slower).
- uno-choice-20180527.hpi
- 686 kB
- ac 1.5.4.jpg
- 32 kB
- uno-choice-1.5.4-0001.hpi
- 1.40 MB
- Promotion Status.png
- 66 kB
- configure.png
- 25 kB
- click-configure.png
- 12 kB
[JENKINS-39593] Performance slow in 1.5 release
Hi totoroliu
Nice job investigating it. It sounds like we need to improve the way that that jenkinsProject variable is injected. There's a bunch of ways of fixing it (disabling that unless the user request that variable, reverting the change if there's no feature broken, find another way of doing that, etc).
Next release should be out soon (by the end of the month I believe).
Thanks
Bruno
I got same similar problem. but seem not only in version 1.5.
I did some testing the found the groovy script which configurate in the Jenkins job will run about 15 times when click Jenkins job name in Jenkins. Then can open the Jenkins job main page.
it is need run about 15 times too when I click "Build with Parameters". also click configuration button.
every click need take about 50 seconds then can open page.
I have try to use version 1.3, 1.4. All like this.
Jenkins version is LTS 1.651.3. java version is 1.7
I have been using the plugin recently to develop a job that allows users dynamic control of the build parameters. While I love the options and control from the plugin, I noticed also a performance issue. When I run Jenkins on my local VM with simple Jenkins installation, the job build parameters respond fast without delay. However when I checkin my code to the server and I build the job from our remote server then the response time is terrible when I click on the various drop-downs, radio-buttons, etc. It updates the dynamic values very slowly.
I see here that the performance is an issue, but I am puzzled to see it work fast on my local VM with Jenkins. So is the issue happening when it calls the Groovy Script for that parameter? Can you explain the problem better for me and let me know if a fix is coming soon? The user experience is a bit annoying.
We are using the latest Jenkins LTS with the plugin 1.5 version.
Thanks!
I found if I use "promoted builds plugin" and " Active Choices Plug-in" in same Jenkins job can reproduce my problem.
Then the "Active Choices Plug-in" will be call many times. it will be call when
click this Jenkins job name,
click configure of this Jenkins job,
click save button in job configure page,
click Build with Parameters of this Jenkins job
click Promotion Status of this Jekins job
click promotion approve of build
If not use "promoted builds plugin" in this Jenkins job.
The "Active Choices Plug-in" only call 2 or 3 times. it happen in when
Click Build with Parameters of this Jenkins job only
These test run in Jenkins ver. 2.19.3 LTS . java1.8, 1.7 also. redhat6.7
Hi totoroliu, I'm on holidays overseas until the first week of February, with limited bandwidth. Will try to - at least - re-read the issue to remember what it was about, and check if it was fixed, or how is it going to happen. Feel free to bump the issue if I take too long to reply.
I am also interested in this one being done. I have one active choices parameter to which refer 10 active choices reactive reference parameters of formatted html type and one checkbox active choice reactive param. The performance might be better. Jenkins hangs up at each active parameter change for couple of seconds. Plus first rendering is literally like one second per each parameter one by one is appearing on a page.
Acknowledged. Let's aim at getting this fixed for 1.5.4. Plus, it sounds like fun to debug and fix this kind of problems
1.5.3 released in a few hours, available on Monday UTC. Working on 1.5.4 right after that.
Creating 1000 projects:
import hudson.model.* def N = 1000 def sourceProject = 'JENKINS-39593' for(i in (1..N)) { def newName = 'DUMMY-PROJECT-' + i def source = Hudson.instance.getJob(sourceProject) def job = Hudson.instance.copy(source, newName) job.save() println(" $sourceProject copied as $newName") }
lsbwant I was not able to reproduce the behaviour you described. With or without promoted builds plugin (not the simple version), my script gets executed just once.
My sample job now has active-choices-plugin + promoted-builds-plugin together. Going to the promotion status triggers once. Going to build with parameters too.
Created a logger (org.biouno, log level ALL), so far the output looks consistent, executing the parameter only. Now will add more parameters (reactive now, before was using a choice parameter)
I am not using promoted-builds-plugin. To reproduce lag on form loading the only thing you need to have is active choices parameter (let's say a single select combobox) and let's say 10 active choices reactive reference parameters (formatted html type) pointing to active choices parameter plugin. Then you should definitely notice a lag during form loading.
That's interesting. My normal param was called once, whereas my reactive param was called three times. Removing the promoted builds plugin, the behaviour persists.
Code for PARAM001:
println("PARAM001") return [0,1,2,3,4]
PARAM002:
println("PARAM002") return ["A", PARAM001, "C", PARAM001 + "111"]
Output of going to :
PARAM001 PARAM002 PARAM002 fallback... PARAM002
Identified some performance problems. Most were already pointed by users, so these were just confirmed. Used JENKINS-42824 example script from jzz0000, combined with the script I attached earlier. So I have 1 seed job, that creates another job that has 40 fields (1 choice param, and 39 reactive cascade params), and have also created other 2000 jobs with my script. So total of 2002 jobs.
carek, jzz0000, ioannis, I believe I have a stable version in my computer now, that has the following improvements:
- A local LRU cache, with 10 items. This cache is a global cache, in memory, of 10 projects. The key is the projectName, and the value is the Project item (same project that becomes jenkinsProject in Groovy context).
- Before, *for each parameter on the screen, we would iterate all projects. Then iterate all of its parameters, trying to match with the current index parameter*. Now the behaviour is still similar, but instead of iterating projects and finding a match project, we simply look at the cache.
- The projectName was not being persisted correct sometimes. And because of that the plug-in would look for a project using the parameter. Now this is the fallback behaviour. The normal workflow will be to find the project name when the job is saved.
- *For each cascade parameter, it could be executed three times instead of a single time*. The first execution, would be in Jelly, when the HTML elements were created. We would evaluate the script to find all the elements to populate the HTML. Immediately after that, still in the Jelly file, we would call the JavaScript method to update the parameter. This method would, on its turn, call the Java proxy methods to retrieve all elements to populate the HTML. Finally, cascade parameter events could trigger a third execution. Now the HTML elements of Cascade and Dynamic parameters are created, but not populated. And then we call the JavaScript method that will use the proxy.
- Each Jelly file would start by clearing all the parameters in the parameter. A Jenkins job parameter had several "parameters" that are passed to the script (e.g. referenced parameters, jenkinsProject, etc). As the parameters are initially created with the default values, there is no need to clear them in the first run.
I'm also doing small changes like using more String constants. This issue is epic, and has the following risks:
- Cases I'm not covering during my development & testing may stop working
- The Visible Item Count logic was linked to populating the HTML elements, rather than in the JavaScript. Which means that now that we are not populating the HTML elements, and instead using the JavaScript method, the logic to calculate the visible item count won't find any elements. In other words, the visibleItemCount will always be 1, then will fallback to the default minimum visible count
I used Chrome developer tools, Yourkit profiler, VisualVM, and DropWizards Metrics to profile the plug-in. Will attach two screenshots, with the current benchmark from DropWizards Metrics JMX Bean. Pay attention to 9xth percentiles and stddev, as these are normally more useful for performance troubleshooting.
Looking at Yourkit, memory-wise not much changed. The performance problems with the plug-in are mainly due to the number of operations we are performing with the objects we have in memory.
I did not record the whole initial session in chrome due to the buffer size, as this would take several seconds (1+ min some times). But with the change, I just started Chrome, loaded the page looking at Timeline, and the whole recording/buffering took 2.10 seconds
ioannis I was planning to include much more in 1.5.4, but now I'm a bit reluctant to ship a release with many changes, and plus this epic performance issue. It will take me another week to finalize the work, push the changes, and release an experimental version.
carek, jzz0000, ioannis, it would be extremely helpful for me to get feedback while I change things. So I will upload a pre experimental hpi file to this ticket as soon as I have finished a few more changes. In case any of you guys could watch the issue, and then grab the hpi, experiment with it in a dev/testbed server, and report back here, *would be of great help*!
Cheers
Bruno
Attaching the first plug-in binary with the performance fixes.
It contains:
- Cache
- Fixed logic for populating & updating parameters
- Using more String constants (though memory is not really a problem for us)
- *Deprecated code was removed*
Maintaining the deprecated code (which is necessary sometimes for backward compatibility) is quite hard, as it increases the maintainability. We haven't deprecated any code in a while, so I believe it's safe to assume users are already using an updated version like 1.5.x, and not 1.4 any more.
So I'm intending to release 1.9 now, or perhaps a 2.0 already. Anyway, we can discuss the deprecated removal in another thread. For now, focus in the performance issues, and let me know what's broken now, please
ps: I skipped unit tests as there are few things that seem to be broken. The following tests are failing right now:
Results : Failed tests: org.biouno.unochoice.TestParametersOrder.testParametersOrder(org.biouno.unochoice.TestParametersOrder) Run 1: TestParametersOrder.testParametersOrder:70 array lengths differed, expected.length=4 actual.length=0 Run 2: TestParametersOrder.testParametersOrder:70 array lengths differed, expected.length=4 actual.length=0 Run 3: TestParametersOrder.testParametersOrder:70 array lengths differed, expected.length=4 actual.length=0 Run 4: TestParametersOrder.testParametersOrder:70 array lengths differed, expected.length=4 actual.length=0 Run 5: TestParametersOrder.testParametersOrder:70 array lengths differed, expected.length=4 actual.length=0 org.biouno.unochoice.issue34818.TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties(org.biouno.unochoice.issue34818.TestGlobalNodePropertiesScript) Run 1: TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties:79 expected:<[[a, b, 20:13:13]]> but was:<[[]]> Run 2: TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties:79 expected:<[[a, b, 20:13:13]]> but was:<[[]]> Run 3: TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties:79 expected:<[[a, b, 20:13:13]]> but was:<[[]]> Run 4: TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties:79 expected:<[[a, b, 20:13:13]]> but was:<[[]]> Run 5: TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties:79 expected:<[[a, b, 20:13:13]]> but was:<[[]]> org.biouno.unochoice.issue38532.TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues(org.biouno.unochoice.issue38532.TestParameterValuesWithEquals) Run 1: TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues:89 Value returned from selection list doesn't match first element in list parameter expected:<[A=1]> but was:<[{}]> Run 2: TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues:89 Value returned from selection list doesn't match first element in list parameter expected:<[A=1]> but was:<[{}]> Run 3: TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues:89 Value returned from selection list doesn't match first element in list parameter expected:<[A=1]> but was:<[{}]> Run 4: TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues:89 Value returned from selection list doesn't match first element in list parameter expected:<[A=1]> but was:<[{}]> Run 5: TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues:89 Value returned from selection list doesn't match first element in list parameter expected:<[A=1]> but was:<[{}]>
Will work during the next week on fixing these issues + visible item count.
Thanks Bruno~
I'll try to setup a test Jenkins instance for 1.5.4.0001.hpi next week to test
One suggestion regarding to the LRU cache with 10 items.
In the final release,
is it possible to make it configurable through WebUI?
Thank you totoroliu. That's definitely possible. However, we would have to create a global configuration page. As the cache is not per job nor per parameter, but rather a global cache.
I had this idea after reading Scriptler plug-in code. Scriptler has also a 10 LRU cache for eval'ed scripts: https://github.com/jenkinsci/scriptler-plugin/blob/a6cd8c16198d723f19014ebf2f40ec0bb8fb5865/src/main/java/org/jenkinsci/plugins/scriptler/util/GroovyScript.java#L37
Definitely doable, but we may skip this configuration option in the next release, and maybe include it in the subsequent release.
I have just installed attached plugin and there are two things worth noticing:
- Lots of: MissingFieldException: No field 'script' found in class 'org.biouno.unochoice.model.GroovyScript', MissingFieldException: No field 'fallbackScript' found in class 'org.biouno.unochoice.model.GroovyScript' messages in warnings page
- Job's execution form have blank parameters (even if I resave job). I am attaching screen.
Code changed in jenkins
User: Bruno P. Kinoshita
Path:
pom.xml
src/main/java/org/biouno/unochoice/AbstractCascadableParameter.java
src/main/java/org/biouno/unochoice/AbstractScriptableParameter.java
src/main/java/org/biouno/unochoice/AbstractUnoChoiceParameter.java
src/main/java/org/biouno/unochoice/CascadeChoiceParameter.java
src/main/java/org/biouno/unochoice/ChoiceParameter.java
src/main/java/org/biouno/unochoice/DynamicReferenceParameter.java
src/main/java/org/biouno/unochoice/model/GroovyScript.java
src/main/java/org/biouno/unochoice/util/Utils.java
http://jenkins-ci.org/commit/active-choices-plugin/2678199aaaac1822687aab11fabe0805bbd9124f
Log:
JENKINS-39593: use a LRU cache, remove deprecated code, use String constants whenever possible, add lang3
*NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/
Functionality will be removed from GitHub.com on January 31st, 2019.
Thanks for testing it carek!
I suspect we had these issues in 1.5.x releases too, and - hopefully - they were all fixed in 2.x. I've re-applied the changes from the old branch, onto the new code base, and created the branch https://github.com/jenkinsci/active-choices-plugin/tree/JENKINS-39593-2
Testing, I couldn't notice much difference. But maybe someone with a test server and a few more example projects could test it too? If anyone has some spare time to test it, here's a new version attach (uno-choice-20180527.hpi).
Thanks!
Bruno
I'm not sure if this is related to the fix in
JENKINS-36590(commit-6a4828b08b24034fd317bf7ad0c9f5db660087c7).I hope it's not each of active-reactive input parameters would have to go through instance.getAllItems().
None of my six active-reactive input parameters are referencing "jenkinsProject" variable name.