-
Bug
-
Resolution: Unresolved
-
Major
-
Ubuntu 14.04 x86_64
Jenkins 2.19.2 LTS
Open JDK 1.7
Active Choices Plug-in 1.5
Folders Plugin 5.13
-
Powered by SuggestiMate
On the same Jenkins server,
I see significant performance slowness in release 1.5.
I have about 2.5K jobs in total within my Jenkins master,
and I have about 5~6 active-reactive input parameters per job.
With release 1.4,
I could fully open the "Build with parameter" page within about 4~6 seconds.
With release 1.5,
the loading time becomes about 20 seconds (approx 4 times slower).
- ac 1.5.4.jpg
- 32 kB
- click-configure.png
- 12 kB
- configure.png
- 25 kB
- Promotion Status.png
- 66 kB
- uno-choice-1.5.4-0001.hpi
- 1.40 MB
- uno-choice-20180527.hpi
- 686 kB
[JENKINS-39593] Performance slow in 1.5 release
I am not using promoted-builds-plugin. To reproduce lag on form loading the only thing you need to have is active choices parameter (let's say a single select combobox) and let's say 10 active choices reactive reference parameters (formatted html type) pointing to active choices parameter plugin. Then you should definitely notice a lag during form loading.
That's interesting. My normal param was called once, whereas my reactive param was called three times. Removing the promoted builds plugin, the behaviour persists.
Code for PARAM001:
println("PARAM001") return [0,1,2,3,4]
PARAM002:
println("PARAM002") return ["A", PARAM001, "C", PARAM001 + "111"]
Output of going to :
PARAM001 PARAM002 PARAM002 fallback... PARAM002
Identified some performance problems. Most were already pointed by users, so these were just confirmed. Used JENKINS-42824 example script from jzz0000, combined with the script I attached earlier. So I have 1 seed job, that creates another job that has 40 fields (1 choice param, and 39 reactive cascade params), and have also created other 2000 jobs with my script. So total of 2002 jobs.
carek, jzz0000, ioannis, I believe I have a stable version in my computer now, that has the following improvements:
- A local LRU cache, with 10 items. This cache is a global cache, in memory, of 10 projects. The key is the projectName, and the value is the Project item (same project that becomes jenkinsProject in Groovy context).
- Before, *for each parameter on the screen, we would iterate all projects. Then iterate all of its parameters, trying to match with the current index parameter*. Now the behaviour is still similar, but instead of iterating projects and finding a match project, we simply look at the cache.
- The projectName was not being persisted correct sometimes. And because of that the plug-in would look for a project using the parameter. Now this is the fallback behaviour. The normal workflow will be to find the project name when the job is saved.
- *For each cascade parameter, it could be executed three times instead of a single time*. The first execution, would be in Jelly, when the HTML elements were created. We would evaluate the script to find all the elements to populate the HTML. Immediately after that, still in the Jelly file, we would call the JavaScript method to update the parameter. This method would, on its turn, call the Java proxy methods to retrieve all elements to populate the HTML. Finally, cascade parameter events could trigger a third execution. Now the HTML elements of Cascade and Dynamic parameters are created, but not populated. And then we call the JavaScript method that will use the proxy.
- Each Jelly file would start by clearing all the parameters in the parameter. A Jenkins job parameter had several "parameters" that are passed to the script (e.g. referenced parameters, jenkinsProject, etc). As the parameters are initially created with the default values, there is no need to clear them in the first run.
I'm also doing small changes like using more String constants. This issue is epic, and has the following risks:
- Cases I'm not covering during my development & testing may stop working
- The Visible Item Count logic was linked to populating the HTML elements, rather than in the JavaScript. Which means that now that we are not populating the HTML elements, and instead using the JavaScript method, the logic to calculate the visible item count won't find any elements. In other words, the visibleItemCount will always be 1, then will fallback to the default minimum visible count
I used Chrome developer tools, Yourkit profiler, VisualVM, and DropWizards Metrics to profile the plug-in. Will attach two screenshots, with the current benchmark from DropWizards Metrics JMX Bean. Pay attention to 9xth percentiles and stddev, as these are normally more useful for performance troubleshooting.
Looking at Yourkit, memory-wise not much changed. The performance problems with the plug-in are mainly due to the number of operations we are performing with the objects we have in memory.
I did not record the whole initial session in chrome due to the buffer size, as this would take several seconds (1+ min some times). But with the change, I just started Chrome, loaded the page looking at Timeline, and the whole recording/buffering took 2.10 seconds
ioannis I was planning to include much more in 1.5.4, but now I'm a bit reluctant to ship a release with many changes, and plus this epic performance issue. It will take me another week to finalize the work, push the changes, and release an experimental version.
carek, jzz0000, ioannis, it would be extremely helpful for me to get feedback while I change things. So I will upload a pre experimental hpi file to this ticket as soon as I have finished a few more changes. In case any of you guys could watch the issue, and then grab the hpi, experiment with it in a dev/testbed server, and report back here, *would be of great help*!
Cheers
Bruno
Attaching the first plug-in binary with the performance fixes.
It contains:
- Cache
- Fixed logic for populating & updating parameters
- Using more String constants (though memory is not really a problem for us)
- *Deprecated code was removed*
Maintaining the deprecated code (which is necessary sometimes for backward compatibility) is quite hard, as it increases the maintainability. We haven't deprecated any code in a while, so I believe it's safe to assume users are already using an updated version like 1.5.x, and not 1.4 any more.
So I'm intending to release 1.9 now, or perhaps a 2.0 already. Anyway, we can discuss the deprecated removal in another thread. For now, focus in the performance issues, and let me know what's broken now, please
ps: I skipped unit tests as there are few things that seem to be broken. The following tests are failing right now:
Results : Failed tests: org.biouno.unochoice.TestParametersOrder.testParametersOrder(org.biouno.unochoice.TestParametersOrder) Run 1: TestParametersOrder.testParametersOrder:70 array lengths differed, expected.length=4 actual.length=0 Run 2: TestParametersOrder.testParametersOrder:70 array lengths differed, expected.length=4 actual.length=0 Run 3: TestParametersOrder.testParametersOrder:70 array lengths differed, expected.length=4 actual.length=0 Run 4: TestParametersOrder.testParametersOrder:70 array lengths differed, expected.length=4 actual.length=0 Run 5: TestParametersOrder.testParametersOrder:70 array lengths differed, expected.length=4 actual.length=0 org.biouno.unochoice.issue34818.TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties(org.biouno.unochoice.issue34818.TestGlobalNodePropertiesScript) Run 1: TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties:79 expected:<[[a, b, 20:13:13]]> but was:<[[]]> Run 2: TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties:79 expected:<[[a, b, 20:13:13]]> but was:<[[]]> Run 3: TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties:79 expected:<[[a, b, 20:13:13]]> but was:<[[]]> Run 4: TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties:79 expected:<[[a, b, 20:13:13]]> but was:<[[]]> Run 5: TestGlobalNodePropertiesScript.testScriptAccessingGlobalProperties:79 expected:<[[a, b, 20:13:13]]> but was:<[[]]> org.biouno.unochoice.issue38532.TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues(org.biouno.unochoice.issue38532.TestParameterValuesWithEquals) Run 1: TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues:89 Value returned from selection list doesn't match first element in list parameter expected:<[A=1]> but was:<[{}]> Run 2: TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues:89 Value returned from selection list doesn't match first element in list parameter expected:<[A=1]> but was:<[{}]> Run 3: TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues:89 Value returned from selection list doesn't match first element in list parameter expected:<[A=1]> but was:<[{}]> Run 4: TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues:89 Value returned from selection list doesn't match first element in list parameter expected:<[A=1]> but was:<[{}]> Run 5: TestParameterValuesWithEquals.testEvaluationWorksEvenThoughWeUsedEqualsInParameterValues:89 Value returned from selection list doesn't match first element in list parameter expected:<[A=1]> but was:<[{}]>
Will work during the next week on fixing these issues + visible item count.
Thanks Bruno~
I'll try to setup a test Jenkins instance for 1.5.4.0001.hpi next week to test
One suggestion regarding to the LRU cache with 10 items.
In the final release,
is it possible to make it configurable through WebUI?
Thank you totoroliu. That's definitely possible. However, we would have to create a global configuration page. As the cache is not per job nor per parameter, but rather a global cache.
I had this idea after reading Scriptler plug-in code. Scriptler has also a 10 LRU cache for eval'ed scripts: https://github.com/jenkinsci/scriptler-plugin/blob/a6cd8c16198d723f19014ebf2f40ec0bb8fb5865/src/main/java/org/jenkinsci/plugins/scriptler/util/GroovyScript.java#L37
Definitely doable, but we may skip this configuration option in the next release, and maybe include it in the subsequent release.
I have just installed attached plugin and there are two things worth noticing:
- Lots of: MissingFieldException: No field 'script' found in class 'org.biouno.unochoice.model.GroovyScript', MissingFieldException: No field 'fallbackScript' found in class 'org.biouno.unochoice.model.GroovyScript' messages in warnings page
- Job's execution form have blank parameters (even if I resave job). I am attaching screen.
Code changed in jenkins
User: Bruno P. Kinoshita
Path:
pom.xml
src/main/java/org/biouno/unochoice/AbstractCascadableParameter.java
src/main/java/org/biouno/unochoice/AbstractScriptableParameter.java
src/main/java/org/biouno/unochoice/AbstractUnoChoiceParameter.java
src/main/java/org/biouno/unochoice/CascadeChoiceParameter.java
src/main/java/org/biouno/unochoice/ChoiceParameter.java
src/main/java/org/biouno/unochoice/DynamicReferenceParameter.java
src/main/java/org/biouno/unochoice/model/GroovyScript.java
src/main/java/org/biouno/unochoice/util/Utils.java
http://jenkins-ci.org/commit/active-choices-plugin/2678199aaaac1822687aab11fabe0805bbd9124f
Log:
JENKINS-39593: use a LRU cache, remove deprecated code, use String constants whenever possible, add lang3
*NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/
Functionality will be removed from GitHub.com on January 31st, 2019.
Thanks for testing it carek!
I suspect we had these issues in 1.5.x releases too, and - hopefully - they were all fixed in 2.x. I've re-applied the changes from the old branch, onto the new code base, and created the branch https://github.com/jenkinsci/active-choices-plugin/tree/JENKINS-39593-2
Testing, I couldn't notice much difference. But maybe someone with a test server and a few more example projects could test it too? If anyone has some spare time to test it, here's a new version attach (uno-choice-20180527.hpi).
Thanks!
Bruno
Created a logger (org.biouno, log level ALL), so far the output looks consistent, executing the parameter only. Now will add more parameters (reactive now, before was using a choice parameter)