-
Bug
-
Resolution: Unresolved
-
Major
-
jenkins ver 2.176.3
jenkins ver 2.190.1
jenkins ver 2.190.3
jenkins ver 2.204.1
jenkins ver 2.204.2
I have a bunch of freestyle jobs, that compile and build an installer for my software on two compilers and in debug/release. Each of them takes aroung half an hour.
Then I tried to switch the build to a pipeline build, integrated build of debug and release to one job and additionally integrated several other stuff (cppcheck,...). This build ended up in taking 3-4 hours for quite a while (which I did not expect and regard it as slow also).
Since some weeks now my pipeline job suddenly started taking 8 hours while the freestyle jobs duration staid the same.
Something about my environment: it's a jenkins-installation as service on a windows server 2016 machine. I am not aware of any changes on the machine installation or job configuration since the job got slower. It's not a single step in the pipeline that eats all the time. It's the overall performance and the time is distributed over all steps. E.g. the compile and build installer step (which is the entire job done by the freestyle job) alone takes 3 hours.
Unfortunately I don't have the build history long enough to be able to identify plugin-versions from before and after the buildtime-increase.
Please contact me in case I can help with any additional information.
[JENKINS-59311] Poor performance after update
Christoph Fetzer
added a comment - Was solved somewhere between V2.176.2 and v2.190.1.
Relation to any plugin or module not clear.
Solution is released.
Christoph Fetzer
added a comment - Was solved somewhere between V2.176.2 and v2.190.1.
Relation to any plugin or module not clear.
Solution is released. This jenkins installation is driving me nuts - the slow behaviour came back.
The last quick run was on September, 27th. After that the build failed for some days due to issues with the code and the libraries used. The first completing build after that on Oct. 9th was slow again (same times - ~45min vs 3:40hours).
OK, I noticed, there are failed builds with indicatino of slow behaviour since October, 2nd. The builds before failed after only 6mins.
I implemented every updates in that time as soon as I realised them. There were a number of them in that time frame. Is there a log that shows the plugins updated and the version change? With that I could restore the state of September 27th and apply all the updates step by step until the build gets slower again.
Christoph Fetzer
added a comment - - edited This jenkins installation is driving me nuts - the slow behaviour came back.
The last quick run was on September, 27th. After that the build failed for some days due to issues with the code and the libraries used. The first completing build after that on Oct. 9th was slow again (same times - ~45min vs 3:40hours).
OK, I noticed, there are failed builds with indicatino of slow behaviour since October, 2nd. The builds before failed after only 6mins.
I implemented every updates in that time as soon as I realised them. There were a number of them in that time frame. Is there a log that shows the plugins updated and the version change? With that I could restore the state of September 27th and apply all the updates step by step until the build gets slower again.
Christoph Fetzer
added a comment - The speed decrease dissappeared again.
Maybe it got lost a little through the history of all the comments:
- the decrease happens only in one step, not distributed over all
- in case it runs slowly, removing a '| grep' from a makefile build rule immediately takes down the speed decrease
I now started tracking all the plugin updates to maybe get a correlation between jenkins updates and speed impact
Christoph Fetzer
added a comment - The speed decrease dissappeared again.
Maybe it got lost a little through the history of all the comments:
the decrease happens only in one step, not distributed over all
in case it runs slowly, removing a '| grep' from a makefile build rule immediately takes down the speed decrease
I now started tracking all the plugin updates to maybe get a correlation between jenkins updates and speed impact Christoph Fetzer
added a comment - A new finding:
for a lot of sequential updates I aborted the running job after recognizing the build speed. That caused hanging processes in the back that made cleaning the workplace impossible. To recover from that I rebooted the machine (without having applied an update). After that the build ran slowly again.
In the meantime the build got a little faste, it 'only' takes 5 hours instead of 8 previously. I can't find out the real time since when it happened because I had a number of failing builds in that time.
Christoph Fetzer
added a comment - A new finding:
for a lot of sequential updates I aborted the running job after recognizing the build speed. That caused hanging processes in the back that made cleaning the workplace impossible. To recover from that I rebooted the machine (without having applied an update). After that the build ran slowly again.
In the meantime the build got a little faste, it 'only' takes 5 hours instead of 8 previously. I can't find out the real time since when it happened because I had a number of failing builds in that time. Arg - this issue doesn't like being chased.
Every time I do a single action like update, reboot or restart behaviour stays the same.
Whenever I do simultaneous actions, something changes.
This time: update of Script security 166->1.67 and restart of jenkins (without reboot) due to hanging job fragments made the build fast again.
Noticeable: I restarted jenkins by starting and stopping the jenkins service this time.
BTW: really no support here? Where else can I get assistance?
Christoph Fetzer
added a comment - - edited Arg - this issue doesn't like being chased.
Every time I do a single action like update, reboot or restart behaviour stays the same.
Whenever I do simultaneous actions, something changes.
This time: update of Script security 166->1.67 and restart of jenkins (without reboot) due to hanging job fragments made the build fast again.
Noticeable: I restarted jenkins by starting and stopping the jenkins service this time.
BTW: really no support here? Where else can I get assistance? After the latest updates I got the problem again. Besides, all my automatic build weren't restartet....
The list of updates:
Jira 3.0.10 -> 3.0.11
Oracle Java SE Development Kit Installer 1.3->1.4
Script Security 1.67->1.68
Warnings Next Generation 7.2.0->7.2.1
Jenkins v2.190.3
It runs quickly again after a service restart.
So I guess, it's an issue of the update procedure or the implementation as a windows service?
Which components do these belong to for a correct assignment of the job?
Christoph Fetzer
added a comment - - edited After the latest updates I got the problem again. Besides, all my automatic build weren't restartet....
The list of updates:
Jira 3.0.10 -> 3.0.11
Oracle Java SE Development Kit Installer 1.3->1.4
Script Security 1.67->1.68
Warnings Next Generation 7.2.0->7.2.1
Jenkins v2.190.3
It runs quickly again after a service restart.
So I guess, it's an issue of the update procedure or the implementation as a windows service?
Which components do these belong to for a correct assignment of the job? Christoph Fetzer
added a comment - The issue is still present. Christoph Fetzer
added a comment - It still is present and restart of the service very reliably and repeatedly solves the issue. Could you please check the update procedure!
Christoph Fetzer
added a comment - It still is present and restart of the service very reliably and repeatedly solves the issue. Could you please check the update procedure! Christoph Fetzer
added a comment - I noticed the msBuild warnings parser also does the job for me and thus removed one of the "grep"-layers. Unforunately the problem still persists.
Christoph Fetzer
added a comment - I noticed the msBuild warnings parser also does the job for me and thus removed one of the "grep"-layers. Unforunately the problem still persists. 
The problem seems to have disappeared after the latest LTS update to v2.190.1. At least at the first test build after that update.