-
Bug
-
Resolution: Fixed
-
Major
-
Jenkins core 1.532.2
Build-timeout plugin 1.13
Upon upgrading plugin "build-timeout" from 1.12.2 to 1.13, all builds took about 50% longer to complete. This resulted in an unacceptably long build queue and builds timing out. This continued for 6 or 7 hours until an emergency downgrade of "build-timeout" from 1.13 back to 1.12.2 was done, which cleared the problem immediately. No other plugins were upgraded or downgraded during this time, nor were any other system wide configuration changes made.
I suspect the use of "synchronized" in the source code change made writing to the console effectively single threaded for all running builds. (My jenkins instance has 125 slave-nodes, so I have several dozens of concurrent builds all the time.)
Priority: I've made this "Major" because a mere install this plugin version causes my Jenkins instance to be unusable due to slower builds which cause a growing build queue and timed-out builds.
Environment: I'm running Jenkins core LTS 1.532.2 version. I'll be glad to furnish more information as you request it.
- is related to
-
JENKINS-26170 Improve the performance for NoActivityTimeOutStrategy
-
- Closed
-
[JENKINS-23012] Build-timeout plugin causes builds to slow
Wow, great job reproducing the problem!
Questions:
- Do you have any special advice I should pass on to my Jenkins users regarding their behavior to avoid slow builds?
- What's the next step in getting the fix?
And thank you so much for your work on this!
ikedam
added a comment - Do you have any special advice I should pass on to my Jenkins users regarding their behavior to avoid slow builds?
As this is a problem of the plugin, there's nothing users must do.
But the investigation I did indicates followings:
- Fast and much log outputs can have builds slow.
- Decreasing log outputs may quicken your builds.
- Outputting logs to a file and archiving that as a artifact may work, but it should depend on CPU, network bandwidths and disk IOs.
What's the next step in getting the fix?
I'll plan to make a new release in this month including other fixes and improvements.
Please wait that.
Please upgrade the plugin carefully as I'm not 100% sure this can resolve your problem. You should be ready to downgrade the plugin whenever you find the problem reproduce.
If you can, you'd better introduce a testing envirinment.
ikedam
added a comment -
Do you have any special advice I should pass on to my Jenkins users regarding their behavior to avoid slow builds?
As this is a problem of the plugin, there's nothing users must do.
But the investigation I did indicates followings:
Fast and much log outputs can have builds slow.
Decreasing log outputs may quicken your builds.
Outputting logs to a file and archiving that as a artifact may work, but it should depend on CPU, network bandwidths and disk IOs.
What's the next step in getting the fix?
I'll plan to make a new release in this month including other fixes and improvements.
Please wait that.
Please upgrade the plugin carefully as I'm not 100% sure this can resolve your problem. You should be ready to downgrade the plugin whenever you find the problem reproduce.
If you can, you'd better introduce a testing envirinment. Code changed in jenkins
User: ikedam
Path:
src/main/java/hudson/plugins/build_timeout/BuildTimeOutStrategy.java
src/main/java/hudson/plugins/build_timeout/BuildTimeoutWrapper.java
http://jenkins-ci.org/commit/build-timeout-plugin/e26ce053ef23921d164f46515d7d0ebb7e30c398
Log:
[FIXED JENKINS-23012] Resolved a performance problem introduced in 1.13 by capturing log outputs.
Code changed in jenkins
User: ikedam
Path:
src/main/java/hudson/plugins/build_timeout/BuildTimeOutStrategy.java
src/main/java/hudson/plugins/build_timeout/BuildTimeoutWrapper.java
http://jenkins-ci.org/commit/build-timeout-plugin/3a0984b42932286d21acc36f19dd96e315655fbb
Log:
Merge pull request #26 from ikedam/feature/JENKINS-23012_PerformanceProblem
JENKINS-23012 Resolved a performance problem introduced in 1.13 by capturing log outputs.
Compare: https://github.com/jenkinsci/build-timeout-plugin/compare/f71585212231...3a0984b42932
ikedam
added a comment - Fixed version is released as 1.14.
It will be available in a day.
Please try that version.
Please upgrade the plugin carefully as I'm almost but not 100% sure this can resolve your problem.
You should be ready to downgrade the plugin whenever you find the problem reproduce.
I'll close this ticket after you see it work.
ikedam
added a comment - Fixed version is released as 1.14.
It will be available in a day.
Please try that version.
Please upgrade the plugin carefully as I'm almost but not 100% sure this can resolve your problem.
You should be ready to downgrade the plugin whenever you find the problem reproduce.
I'll close this ticket after you see it work. I might be able to install it over the weekend. And I'll be careful. 
Thanks for your help with this! I'll be in touch.
Khai Do
added a comment - Hello. Thanks for your fix. I have installed and tested 1.14 in our multi-master jenkins CI and seems happy. I was not able to reproduce the performance issue.
Khai Do
added a comment - Hello. Thanks for your fix. I have installed and tested 1.14 in our multi-master jenkins CI and seems happy. I was not able to reproduce the performance issue. ikedam
added a comment - Thanks for the fix report.
Please reopen this issue for the case the problem remains.
ikedam
added a comment - Thanks for the fix report.
Please reopen this issue for the case the problem remains. 
a b
added a comment - I just noticed this ticket now, so sorry for the late comment. But taking a quick look at the code, the performance issue is quite obvious: in BuildTimeoutWrapper.decorateLogger you are only overriding write(byte) in the wrapper OutputStream - that is hugely inefficient, as most writes occur through write(byte[]) or write(byte[], int, int), and those translate to calling write(byte) one byte at a time. So, override write(byte[], int, int) too (write(byte[]) can be left as is) and I think you'll find a noticeable improvement.
a b
added a comment - I just noticed this ticket now, so sorry for the late comment. But taking a quick look at the code, the performance issue is quite obvious: in BuildTimeoutWrapper.decorateLogger you are only overriding write(byte) in the wrapper OutputStream - that is hugely inefficient, as most writes occur through write(byte[]) or write(byte[], int, int), and those translate to calling write(byte) one byte at a time. So, override write(byte[], int, int) too ( write(byte[] ) can be left as is) and I think you'll find a noticeable improvement.
https://github.com/jenkinsci/build-timeout-plugin/pull/26