I hate to create a general "core" bug, as I wish I could redirect this to the correct component. Unfortunately, I can not identify which component is hanging and why, so I do not know how to direct this problem.
This problem started about 2 weeks ago, as we have been adding new Pipeline builds to our build server. So it could be related to one of the pipeline plugins.
The behavior is the following:
- 1 to 2 times a day, all builds on all build slaves will hang. The console log of the build just stops moving forward, and stays stuck at the last line executed / last line returned.
- Once this occurs, attempting to stop a build fails. Clicking stop results in no change in the build status or console log output
- New builds will not start. They sit in the queue, but the slaves will not be started.
- The UI continues to function, so it is possible to view config, get threaddumps, etc.
The only resolution is to restart the Jenkins server.
We are using the vCenter plugin to dynamically start all build slaves. Though, we have been using this configuration for months, and the problem just started.
We have recreated this on both latest Jenkins level (2.26) and Jenkins LTS version 2.19.1
I am attaching a threaddump of the server at the time of one of these hangs.
I can provide any other information that might help in diagnosing this problem