-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Windows Server 2003
We have a Jenkins project that uses the build step "Trigger builds on other projects" and is set to "Block until the triggered projects finish their builds". This project has a lot of those dependencies and runs for several hours total (3 to 5). Sometimes the thinBackup kicks right in in the middle and puts Jenkins into Shutdown mode. In this case the main project is still active because it is blocked by the other triggered projects. And because Jenkins is in shutdown mode, it won't start building those; they stay in the queue. And the thinBackup client doesn't run either. Maybe because the Main Project is still active? (just an assumption). At this stage we have a deadlock. Nothing is going to build anymore, the backup is not happening and Jenkins won't come out of Shutdown mode. If I cancel the shutdown mode, one of the queued up projects starts to build, but shortly after that the shutdown mode will be re-enabled by the thinBackup plugin. You can see that in the log. The only way to get out of this state is to repeatedly cancel the shutdown until all queued projects finished building. Then the thinBackup plugin backs things up and takes Jenkins out of shutdown mode.
Could someone please take a look? I have the Jenkins error log attached.
Thanks,
Sandra
[JENKINS-16434] thinBackup plugin puts Jenkins into shutdown mode and does not come out of it
Hi Thomas,
I have attached a dependency graph screenshot of the involved projects. What happens is that the nightly starts and kicks off the rebuild_all. The rebuild_all is the one that takes a few hours and where the thinBackup kicks in while it's running. Jenkins is then put in quite mode and lets the rebuild_all finish. After that everything goes into deadlock.
As a workaround I have bumped the time when Jenkins is forced into quite mode to 10 hours. That helps for now since nothing takes that long to build for us. I haven't seen the deadlock since then (but it was just yesterday).
Do you want me to upload the config.xml files of some of the projects?
Thanks for looking into it so quickly, this is really appreciated 
Sandra
Hi Sandra,
many thanks for the upload. No, I do not think that it is neccessary to have the config.xml(s). I think the problem is clear and valid, I just need to findout how I can detect the deadlock.
BTW: interesting build structure 
looks like something realy awesome will be built
thomas
Thomas Fürer
added a comment - - edited Hi Sandra,
many thanks for the upload. No, I do not think that it is neccessary to have the config.xml(s). I think the problem is clear and valid, I just need to findout how I can detect the deadlock.
BTW: interesting build structure looks like something realy awesome will be built
thomas Thomas Fürer
added a comment - would be addressed in a major update of thinbackup, because currently it is a way to complicated to findout and fix the problem.
Thomas Fürer
added a comment - would be addressed in a major update of thinbackup, because currently it is a way to complicated to findout and fix the problem. 
Adrien CLERC
added a comment - We are experiencing the same issue.
See JENKINS-44952
Adrien CLERC
added a comment - We are experiencing the same issue.
See JENKINS-44952 for more information. 
as a quickfix we should work on JENKINS-15771; to fix the essential problem (deadlock) we need to findout more about the running job to detect the deadlock.