-
Bug
-
Resolution: Fixed
-
Major
-
None
-
-
v1.0.39
There is a build which is stuck on intial stage.
I saw in monitoring screen the below message:
Threads on 50aa853439c2@XXXXXX: Number = 110, Maximum = 526, Total started = 464,563
Warning, the following threads are deadlocked : Handling POST************/214/kill from XXXXX : qtp544724190-466789, jenkins.util.Timer 9, OkHttp https://*******.com:XXXX/..., OkHttp https://*******.com:XXXX/...
I tried to kill that thread but it's not getting deleted.
Then I went to the threadDump log and saw:
jenkins.util.Timer 9
"jenkins.util.Timer 9" Id=71 Group=main BLOCKED on java.lang.String@39c5d884 owned by "OkHttp https://*******.com:XXXX/..." Id=139290 at io.fabric8.jenkins.openshiftsync.JenkinsUtils.triggerJob(JenkinsUtils.java:422) - blocked on java.lang.String@39c5d884 at io.fabric8.jenkins.openshiftsync.BuildWatcher.addEventToJenkinsJobRun(BuildWatcher.java:345) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.handleBuildList(JenkinsUtils.java:815) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.maybeScheduleNext(JenkinsUtils.java:717) at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.onDeleted(BuildSyncRunListener.java:184) at hudson.model.listeners.RunListener.fireDeleted(RunListener.java:269) at hudson.model.Run.delete(Run.java:1573) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.deleteRun(JenkinsUtils.java:567) - locked java.lang.Class@b802391 at io.fabric8.jenkins.openshiftsync.BuildWatcher.reconcileRunsAndBuilds(BuildWatcher.java:494) at io.fabric8.jenkins.openshiftsync.BuildWatcher.access$100(BuildWatcher.java:65) at io.fabric8.jenkins.openshiftsync.BuildWatcher$1.doRun(BuildWatcher.java:148) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@28c61284
so I guess it's something with the openshift sync plugin. can you suggest what is the next step I should try to do?
[JENKINS-58382] Build is stuck due to blocked thread
Gabe Montero
added a comment - PR https://github.com/openshift/jenkins-sync-plugin/pull/317 is up for addressing the deadlock
After it merges, I'll cherry pick it into https://github.com/jenkinsci/openshift-sync-plugin and cut a new version of the plugin
Gabe Montero
added a comment - PR https://github.com/openshift/jenkins-sync-plugin/pull/317 is up for addressing the deadlock
After it merges, I'll cherry pick it into https://github.com/jenkinsci/openshift-sync-plugin and cut a new version of the plugin 
Ariel M
added a comment - Will it solve also the below issue I'm facing:
Hitting build with parameters not triggering the build, only rebuild option works for me
Ariel M
added a comment - Will it solve also the below issue I'm facing:
Hitting build with parameters not triggering the build, only rebuild option works for me Gabe Montero
added a comment - I think so yes Gabe Montero
added a comment - so v1.0.39 of the sync plugin has been released and it has the fix
thanks
Gabe Montero
added a comment - so v1.0.39 of the sync plugin has been released and it has the fix
thanks Ariel M
added a comment - Gabe, why do you think it will fix the build with parameters issue? what is the connection between two things?
Ariel M
added a comment - Gabe, why do you think it will fix the build with parameters issue? what is the connection between two things? Gabe Montero
added a comment - because the parameter propagation goes through some of the same callback channels that your original deadlock went through Ariel
That said, without debug data, I'm making an educated guess. But if you want to pursue debugging, open a separate issue and provide another thread dump and the jenkins logs / job logs when the failure occurs.
Gabe Montero
added a comment - because the parameter propagation goes through some of the same callback channels that your original deadlock went through Ariel
That said, without debug data, I'm making an educated guess. But if you want to pursue debugging, open a separate issue and provide another thread dump and the jenkins logs / job logs when the failure occurs. 
hey arielma - without seeing the full thread dump, it is hard to say. Possibly cancelling all openshift pipeline strategy builds running on jenkins might help. Otherwise, restarting Jenkins is the only option.
I think I have enough from the single thread stack trace to devise a fix, but again, if by some chance you have the full thread dump still, please attach as a text file (posting them as Jira comments are hard to read).
thanks