-
Bug
-
Resolution: Fixed
-
Major
-
None
-
-
v1.0.39
There is a build which is stuck on intial stage.
I saw in monitoring screen the below message:
Threads on 50aa853439c2@XXXXXX: Number = 110, Maximum = 526, Total started = 464,563
Warning, the following threads are deadlocked : Handling POST************/214/kill from XXXXX : qtp544724190-466789, jenkins.util.Timer 9, OkHttp https://*******.com:XXXX/..., OkHttp https://*******.com:XXXX/...
I tried to kill that thread but it's not getting deleted.
Then I went to the threadDump log and saw:
jenkins.util.Timer 9
"jenkins.util.Timer 9" Id=71 Group=main BLOCKED on java.lang.String@39c5d884 owned by "OkHttp https://*******.com:XXXX/..." Id=139290 at io.fabric8.jenkins.openshiftsync.JenkinsUtils.triggerJob(JenkinsUtils.java:422) - blocked on java.lang.String@39c5d884 at io.fabric8.jenkins.openshiftsync.BuildWatcher.addEventToJenkinsJobRun(BuildWatcher.java:345) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.handleBuildList(JenkinsUtils.java:815) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.maybeScheduleNext(JenkinsUtils.java:717) at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.onDeleted(BuildSyncRunListener.java:184) at hudson.model.listeners.RunListener.fireDeleted(RunListener.java:269) at hudson.model.Run.delete(Run.java:1573) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.deleteRun(JenkinsUtils.java:567) - locked java.lang.Class@b802391 at io.fabric8.jenkins.openshiftsync.BuildWatcher.reconcileRunsAndBuilds(BuildWatcher.java:494) at io.fabric8.jenkins.openshiftsync.BuildWatcher.access$100(BuildWatcher.java:65) at io.fabric8.jenkins.openshiftsync.BuildWatcher$1.doRun(BuildWatcher.java:148) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@28c61284
so I guess it's something with the openshift sync plugin. can you suggest what is the next step I should try to do?
[JENKINS-58382] Build is stuck due to blocked thread
Ariel M
created issue -
Ariel M
made changes -
| Attachment | New: Thread Dump.docx [ 47950 ] |
Gabe Montero
made changes -
| Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
Gabe Montero
added a comment - PR https://github.com/openshift/jenkins-sync-plugin/pull/317 is up for addressing the deadlock
After it merges, I'll cherry pick it into https://github.com/jenkinsci/openshift-sync-plugin and cut a new version of the plugin
Gabe Montero
added a comment - PR https://github.com/openshift/jenkins-sync-plugin/pull/317 is up for addressing the deadlock
After it merges, I'll cherry pick it into https://github.com/jenkinsci/openshift-sync-plugin and cut a new version of the plugin Ariel M
added a comment - Will it solve also the below issue I'm facing:
Hitting build with parameters not triggering the build, only rebuild option works for me
Ariel M
added a comment - Will it solve also the below issue I'm facing:
Hitting build with parameters not triggering the build, only rebuild option works for me Gabe Montero
added a comment - I think so yes Gabe Montero
added a comment - so v1.0.39 of the sync plugin has been released and it has the fix
thanks
Gabe Montero
added a comment - so v1.0.39 of the sync plugin has been released and it has the fix
thanks Gabe Montero
made changes -
| Released As | New: v1.0.39 | |
| Resolution | New: Fixed [ 1 ] | |
| Status | Original: In Progress [ 3 ] | New: Resolved [ 5 ] |
hey arielma - without seeing the full thread dump, it is hard to say. Possibly cancelling all openshift pipeline strategy builds running on jenkins might help. Otherwise, restarting Jenkins is the only option.
I think I have enough from the single thread stack trace to devise a fix, but again, if by some chance you have the full thread dump still, please attach as a text file (posting them as Jira comments are hard to read).
thanks