Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58382

Build is stuck due to blocked thread

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      v1.0.39

      Description

      There is a build which is stuck on intial stage.

      I saw in monitoring screen the below message:

      Threads on 50aa853439c2@XXXXXX: Number = 110, Maximum = 526, Total started = 464,563
      Warning, the following threads are deadlocked : Handling POST************/214/kill from XXXXX : qtp544724190-466789, jenkins.util.Timer 9, OkHttp https://*******.com:XXXX/..., OkHttp https://*******.com:XXXX/...
       
      I tried to kill that thread but it's not getting deleted.
      Then I went to the threadDump log and saw:

      jenkins.util.Timer 9

      "jenkins.util.Timer 9" Id=71 Group=main BLOCKED on java.lang.String@39c5d884 owned by "OkHttp https://*******.com:XXXX/..." Id=139290 at io.fabric8.jenkins.openshiftsync.JenkinsUtils.triggerJob(JenkinsUtils.java:422) - blocked on java.lang.String@39c5d884 at io.fabric8.jenkins.openshiftsync.BuildWatcher.addEventToJenkinsJobRun(BuildWatcher.java:345) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.handleBuildList(JenkinsUtils.java:815) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.maybeScheduleNext(JenkinsUtils.java:717) at io.fabric8.jenkins.openshiftsync.BuildSyncRunListener.onDeleted(BuildSyncRunListener.java:184) at hudson.model.listeners.RunListener.fireDeleted(RunListener.java:269) at hudson.model.Run.delete(Run.java:1573) at io.fabric8.jenkins.openshiftsync.JenkinsUtils.deleteRun(JenkinsUtils.java:567) - locked java.lang.Class@b802391 at io.fabric8.jenkins.openshiftsync.BuildWatcher.reconcileRunsAndBuilds(BuildWatcher.java:494) at io.fabric8.jenkins.openshiftsync.BuildWatcher.access$100(BuildWatcher.java:65) at io.fabric8.jenkins.openshiftsync.BuildWatcher$1.doRun(BuildWatcher.java:148) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@28c61284
       
       
      so I guess it's something with the openshift sync plugin. can you suggest what is the next step I should try to do?

        Attachments

          Activity

          Hide
          gmontero Gabe Montero added a comment -

          hey Ariel M - without seeing the full thread dump, it is hard to say.  Possibly cancelling all openshift pipeline strategy builds running on jenkins might help.  Otherwise, restarting Jenkins is the only option.

           

          think I have enough from the single thread stack trace to devise a fix, but again, if by some chance you have the full thread dump still, please attach as a text file (posting them as Jira comments are hard to read).

           

          thanks

          Show
          gmontero Gabe Montero added a comment - hey Ariel M - without seeing the full thread dump, it is hard to say.  Possibly cancelling all openshift pipeline strategy builds running on jenkins might help.  Otherwise, restarting Jenkins is the only option.   I  think I have enough from the single thread stack trace to devise a fix, but again, if by some chance you have the full thread dump still, please attach as a text file (posting them as Jira comments are hard to read).   thanks
          Hide
          arielma Ariel M added a comment -

          attached

          Show
          arielma Ariel M added a comment - attached
          Hide
          gmontero Gabe Montero added a comment -

          PR https://github.com/openshift/jenkins-sync-plugin/pull/317 is up for addressing the deadlock

           

          After it merges, I'll cherry pick it into https://github.com/jenkinsci/openshift-sync-plugin and cut a new version of the plugin

          Show
          gmontero Gabe Montero added a comment - PR https://github.com/openshift/jenkins-sync-plugin/pull/317 is up for addressing the deadlock   After it merges, I'll cherry pick it into https://github.com/jenkinsci/openshift-sync-plugin and cut a new version of the plugin
          Hide
          arielma Ariel M added a comment -

          Will it solve also the below issue I'm facing:

          Hitting build with parameters not triggering the build, only rebuild option works for me

          Show
          arielma Ariel M added a comment - Will it solve also the below issue I'm facing: Hitting build with parameters not triggering the build, only rebuild option works for me
          Hide
          gmontero Gabe Montero added a comment -

          I think so yes

          Show
          gmontero Gabe Montero added a comment - I think so yes
          Hide
          gmontero Gabe Montero added a comment -

          so v1.0.39 of the sync plugin has been released and it has the fix

           

          thanks

          Show
          gmontero Gabe Montero added a comment - so v1.0.39 of the sync plugin has been released and it has the fix   thanks
          Hide
          arielma Ariel M added a comment -

          Gabe, why do you think it will fix the build with parameters issue? what is the connection between two things?

          Show
          arielma Ariel M added a comment - Gabe, why do you think it will fix the build with parameters issue? what is the connection between two things?
          Hide
          gmontero Gabe Montero added a comment -

          because the parameter propagation goes through some of the same callback channels that your original deadlock went through Ariel

           

          That said, without debug data, I'm making an educated guess.  But if you want to pursue debugging, open a separate issue and provide another thread dump and the jenkins logs / job logs when the failure occurs.

          Show
          gmontero Gabe Montero added a comment - because the parameter propagation goes through some of the same callback channels that your original deadlock went through Ariel   That said, without debug data, I'm making an educated guess.  But if you want to pursue debugging, open a separate issue and provide another thread dump and the jenkins logs / job logs when the failure occurs.

            People

            Assignee:
            gmontero Gabe Montero
            Reporter:
            arielma Ariel M
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: