Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28532

Jobs get stuck in the queue

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • core
    • None
    • Jenkins 1.611, prioritysorter 3.2

      Unknown if core or plugin issue.
      Had a group of jobs stuck waiting in the queue.
      Hovering over them shows triggers and wait time as usual, but no indication of what they're waiting for.

      Suspicious log entry:
      May 21, 2015 4:43:19 PM hudson.util.DescribableList buildDependencyGraph
      SEVERE: Failed to build dependency graph for hudson.model.FreeStyleProject@57fdabe[MyStuckJob]
      java.lang.NullPointerException
      at hudson.tasks.Fingerprinter$FingerprintAction.getFingerprints(Fingerprinter.java:373)
      at hudson.tasks.Fingerprinter$FingerprintAction.getDependencies(Fingerprinter.java:403)
      at hudson.tasks.Fingerprinter$FingerprintAction.getDependencies(Fingerprinter.java:390)
      at hudson.tasks.Fingerprinter.buildDependencyGraph(Fingerprinter.java:157)
      at hudson.util.DescribableList.buildDependencyGraph(DescribableList.java:219)
      at hudson.model.Project.buildDependencyGraph(Project.java:207)
      at hudson.model.DependencyGraph.build(DependencyGraph.java:95)
      at jenkins.model.Jenkins.rebuildDependencyGraph(Jenkins.java:3748)
      at jenkins.model.Jenkins$25.call(Jenkins.java:3770)
      at jenkins.model.Jenkins$25.call(Jenkins.java:3766)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

      Possibly caused by deleting some other jobs - they weren't supposed to be in the dependency tree, but coincidentally included identical files that were fingerprinted at some point.

      Queuing strategy is Weighted Fair Queuing.
      Weren't in the queue on startup, unless lack of JENKINS-28486 pulling them in would count.

          [JENKINS-28532] Jobs get stuck in the queue

          James Howe created issue -
          James Howe made changes -
          Description Original: Unknown if core or plugin issue.
          Had a group of jobs stuck waiting in the queue.
          Hovering over them shows triggers and wait time as usual, but no indication of what they're waiting for.

          May 21, 2015 4:43:19 PM hudson.util.DescribableList buildDependencyGraph
          SEVERE: Failed to build dependency graph for hudson.model.FreeStyleProject@57fdabe[MyStuckJob]
          java.lang.NullPointerException
                  at hudson.tasks.Fingerprinter$FingerprintAction.getFingerprints(Fingerprinter.java:373)
                  at hudson.tasks.Fingerprinter$FingerprintAction.getDependencies(Fingerprinter.java:403)
                  at hudson.tasks.Fingerprinter$FingerprintAction.getDependencies(Fingerprinter.java:390)
                  at hudson.tasks.Fingerprinter.buildDependencyGraph(Fingerprinter.java:157)
                  at hudson.util.DescribableList.buildDependencyGraph(DescribableList.java:219)
                  at hudson.model.Project.buildDependencyGraph(Project.java:207)
                  at hudson.model.DependencyGraph.build(DependencyGraph.java:95)
                  at jenkins.model.Jenkins.rebuildDependencyGraph(Jenkins.java:3748)
                  at jenkins.model.Jenkins$25.call(Jenkins.java:3770)
                  at jenkins.model.Jenkins$25.call(Jenkins.java:3766)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)

          Possibly caused by deleting some other jobs - they weren't supposed to be in the dependency tree, but coincidentally included identical files that were fingerprinted at some point.

          Queuing strategy is Weighted Fair Queuing.
          Weren't in the queue on startup, unless lack of JENKINS-28486 pulling them in would count.           		New: Unknown if core or plugin issue.
          Had a group of jobs stuck waiting in the queue.
          Hovering over them shows triggers and wait time as usual, but no indication of what they're waiting for.

          Suspicious log entry:
          May 21, 2015 4:43:19 PM hudson.util.DescribableList buildDependencyGraph
          SEVERE: Failed to build dependency graph for hudson.model.FreeStyleProject@57fdabe[MyStuckJob]
          java.lang.NullPointerException
                  at hudson.tasks.Fingerprinter$FingerprintAction.getFingerprints(Fingerprinter.java:373)
                  at hudson.tasks.Fingerprinter$FingerprintAction.getDependencies(Fingerprinter.java:403)
                  at hudson.tasks.Fingerprinter$FingerprintAction.getDependencies(Fingerprinter.java:390)
                  at hudson.tasks.Fingerprinter.buildDependencyGraph(Fingerprinter.java:157)
                  at hudson.util.DescribableList.buildDependencyGraph(DescribableList.java:219)
                  at hudson.model.Project.buildDependencyGraph(Project.java:207)
                  at hudson.model.DependencyGraph.build(DependencyGraph.java:95)
                  at jenkins.model.Jenkins.rebuildDependencyGraph(Jenkins.java:3748)
                  at jenkins.model.Jenkins$25.call(Jenkins.java:3770)
                  at jenkins.model.Jenkins$25.call(Jenkins.java:3766)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)

          Possibly caused by deleting some other jobs - they weren't supposed to be in the dependency tree, but coincidentally included identical files that were fingerprinted at some point.

          Queuing strategy is Weighted Fair Queuing.
          Weren't in the queue on startup, unless lack of JENKINS-28486 pulling them in would count.
          James Howe made changes -
          Priority Original: Minor [ 4 ] New: Major [ 3 ]

          Magnus Sandberg added a comment -

          PrioritySorter will only order the Jobs in the Queue and will not interfere with execution/scheduling (if you do not use "Run Exclusive") so this does look like a problem somewhere else.

          Totally of the topic: You are the first one I see that uses anything else besides "Absolute" sorting - I would be happy if you can share your thought on the Queueing strategy, feel free to email me. Thanks,

          Magnus Sandberg added a comment - PrioritySorter will only order the Jobs in the Queue and will not interfere with execution/scheduling (if you do not use "Run Exclusive") so this does look like a problem somewhere else. Totally of the topic: You are the first one I see that uses anything else besides "Absolute" sorting - I would be happy if you can share your thought on the Queueing strategy, feel free to email me. Thanks,

          James Howe added a comment - - edited

          Downgraded plugin to 2.12. Restarted.
          Dependency graph errors not seen. Items already stuck in queue went through, though a few hours later they were stuck again.
          Status now shows as (pending—???)

          Manually deleted from disk fingerprints that mentioned jobs I'd recently deleted. Reloaded.
          As above.

          Opened and saved the config of a job that was stuck without making changes.
          Immediately all jobs that were stuck began executing.
          Next time the job came through the queue, it and some others got stuck again.

          James Howe added a comment - - edited Downgraded plugin to 2.12. Restarted. Dependency graph errors not seen. Items already stuck in queue went through, though a few hours later they were stuck again. Status now shows as (pending—???) Manually deleted from disk fingerprints that mentioned jobs I'd recently deleted. Reloaded. As above. Opened and saved the config of a job that was stuck without making changes. Immediately all jobs that were stuck began executing. Next time the job came through the queue, it and some others got stuck again.

          James Howe added a comment - - edited

          Doesn't appear to have gotten stuck again since then.

          Jobs got stuck again for a few hours. Fewer than before.
          This time clicking on one of them was enough to have them immediately execute.

          James Howe added a comment - - edited Doesn't appear to have gotten stuck again since then. Jobs got stuck again for a few hours. Fewer than before. This time clicking on one of them was enough to have them immediately execute.
          James Howe made changes -
          Priority Original: Major [ 3 ] New: Minor [ 4 ]
          James Howe made changes -
          Component/s Original: prioritysorter-plugin [ 15771 ]

          James Howe added a comment -

          Got stuck with a blank status, and a config save no longer pushes them though.
          Maybe there are two related issues here.

          May 28, 2015 2:12:32 PM FINE PrioritySorter.Queue.Items
          Blocking: Id: 1093, JobName: MyJob, jobGroupId: 1, reason: <none>, priority: 4, weight: 5.9999997E-4, status: BLOCKED

          James Howe added a comment - Got stuck with a blank status, and a config save no longer pushes them though. Maybe there are two related issues here. May 28, 2015 2:12:32 PM FINE PrioritySorter.Queue.Items Blocking: Id: 1093, JobName: MyJob, jobGroupId: 1, reason: <none>, priority: 4, weight: 5.9999997E-4, status: BLOCKED

          Magnus Sandberg added a comment -

          The log-line you have is telling us that the jenkins-core has changed status of the Job to BLOCKED - not more than that.

          Magnus Sandberg added a comment - The log-line you have is telling us that the jenkins-core has changed status of the Job to BLOCKED - not more than that.

            emsa23 Magnus Sandberg
            jameshowe James Howe
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: